What's the most efficient way of working with #FoundationModels and #LLMs? Generally I have found that I like to keep machines busy. So, something is continuously training and running validations, while I'm preparing for the next experiments and improvements.
This means that the machine isn't waiting idle for my work and my work isn't waiting idle for the machine to finish. It means that I have to mentally keep track of runs started a while ago, to make maximal use of their results, even if I am already working on the next thing which is a moving target.
It means I have to carefully design the experiments or runs so that I know what knowledge I gain from them, so that I can add that knowledge to the pile of learnings even if the actual codebase has progressed from that point already.
It also means that I have to make multiple training or validation runs in parallel in a way that doesn't stop me from working on something else, while keeping the information of the runs somewhere so that they can be retrieved later. Also, it means that I often need to incorporate multiple learnings from these runs to the subsequent runs all together, in a YOLO sort of a way, instead of trying to do slow but structured "change one thing at a time" type of more systematic progress.
In my experience this is the most effective way to work on these types of things.
Keep machines running, do not wait for results, but make sure the results are useful when you eventually get them.