Keeping AI sharp.
At this snapshot in time, we have an understanding of best practices for physicians. Those are encoded in licensing exams, continuous education materials, and other mediums.
What if, tomorrow, that changed? What if we learned that the best way to treat disease X wasn’t treatment Y but treatment Z. Well, we’d have to update our medical licensing exams, and start teaching med school students the new way.
The med school students can learn, but can an LLM handle such targeted knowledge updates without throwing everything else off? Or would a new training run be needed? If so, is it feasible to identify all of the training data with the old information so that it can be replaced with the new information?
This is a tricky question and requires research, but in the meantime, we’ll see model rot, akin to software rot, where these models don’t hold up. So there will be additional pressure on firms to train and release the latest and greatest model. They need to stay ahead of the competition on performance and also stay ahead of themselves and the rot in their older models.