I’ve mentioned this before but it’s worth revisiting.
As the costs associated with LLMs increase, and the models get harder and harder to train, host, pay for, etc. there will be an odd phenomenon of black box investigation of what is effectively a probabilistic information retrieval system. Will technical researchers need social science skills to interrogate these models, as anthropologists do? Or is another option to lean heavily into reproducibilty, where every version, API call, prompt, sequence, etc. is logged so that research work can be reproduced? In that case, what happens when firms are bought and shutter an API? Now on top of the inherently black box of the model, there is the black box of the organization that trains and hosts the model.