I agree with you, of course, that we should test our assumptions empirically as a general point.
However, there isn't time to test out every single assumption we could generally have.
Therefore, the more worthwhile experiments are ones where we learn something interesting no matter what happens. I'm equating this with "surprise," as in, we have done some meaningful gradient descent or Bayesian update, we've changed our views, we know something that wasn't obvious before.
You could disagree with semantics there, but hopefully we agree with the idea of more vs. less valuable experiments.
I'm just not sure whose model of LLM dynamics was updated by this paper. Then again, I only listened to a couple minutes of their linked YouTube discussion before getting bored.
However, there isn't time to test out every single assumption we could generally have.
Therefore, the more worthwhile experiments are ones where we learn something interesting no matter what happens. I'm equating this with "surprise," as in, we have done some meaningful gradient descent or Bayesian update, we've changed our views, we know something that wasn't obvious before.
You could disagree with semantics there, but hopefully we agree with the idea of more vs. less valuable experiments.
I'm just not sure whose model of LLM dynamics was updated by this paper. Then again, I only listened to a couple minutes of their linked YouTube discussion before getting bored.