Part of the benefit of FOSS & open source is that a curious user can inspect how something is made and learn from it. It matters that open weights are no different from a compiled program. Sure, you can always modify an executable's instructions, but there's no openness there.

Then there's the problems of the content of the training data, which parallel the dangers of opaque algorithms.