But that's the same with AMP. If every site doesn't have AMP, penalizing them for not having AMP changes nothing.

In both cases it's an unstable equilibrium. The first site to be fast will get all the clicks. Or the first site to use AMP.