In those places where a system like Sweden's has been implemented, the usage is constricted to certain areas. And in the case where it's used elsewhere, that's an option that is not mandatory (and in any case far and few between). A way to identify an individual is typically related to financial or contractual issues. So far, at least. Looking at you, the UK
Yes, but those "certain areas" are mandatory for functioning in society. And that ID is managed by a single central authority.
The US by contrast, has a distributed system where there are many authorities that can issue IDs that are valid for the activities of daily life.
The only common nationally issued ID in the US is a passport and people only get that for international travel -- and it wasn't even until 2024 that a majority of Americans even had one.
With many authorities then you have as many more possibilities to break them, right? Note that the central digital ID used in e.g. Sweden is not the same as a central place for storing your private information.
The problem occurs any time someone need an ID to do something they need to do, and are unable to get one. It is better with multiple authorities because people have more options for obtaining that prerequisite valid ID.
There are many instances throughout history of ID requirements being weaponized to suppress minority groups, from Apartheid Pass Laws, Jim Crow, or recent US suppression of minority, young, or transgender voters.
Of course, these things certainly aren't fixed by simply having more options to get a valid ID, but they are mitigated to some degree.