If you don't want to host Docker images, you can just provide a Dockerfile. That way the onus of resolving all the complications is on you - your user only needs to have Docker running on their system.
Arguably, it's a pretty reasonable requirement. Widely used, mature, easy to set up.
I don't remember when I switched to running all my dev envs in Docker, but I wouldn't go back.
How far are we going to need to go to fully abstract these systems? Am I going to need a separate computer running a VM with a server image for hosting a docker image of python venv to manage a package that prints some text?
Arguably, it's a pretty reasonable requirement. Widely used, mature, easy to set up.
I don't remember when I switched to running all my dev envs in Docker, but I wouldn't go back.