IMO some of this stems from a kind of author/character confusion by human observers.
The LLM can generate an engaging story from the perspective of a ravenous evil vampire, but that doesn't mean that vampires are real nor that the LLM wants to suck your blood.
Re-running the same process but changing the prompt from "you are a vampire" to "you are an LLM", doesn't magically create a metaphysical connection to the real-world algorithm.
The LLM can generate an engaging story from the perspective of a ravenous evil vampire, but that doesn't mean that vampires are real nor that the LLM wants to suck your blood.
Re-running the same process but changing the prompt from "you are a vampire" to "you are an LLM", doesn't magically create a metaphysical connection to the real-world algorithm.