I don't believe that they believe it, I believe that they're all in on doing all the things you'd do if your goal was to demonstrate to investors that you truly believe it.
The safety-focused labs are the marketing department.
An AI that can actually think and reason, and not just pretend to by regurgitating/paraphrasing text that humans wrote, is not something we're on any path to building right now. They keep telling us these things are going to discover novel drugs and do all sorts of important science, but internally, they are well aware that these LLM architectures fundamentally can't do that.
A transformer-based LLM can't do any of the things you'd need to be able to do as an intelligent system. It has no truth model, and lacks any mechanism of understanding its own output. It can't learn and apply new information, especially not if it can't fit within one context window. It has no way to evaluate if a particular sequence of tokens is likely to be accurate, because it only selects them based on the probability of appearing in a similar sequence, based on the training data. It can't internally distinguish "false but plausible" from "true but rare." Many things that would be obviously wrong to a human, would appear to be "obviously" correct when viewed from the perspective of an LLM's math.
These flaws are massive, and IMO, insurmountable. It doesn't matter if it can do 50% of a person's work effectively, because you can't reliably predict which 50% it will do. Given this unpredictability, its output has to be very carefuly reviewed by an expert in order to be used for any work that matters. Even worse, the mistakes it makes are meant to be difficult to spot, because it will always generate the text that looks the most right. Spotting the fuckup in something that was optimized not to look like a fuckup is much more difficult than reviewing work done by a well-intentioned human.
No, Anthropic and OpenAI definitely actually believe what they're saying. If you believe companies only care about their shareholders, then you shouldn't believe this about them because they don't even have that corporate structure - they're PBCs.
There doesn't seem to be a reason to believe the rest of this critique either; sure those are potential problems, but what do any of them have to do with whether a system has a transformer model in it? A recording of a human mind would have the same issues.
> It has no way to evaluate if a particular sequence of tokens is likely to be accurate, because it only selects them based on the probability of appearing in a similar sequence, based on the training data.
This in particular is obviously incorrect if you think about it, because the critique is so strong that if it was true, the system wouldn't be able to produce coherent sentences. Because that's actually the same problem as producing true sentences.
(It's also not true because the models are grounded via web search/coding tools.)
> if it was true, the system wouldn't be able to produce coherent sentences. Because that's actually the same problem as producing true sentences
It is...not at all the same? Like they said, you can create perfectly coherent statements that are just wrong. Just look at Elon's ridiculously hamfisted attempts around editing Grok system prompts.
Also, a lot of information on the web is just wrong or out of date, and coding tools only get you so far.
"Paris is the capital of France" is a coherent sentence, just like "Paris dates back to Gaelic settlements in 1200 BC", or "France had a population of about 97,24 million in 2024".
The coherence of sentences generated by LLMs is "emergent" from the unbelievable amount of data and training, just like the correct factoids ("Paris is the capital of France").
It shows that Artificial Neural Networks using this architecture and training process can learn to fluently use language, which was the goal? Because language is tied to the real world, being able to make true statements about the world is to some degree part of being fluent in a language, which is never just syntax, also semantics.
I get what you mean by "miracle", but your argument revolving around this doesn't seem logical to me, apart from the question: what is the the "other miracle" supposed to be?
Zooming out, this seems to be part of the issue: semantics (concepts and words) neatly map the world, and have emergent properties that help to not just describe, but also sometimes predict or understand the world.
But logic seems to exist outside of language to a degree, being described by it. Just like the physical world.
Humans are able to reason logically, not always correctly, but language allows for peer review and refinement. Humans can observe the physical world. And then put all of this together using language.
But applying logic or being able to observe the physical world doesn't emerge from language. Language seems like an artifact of doing these things and a tool to do them in collaboration, but it only carries logic and knowledge because humans left these traces in "correct language".
> But applying logic or being able to observe the physical world doesn't emerge from language. Language seems like an artifact of doing these things and a tool to do them in collaboration, but it only carries logic and knowledge because humans left these traces in "correct language".
That's not the only element that went into producing the models. There's also the anthropic principle - they test them with benchmarks (that involve knowledge and truthful statements) and then don't release the ones that fail the benchmarks.
And there is Reinforcement Learning, which is essential to make models act "conversational" and coherent, right?
But I wanted to stay abstract and not go into to much detail outside my knowledge and experience.
With the GPT-2 and GPT-3 base models, you were easily able to produce "conversations" by writing fitting preludes (e.g. Interview style), but these went off the rails quickly, in often comedic ways.
Part of that surely is also due to model size.
But RILHF seems more important.
I enjoyed the rambling and even that was impressive at the time.
I guess the "anthropic principle" you are referring to works in a similar direction, although in a different way (selection, not training).
The only context in which I've heard details about selection processes post-training so far was this article about OpenAIs model updates from GPT-4o onwards, discussed earlier here:
The parts about A/B-Testing are pretty interesting.
The focus is ChatGPT as an enticing consumer product and maximizing engagement, not so much the benchmarks and usefulness of models. It briefly addresses the friction between usefulness and sycophancy though.
Anyway, it's pretty clever to use the wording "anthropic principle" here, I only knew the metaphysical usage (why do humans exist).
Because it's not a miracle? I'm not being difficult here, it's just true. It's neat and fun to play with, and I use it, but in order to use anything well, you have to look critically at the results and not get blinded by the glitter.
Saying "Why can't you be amazed that a horse can do math?" [0] means you'll miss a lot of interesting phenomena.
If you dont believe they believe it you havent paid any attention to the company. Maybe Dario is lying, although that would be an extremely long con, but the rank and file 100% believe it.
The safety-focused labs are the marketing department.
An AI that can actually think and reason, and not just pretend to by regurgitating/paraphrasing text that humans wrote, is not something we're on any path to building right now. They keep telling us these things are going to discover novel drugs and do all sorts of important science, but internally, they are well aware that these LLM architectures fundamentally can't do that.
A transformer-based LLM can't do any of the things you'd need to be able to do as an intelligent system. It has no truth model, and lacks any mechanism of understanding its own output. It can't learn and apply new information, especially not if it can't fit within one context window. It has no way to evaluate if a particular sequence of tokens is likely to be accurate, because it only selects them based on the probability of appearing in a similar sequence, based on the training data. It can't internally distinguish "false but plausible" from "true but rare." Many things that would be obviously wrong to a human, would appear to be "obviously" correct when viewed from the perspective of an LLM's math.
These flaws are massive, and IMO, insurmountable. It doesn't matter if it can do 50% of a person's work effectively, because you can't reliably predict which 50% it will do. Given this unpredictability, its output has to be very carefuly reviewed by an expert in order to be used for any work that matters. Even worse, the mistakes it makes are meant to be difficult to spot, because it will always generate the text that looks the most right. Spotting the fuckup in something that was optimized not to look like a fuckup is much more difficult than reviewing work done by a well-intentioned human.