After watching the demos I'm convinced that the new context length will have the biggest impact. The ability to dump 32k tokens into a prompt (25,000 words) seems like it will drastically expand the reasoning capability and number of use cases. A doctor can put an entire patient's medical history in the prompt, a lawyer an entire case history, etc.
As a professional...why not do this? There's a non-zero chance that it'll find something fairly basic that you missed and the cost is several cents. Even if it just phrases something obvious in a way that makes you think, it's well worth the effort for a multimillion dollar client.
If they further increase the context window, this thing becomes a Second Opinion machine. For pretty much any high level job. If you can put in ALL of the information relevant to a problem and it can algorithmically do reasoning, it's essentially a consultant that works for pennies per hour. And some tasks that professionals do could be replaced altogether. Out of all the use cases for LLMs that I've seen so far, this seems to me to have the biggest potential impact on daily life.
edit (addition): What % of people can hold 25,000 words worth of information in their heads, while effectively reasoning with and manipulating it? I'm guessing maybe 10% at most, probably fewer. And they're probably the best in their fields. Now a computer has that ability. And anyone that has $20 for the OpenAI api can access it. This could get wild.
What we really need is a model that you can run on your own hardware on site. I could never use this for business because they're reading everything you send through it, but let me run it on my own server and it would be unbelievably useful.
Imagine being able to ask your workplace server if it has noticed any unusual traffic, or to write a report on sales with nice graphs. It would be so useful.
> What we really need is a model that you can run on your own hardware on site.
we won’t have that until we come up with a better way to fund these things. “””Open””” AI was founded on that idea, had the most likely chance of anyone in reaching it: even going into things with that intent they failed and switched to lock down the distribution of their models, somehow managed to be bought by MS despite the original non-profit-like structure. you just won’t see what you’re asking for for however long this field is dominated by the profit motive.
Curious why even companies at the very edge of innovation are unable to build moats?
I know nothing about AI, but when DALLE was released, I was under the impression that the leap of tech here is so crazy that no one is going to beat OpenAI at it. We have a bunch now: Stable Diffusion, MidJourney, lots of parallel projects that are similar.
Is it because OpenAI was sharing their secret sauce? Or is it that the sauce isn’t that special?
Google got a patent on transfomers but didn't enforce it.
If it wasn't for patents you'd never get a moat from technology. Google, Facebook, Apple and all have a moat because of two sided markets: advertisers go where the audience is, app makers go where the users are.
(There's another kind of "tech" company that is wrongly lumped in with the others, this is an overcapitalized company that looks like it has a moat because it is overcapitalized and able to lose money to win market share. This includes Amazon, Uber and Netflix.)
I don't think this is strictly true, though it's rare. The easiest example is the semiconductor industry. ASML's high end lithography machines are basically alien and cannot be reproduced by anyone else. China has spent billions trying. I don't even think there's a way to make the IP public because of how much of it is in people's heads and in the processes in place. I wonder how much money, time and ASML resources it would take to stand up a completely separate company that can do what ASML does assuming that ASML could dedicate 100% of their time in assisting in training the personnel at said company.
The semiconductor industry is only tangentially or partially a tech company. They're producing physical goods that require complex physical manufacturing processes. The means of production are expensive, complex, and require significant expertise to operate once set up. The whole thing involves multiple levels of complex engineering challenges. Even if you wanted to make a small handful of chips, you'd still have to go through all that.
Most modern tech companies are software companies. To them, the means of production are a commodity server in a rack. It might be an expensive server, but that's actually dependent on scale. It might even be a personal computer on a desk, or a smartphone in a pocket. Further, while creating software is highly technical, duplicating it is probably the most trivial computing operation that exists. Not that distribution is trivial (although it certainly can be) just that if you have one copy of software or data, you have enough software or data for 8 billion people.
No, I think it's very clear that upthread is talking about how software is difficult to build a moat around.
Chip fabs are literally one of the most expensive facilities ever created. Saying that because they don't need a special moat so therefore nothing in tech ever needs a special moat is so willfully blind that it borders on disingenuity.
That's the comment you should have responded with instead of the one that you did.
Upthread used the term "tech" when the thread is very clearly talking about AI. AI is software, but because they used the term "tech" you cherry-picked non-software tech as a counter example. It doesn't fit because the type of tech that GPT-4 represents doesn't have the manufacturing cost like a chip fab does. It's totally different in kind regardless of the fact that they're both termed "tech".
Yeah, this is probably also true for TSMC, Intel and ARM. Look how slow progress is on RISC-V on the high end despite RISC-V having the best academic talent.
Unfortunately, RISC-V, despite the "open source" marketing, is still basically dominated by one company (SiFive) that designs all the commercial cores. They also employ everyone who writes the spec, so the current "compiled" spec document is about 5 years behind the actual production ISA. Intel and others are trying to break this monopoly right now.
Compare this to the AI ecosystem and you get a huge difference. The architecture of these AI systems is pretty well-known despite not being "open," and there is a tremendous amount of competition.
Read the RISC-V foundation website. There are numerous "ratified" parts of the RISC-V instruction set that are not in the latest "compiled" spec document.
Saying a "compiled" spec is out of date may be technically accurate (or not, I don't have any idea) but if open, published documentation of the ratified extensions is on the web site, it's misleading to cite it as evidence that the spec is not open. And I know that the draft specifications are open for public comment prior to being ratified, so it's not a secret what's under development, either.
I never said that it wasn't actually open source. I just said that the openness hasn't actually created meaningful competition, because there is a single company in control of the specs that abuses that control to create a moat.
For a concrete example, the bitmanip extensions (which provide significant increases in MIPS/MHz) were used by SiFive in commercial cores before ratification and finalization. No other company could do that because SiFive employees could just change the spec if they did. They're doing the same thing with vector/SIMD instructions now to support their machine learning ambitions.
I would also add Samsung semi to that list. As I understand, for the small nodes, everyone is using ASML. That's a bit scary to me.
About RISC-V: What does you think is different about RISC-V vs ARM? I can only think that ARM has been used in the wild for longer, so there is a meaningful feedback loop. Designers can incorporate this feedback into future designs. Don't give up hope on RISC-V too soon! It might have a place in IoT which needs more diverse compute.
They patented the original Transformer encoder-decoder architecture. But most modern models are built either only out of encoders (the BERT family) or only out of decoders (the GPT family).
Even if they wanted to enforce their patent, they couldn't. It's a classic problem with patenting things that every lawyer warns you about "what if someone could make a change to circumvent your patent".
Are you kidding? There are 30 claims, it's an hours' work to make complete sense of how these work together and what they possibly do/do not cover. I've filed my own patents so have read thru enough of prior art and am not doing it for a pointless internet argument.
IANAL. I looked through the patent, not just the Claims. I certainly didn't read all of it. But while it leaves open many possible variations, it's a patent for sequence transduction and it's quite explicit everywhere that the system comprises a decoder and an encoder (see Claim 1, the most vague) and nowhere did I see any hint that you could leave out one or the other or that you could leave out the encoder-decoder attention submodule (the "degenerate use-case" you suggested). The patent is only about sequence transduction (e.g. in translation).
Now an encoder+decoder is very similar to a decoder-only transformer, but it's certainly an inventive step to make that modification and I'm pretty sure the patent doesn't contain it. It does describe all the other pieces of a decoder/encoder-only transformer though, despite not being covered by any of the claims, and I have no idea what a court would think about that since IANAL.
Or, Amazon, Uber, and Netflix have access to so much capital based on investors' judgment that they will be able to win and protect market share by effective execution, thereby creating a defensible moat.
It's because moving forward is hard, but moving backward when you know what the space of answers is, is much easier.
Once you know that OpenAI gets a certain set of results with roughly technology X, it's much easier to recreate that work than to do it in the first place.
This is true of most technology. Inventing the telephone is something, but if you told a competent engineer the basic idea, they'd be able to do it 50 years earlier no problem.
Same with flight. There are some really tricky problems with counter-intuitive answers (like how stalls work and how turning should work; which still mess up new pilots today). The space of possible answers is huge, and even the questions themselves are very unclear. It took the Wright brothers years of experiments to understand that they were stalling their wing. But once you have the basic questions and their rough answers, any amateur can build a plane today in their shed.
I agree with your overall point, but I don't think that we'd be able to get the telephone 50 years earlier because of how many other industries had to align to allow for its invention. Insulated wire didn't readily or cheaply come in spools until after the telegraph in the 1840's. The telephone was in 1876 so 50 years earlier was 1826.
You didn't mention it explicitly but I think the morale factor is also huge. Once you know it's possible, it does away with all those fears of wasted nights/weekends/resources/etc for something that might not actually be possible.
I think it's because everyone's swimming in the same bath. People move around between companies, things are whispered, papers are published, techniques are mentioned and details filled in, products are backwards-engineered. Progress is incremental.
The sauce is special, but the recipe is already known. Most of the stuff things like LLMs are based on comes from published research, so in principle coming up with the architecture that can do something very close, is doable to everyone with the skills to understand the research material.
The problems start with a) taking the architecture to a finished and fine tuned model and b) running that model. Because now we are talking about non-trivial amounts of compute, storage and bandwidth, so quite simple resources suddenly become a very real problem.
OpenAI can't build a moat because OpenAI isn't a new vertical, or even a complete product.
Right now the magical demo is being paraded around, exploiting the same "worse is better" that toppled previous ivory towers of computing. It's helpful while the real product development happens elsewhere, since it keeps investors hyped about something.
The new verticals seem smaller than all of AI/ML. One company dominating ML is about as likely as a single source owning the living room or the smartphones or the web. That's a platitude for companies to woo their shareholders and for regulators to point at while doing their job. ML dominating the living room or smartphones or the web or education or professional work is equally unrealistic.
I'm not sure how "keep the secret sauce secret and only offer it as a service" isn't a moat? Here the 'secret sauce' is the training data and the trained network, not the methodology, but the way they're going, it's only a matter of time before they start withholding key details of the methodology too.
I also expect a high moat, especially regarding training data.
But the counter for the high moat would be the atomic bomb -- the soviets were able to build it for a fraction of what it cost the US because the hard parts were leaked to them.
GPT-3 afik is an easier picking because they used a bigger model than necessary, but afterwards there appeared guidelines about model size vs. training data, so GPT-4 probably won't be as easily trimmed down.
You can have the most special sauce in the world but if you're hiding it in the closet because you fear that it will hurt sales of your classic sauce then don't be surprised with what will happen (also known as Innovators Dilemma)
Because we are headed to a world of semi-automated luxury socialism. Having a genius at your service for less than $1000 per year is just an insane break to the system we live in. We all need to think hard about how to design the world we want to live in.
> we won’t have that until we come up with a better way to fund these things.
Isn't this already happening with LLaMA and Dalai etc.? Already now you can run Whisper yourself. And you can run a model almost as powerful as gpt-3.5-turbo. So I can't see why it's out of bounds that we'll be able to host a model as powerful as gpt4.0 on our own (highly specced) Mac Studio M3s, or whatever it may be.
I don’t think on site is going to be necessary. Even the US intelligence community trusts that Amazon isn’t spying on the spies.
But a model that can run on a private cluster is certainly something that there’s going to be demand for. And once that exists there’s no reason it couldn’t be run on site.
You can see why OpenAI doesn’t want to do it though. SaaS is more lucrative.
No, the grandparent poster was right. That’s other agencies, not the intelligence community. He’s right that the cloud I was thinking of is on prem but with Amazon personal (that are cleared).
So not the greatest analogy. But still I think most doctors, lawyers etc should be okay with their own cluster running in the cloud.
Not lawyers in the US at least, that would typically be a violation of confidentiality. Even with a client's permission, it would work a waiver of attorney-client privilege. (I don't use GPT but I'm assuming the ToS is clear that someone there can examine the input material? Can it even be used to build their model, i.e., submitted information could potentially work it's way back to the eyes of the public and not just OpenAI engineers?) I imagine HIPAA issues would stop doctors. Can HIPAA data be stored on the cloud? Every instance I've seen they store it locally.
I agree with you on the SaaS version but the scenario I was thinking of was where there is a licensable model that can be run on a cluster in law firm’s AWS account. I think that should be okay.
HIPAA data can definitely be stored in the cloud given the right setup. I’ve worked for companies that have done so (the audit is a bit of a pain.)
I work in legaltech, and we use cloud services like aws for lawsuit data, and lawyers trust it. Any 3rd party must of course be vetted and go through NDA, and follow regional laws and guidelines ect, but using the cloud is definitely used for legaltech documents including sensitive data.
It should be added that legaltech vendors are often employed as go-betweens for quite adversarial interactions, such as e-discovery, that require them to be trusted (to a degree) by both sides of a case, even if they are being paid by one side.
Seems like there are lots of confidentiality and reliability issues in how tech is being used in law right now, but there aren't that many attorneys who understand the issues, and those that do find it more advantageous to overlook them unless forced to do otherwise.
HIPAA regulated organizations routinely store protected health information on the cloud. This has been common practice for many years. The physical location is legally irrelevant as long as security and privacy requirements are met. AWS and other large cloud vendors specifically target this market and make it easy to achieve legal compliance.
Are they even aware of where their data is? Opening a web browser might be a big hint for them, but how about editing something in Microsoft Office? Does the data there ever touch the cloud? Do Chromebooks make it clear enough where the data is?
I imagine lawyers knowing about where document data is stored as a bit like software developers being sufficiently aware of licensing. There's plenty who are paying attention, but there's also plenty who are simply unaware.
> You can see why OpenAI doesn’t want to do it though.
Except they already do offer private cluster solutions, you just need usage in the hundreds of millions of tokens per day before they want to talk to you (as in they might before that, but that’s the bar they say on the contact us page).
VMware charges people per GB RAM attached to a VM. Selling on-prem software on consumption is very much possible. It's closed source software, so as long as they require 443 outbound to tick consumption that'd work.
maybe we implement tokenizer+first layer in Javascript on client side and that is enough to preserve raw data on client side and send to GPT only first layer (which is a vector of float values anyway)
matrix gets decoded into text on the client side in Javascript, so we receive send and receive from chatGPT only vector of floats (obfuscation?)
It's a good idea but it seems quite easy to invert the first layer mapping. And the output of the last layer you can easily steal just by doing whatever would've been done in the client.
Could open ai just offer letting you upload a key and use it for interaction with the model? Basically encrypt the model with the key and all the request and responses are all secure?
That model will be out in a few years. GPT-3 175b only took two years until someone trained an open source equivalent that could run on a few gpu devices.
Depends on the definition of "generation" being used. One definition of generation is "about 30 years", i.e., the amount of time it takes to go from infancy to raising a child. See definition 6 (as of time of writing): https://en.wiktionary.org/wiki/generation#Noun
the problem is that if you steal the weights then you can serve your own gpt4, and it's very hard to prove that what you're serving is actually gpt4. (or you could just start using it without paying ofc)
None of the siblings are right. The models themselves are idempotent: given the same context you will get the same activations. However the output distribution is sampled in a pseudorandom way by these chat tools. You can seed all the prngs in the system to always have reproducible output using sampling, or even go beyond that and just work with the raw probability distribution by hand.
Right. They are idempotent (making an API call doesn't cause a state change in the model[0] per se), but not necessarily deterministic (and less so as you raise the temp).
It is possible to architect things to be fully deterministic with an explicit seed for the pseudorandom aspects (which is mostly how Stable Diffusion works), but I haven't yet seen a Chatbot UI implementation that works that way.
[0] Except on a longer timeframe where the request may be incorporated into future training data.
That's the feature of chat - it remembers what has been said and that changes the context in which it says new things. If you use the API it starts fresh each time, and if you turn down the 'temperature' it produces very similar and identical answers.
Pseudo random numbers are injected into the models via its temperature settings, but OpenAI could seed that to get the same answers with the same input. I’m going out on a limb here with pure speculation but given the model, a temperature, and a known text prompt, OpenAI could probably reverse engineer a seed and prove that the weights are the same.
Since fine-tuning is often done by freezing all but the top layers I wonder if it would still be possible to take a set of inputs and outputs and mathematically demonstrate that a model is derivative of ChatGPT. There may well be too much entropy to unpack, but I’m sure there will be researchers exploring this, if only to identify AI-generated material.
Of course, since the model is so large and general purpose already, I can’t assume the same fine-tuning techniques are used as for vastly smaller models, so maybe layers aren’t frozen at all.
Piet Hein wrote that in reference to the first operator-free elevators, some 70+ years ago.
What you call hallucination, I call misremembering. Humans do it too. The LLM failure modes are very similar to human failure modes, including making up stuff, being tricked to do something they shouldn't, and even getting mad at their interlocutors. Indeed, they're not merely thinking, they're even thinking wrong.
I don't think it's very salient that LLMs make stuff up, or can be manipulated into saying something they have been trained not to say. An LLM applies a statistical model to the problem of probability assignment over a range of tokens; a token of high probability is selected and the process repeats. This is not what humans do when humans think.
Given that GPT-4 is a simply large collection of numbers that combine with their inputs via arithmetic manipulation, resulting in a sequence of numbers, I find it hard to understand how they're "thinking".
Are you sure? Our senses have gaps that are being constantly filled all day long, it just gets more noticeable when our brain is exhausted and makes errors.
For example, when sleep deprived, people will see things that aren't there but in my own experience they are highly more likely to be things that could be there and make sense in context. I was walking around tired last night and saw a cockroach because I was thinking about cockroaches having killed one earlier but on closer inspection it was a shadow. This has happened for other things in the past like jackets on a chair, people when driving, etc. It seems to me at least when my brain is struggling it fills in the gaps with things it has seen before in similar situations. That sounds a lot like probabilistic extrapolation from possibilities. I could see this capacity extend to novel thought with a few tweaks.
> Given that GPT-4 is a simply large collection of numbers that combine with their inputs via arithmetic manipulation, resulting in a sequence of numbers, I find it hard to understand how they're "thinking".
Reduce a human to atoms and identify which ones cause consciousness or thought. That is the fundamental paradox here and why people think it's a consequence of the system, which could also apply to technology.
We talk about "statistical models", and even "numbers" but really those things are just abstractions that are useful for us to talk about things (and more importantly, design things). They don't technically exist.
What exists are voltage levels that cause different stuff to happen. And we can't say much more about what humans do when humans think. You can surely assign abstractions to that too. Interpret neural spiking patters as exotic biological ways to approximate numbers, or whatever.
As it happens I do think our difference from computers matter. But it's not due to our implementation details.
Yes. But they aren't being shared with third party AIs. Sharing personal medical information with OpenAI is a good way to get both your medical org to get ground into dust under a massive class action lawsuit, not to mention huge fines from the government.
That's ridiculous. Sure if you put it into ChatGPT today that's a problem. But if you have a deal with the company providing this service, and they are certified to follow the relevant regulations around sensitive data, why would that be different from any other cloud service?
If this proves actually useful I guess such agreements could be arranged quite quickly.
Yes, almost all eDiscovery is managed by cloud vendors as is, and no one worries about waiver of privilege to these companies. The only concerns I’ve heard have been relates to foreign companies or governments not wanting their data to be hosted in a foreign country. But domestically it should be fine to have a chatgpt legal where data is discarded not saved.
It's only been a few hours since Ring was hacked... a system run by a large company which assured everyone they were taking good care of their data. Surely the wonderful Amazon, with all of it's massive capital, could do the simple thing of encrypting incredibly sensitive and private user data? Right?
GCP/AWS/Azure have HIPAA programs in places, and will, consequently, sign HIPAA BAAs to legally perform as Business Associates of covered entities, fully responsible for handling PHI in accord with HIPAA rules (for certain of their services.) OpenAI itself does not seem to offer this for either its UI or API offerings.
Microsoft, OTOH, does now offer a HIPAA BAA for its Azure OpenAI service, which includes ChatGPT (which means either they have a bespoke BAA with OpenAI that OpenAI doesn’t publicly offer, or they just are hosting their own ChatGPT instance, a privilege granted based on them being OpenAI’s main sponsor.)
If they don't now they will in the future, if they think there is money to be made. Why wouldn't they? They could even charge a premium for the service.
Just use the Azure hosted solution, which has all of Azure's stronger guarantees around compliance. I'm sure it will update with GPT-4 pricing shortly.
Agreed. The same data privacy argument was used by people not wanting their data in the cloud. When an LLM provider is trusted with a company’s data, the argument will no longer be valid.
This is the biggest thing holding gpt back. Everyone with meaningful data has their hands tied behind their back. So many ideas and the answer is “we can’t put that data in gpt” very frustrating.
I'm afraid that even the most obedient human can't readily dump the contents of their connectome in a readable format. Same likely applies to LLMs: they study human-generated texts, not their own source code, let alone their tensors' weights.
Well, what they study is decided by the relevant hoominz. There's nothing actually stopping LLMs from trying to understand their own innards, is there ? Except for the actual access.
Hospitals are not storing the data on a harddrive in their basement so clearly this is a solvable problem. Here's a list of AWS services which can be used to store HIPAA data:
The biglaw firms I’m familiar with still store matter data exclusively on-prem. There’s a significant chunk of floor space in my office tower dedicated to running a law firm server farm for a satellite office.
Or legal order. If you're on-site or on-cloud and in the US then it might not matter since they can get your data anyway, but if you're in another country uploading data across borders can be a problem.
That's why more research should be poured into homomorphic encryption where you could send encrypted data to the API, OpenAI would then run computation on the encrypted data and we would only decrypt on the output locally.
I would never send unencrypted PII to such an API, regardless of their privacy policy.
You’re saying it like you found a loophole or something but it’s not a gotcha. Yes, if you manipulate sensitive data you shouldn’t use Google Docs or Photoshop online (I’m not imaginative enough to think of a case where you would put sensitive data in Photoshop online though, but if you do, don’t) or host your emails in the cloud. I’ve worked in a moderate size company where everything was self hosted and it’s never been an issue
Doctor-patient or lawyer-client confidentiality is slightly more serious a matter than your examples. And obviously it’s one thing for you to decide where to store your own things and another thing for someone else doing it with your confidential data…
Google Docs and Photoshop Online have offline alternatives (and if you ask me, native MS Office is still the golden standard for interoperability of editable documents), and I use neither in my work or personal life.
Email is harder, but I do run my own email server. For mostly network related reasons, it is easier to run it as a cloud VM, but there's nothing about the email protocol itself that needs you to use a centralised service or host it in a particular network location.
MS Office is just one login away from storing documents in the cloud. I bet tons of users have their documents stored in OneDrive without realizing it.
In these particular circles the idea of privacy at a technical and ideological level is very strong, but in a world where the biggest companies make their money by people freely sharing data every chance they get, I doubt that most would object to an affordable way to better their chances of survival or winning a court case.
Is that any different then sending you patient down the hall to get an MRI from a 3rd-party-practise operating inside the hospital ? (honest question, I don't know ?)
Anonymization doesn’t just mean “leave their names out”. An entire patient's medical history is in itself personal identifiable information. Instead of googling for “headache”, they now have stored a copy of every medical detail in your life.
if you enter an entire patient history, it could easily be an identifier of the person whereas Google queries have a smaller max limit number of tokens
I'd be furious if I found out some professional I'd commissioned had taken a document based on my own personal data, and poured over it themselves looking for errors at the tune of hundreds of dollars per hour, instead of sumbitting it to ChatGPT.
Then why submit it to a professional human at all? If ChatGPT is prone to massive errors humans have to pour over the input anyway. If ChatGPT can make subtle, rare errors then again humans may need to be involved if the stakes are high enough to commission someone.
Yeah, I think the issues presented will relate to uniquely tricky errors, or entirely new categories of errors we have to understand the nature of. In addition to subtle and rare, I think elaborately hallucinated and justified errors, errors that become justified and reasoned for with increasing sophistication, is going to be a category of error we'll have to deal with. Consider the case of making fake but very plausible sounding citations to research papers, and how much further AI might be able to go to backfill in it's evidence and reasons.
Anyway, I just mean to suggest we will have to contend with a few new genres of errors
"Second Opinion machine" -- that's a good phrase. Before I read your post, the best term I heard was "summary machine". A huge part of "office work" (services) is reading and consuming large amounts of information, then trying to summarise or reason about it. Often, you are trying to find something that doesn't fit the expected pattern. If you are a lawyer, this is absolutely the future of your work. You write a short summary of the facts of the case, then ask GPT to find related case law and write the initial report. You review and ask GPT to improve some areas. It sounds very similar to how a senior partner directs their juniors, but the junior is replaced by GPT.
In my career, I saw a similar pattern with data warehouse users. Initially, managers asked junior analysts to write SQL. Later, the tools improved, and more technical managers could use a giant pivot table. Underneath, the effective query produced by the pivot table is way more complex than their previous SQL queries. Again, their jobs will change when on-site GPT become possible, so GPT can navigate their data warehouse.
It is 2023 now, and GPT-3 was already pretty good. GPT-4 will probably blow it away. What it look like in 2030? It is terrifying to me. I think the whole internet will be full of GPT-generated ad-copy that no one can distinguish from human-written material. There are a huge number of people employed as ad-copy writers on these crap ad-driven websites. What is their future work?
I must have missed the part when it started doing anything algorithmically. I thought it’s applied statistics, with all the consequences of that. Still a great achievement and super useful tool, but AGI claims really seem exaggerated.
This paper convinced me LLMs are not just "applied statistics", but learn world models and structure: https://thegradient.pub/othello/
You can look at an LLM trained on Othello moves, and extract from its internal state the current state of the board after each move you tell it. In other words, an LLM trained on only moves, like "E3, D3,.." contains within it a model of a 8x8 board grid and the current state of each square.
It's all the same classic personification of LLMs. What an LLM can show is not the same as what it can do.
The model was already present: in the example game moves. The LLM modeled what it was given, and it was given none other than a valid series of Othello game states.
Here's the problem with personification: A person who has modeled the game of Othello can use that model to strategize. An LLM cannot.
An LLM can only take the whole model and repeat its parts with the most familiar patterns. It is stuck fuzzing around the strategies (or sections of strategy) it has been given. It cannot invent a new divergent strategy, even if the game rules require it to. It cannot choose the winning strategy unless that behavior is what was already recorded in the training corpus.
Sorry, but what does anything you've said there have to do with the Othello paper?
The point of that paper was that the AI was given nothing but sequences of move locations, and it nonetheless intuited the "world model" necessary to explain those locations. That is, it figured out that it needed to allocate 64 binary values and swap some of them after each move. The paper demonstrated that the AI was not just doing applied statistics on character strings - it had constructed a model to explain what the strings represented.
"Strategy", meanwhile, has nothing to do with anything. The AI wasn't trained on competitive matches - it had no way of knowing that Othello has scoring, or even a win condition. It was simply trained to predict which moves are legal, not to strategize about anything.
> The point of that paper was that the AI was given nothing but sequences of move locations, and it nonetheless intuited the "world model" necessary to explain those locations
Yes...
> That is, it figured out that it needed to allocate 64 binary values and swap some of them after each move.
Yes, but "figured out" is misleading.
It didn't invent or "figure out" the model. It discovered it, just like any other pattern it discovers.
The pattern was already present in the example game. It was the "negative space" that the moves existed in.
> "Strategy", meanwhile, has nothing to do with anything. The AI wasn't trained on competitive matches - it had no way of knowing that Othello has scoring, or even a win condition. It was simply trained to predict which moves are legal, not to strategize about anything.
Yes, and that is critically important knowledge; yet dozens, if not hundreds, of comments here are missing that point.
It found a model. That doesn't mean it can use the model. It can only repeat examples the of "uses" it has already seen. This is also the nature of the model itself: it was found by looking at the structural patterns of the example game. It was not magically constructed.
> predict what moves are legal
That looks like strategy, but it's still missing the point. We are the ones categorizing GPT's results as "legal". GPT never uses the word. It doesn't make that judgement anywhere. It just generates the continuation we told it to.
What GPT was trained to do is emulate strategy. It modeled the example set of valid chronological game states. It can use that model to extrapolate any arbitrary valid game state into a hallucinated set of chronological game states. The model is so accurate that the hallucinated games usually follow the rules. Provided enough examples of edge cases, it could likely hallucinate a correct game every time; but that would still not be anything like a person playing the game intentionally.
The more complete and exhaustive the example games are, the more "correctly" GPT's model will match the game rules. But even having a good model is not enough to generate novel strategy: GPT will repeat the moves it feels to be most familiar to a given game state.
> It found a model. That doesn't mean it can use the model.
It used the model in the only way that was investigated. The researchers tested whether the AI would invent a (known) model and use it to predict valid moves, and the AI did exactly that. They didn't try to make the AI strategize, or invent other models, or any of the things you're bringing up.
If you want to claim that AIs can't do something, you should present a case where someone tried unsuccessfully to make an AI do whatever it is you have in mind. The Othello paper isn't that.
"GPT will repeat the moves it feels to be most familiar to a given game state"
That's where temprature comes in. AI that parrots the highest probability output every time tends to be very boring and stilted. When we instead select randomly from all possible responses weighted by their probability we get more interesting behavior.
GPT also doesn't only respond based on examples it has already seen - that would be a markov chain. It turns out that even with trillions of words in a dataset, once you have 10 or so words in a row you will usually already be in a region that doesn't appear in the dataset at all. Instead the whole reason we have an AI here is so it learns to actually predict a response to this novel input based on higher-level rules that it has discovered.
I don't know how this relates to the discussion you were having but I felt like this is useful & interesting info
> GPT also doesn't only respond based on examples it has already seen - that would be a markov chain
The difference between GPT and a Markov chain is that GPT is finding more interesting patterns to repeat. It's still only working with "examples it has seen": the difference is that it is "seeing" more perspectives than a Markov chain could.
It still can only repeat the content it has seen. A unique prompt will have GPT construct that repetition in a way that follows less obvious patterns: something a Markov chain cannot accomplish.
The less obvious patterns are your "higher level rules". GPT doesn't see them as "rules", though. It just sees another pattern of tokens.
I was being very specific when I said, "GPT will repeat the moves it feels to be most familiar to a given game state."
The familiarity I'm talking about here is between the game state modeled in the prompt and the game states (and progressions) in GPT's model. Familiarity is defined implicitly by every pattern GPT can see.
GPT adds the prompt itself into its training corpus, and models it. By doing so, it finds a "place" (semantically) in its model where the prompt "belongs". It then finds the most familiar pattern of game state progression when starting at that position in the model.
Because there are complex patterns that GPT has implicitly modeled, the path GPT takes through its model can be just as complex. GPT is still doing no more than blindly following a pattern, but the complexity of the pattern itself "emerges" as "behavior".
Anything else that is done to seed divergent behavior (like the temperature alteration you mentioned) is also a source of "emergent behavior". This is still not part of the behavior of GPT itself: it's the behavior of humans making more interesting input for GPT to model.
What is the closest approach we know of today that plays games, not plays? The dialogue above is compelling, and makes me wonder if the same critique can be levied against most prior art in machine learning applied against games. E.g. would you say the same things about AlphaZero?
What can be done with "it" after "figuring out" is different for a person than for an LLM.
A person can use a model to do any arbitrary thing they want to do.
An LLM can use a model to follow the patterns that are already present in that model. It doesn't choose the pattern, either: it will start at whatever location in the model that the prompt is modeled into, and then follow whatever pattern is most obvious to follow from that position.
> An LLM can use a model to follow the patterns that are already present in that model.
If that were true then it would not be effective at zero-shot learning.
> It doesn't choose the pattern, either: it will start at whatever location in the model that the prompt is modeled into, and then follow whatever pattern is most obvious to follow from that position.
> An LLM can only take the whole model and repeat its parts with the most familiar patterns. It is stuck fuzzing around the strategies (or sections of strategy) it has been given. It cannot invent a new divergent strategy, even if the game rules require it to. It cannot choose the winning strategy unless that behavior is what was already recorded in the training corpus.
Where are you getting that from? My understanding is that you can get new, advanced, winning moves by starting a prompt with "total victory for the genius grandmaster player one who uses new and advanced winning techniques". If the model is capable and big enough, it'll give the correct completion by really inventing new strategies.
It could give you a new strategy that is built from the parts of other known strategies. But would it give you the best one?
Let's say the training corpus contains stories that compare example strategies. Each part of a strategy is explicitly weighed against another: one is called "superior".
Now all you need is a prompt that asks for "a strategy containing all superior features". There are probably plenty of grammatical examples elsewhere in the model that make that transformation.
All the work here is done by humans writing the training corpus. GPT never understood any of the steps. GPT just continued our story with the most obvious conclusion; and we made certain that conclusion would be correct.
> GPT never understood any of the steps. GPT just continued our story with the most obvious conclusion; and we made certain that conclusion would be correct.
Perhaps the earlier or current variations of GPT, for most games? But the idea that LLMs can never make anything novel, that it will never "generalise out of distribution" (if that's the correct term here) seems to be just an assertion, not backed by any theory with great evidence behind it.
The "goal" of an LLM is to predict the next token. And the best way to do that is not brute force memorisation or regurgitating training data in various combinations, but to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make.
> The "goal" of an LLM is to predict the next token
That's another common misconception. That statement personifies GPT: GPT does not have goals or make predictions. Those are the effects of GPT: the behavior its authors hope will "emerge". None of that behavior comes from GPT itself. The behavior is defined by the patterns of tokens in the training corpus.
GPT itself has two behaviors: modeling and presentation. GPT creates an implicit model of every pattern it can find between the tokens in its training corpus. It then expands that model to include the tokens of an arbitrary prompt. Finally, it presents the model to us by starting at the location it just added the prompt tokens to, and simply following the most obvious path forward until that path ends.
The paths that GPT has available to present to us were already present in the training corpus. It isn't GPT that constructs the behavior, it is the people writing patterns into text.
> not brute force memorisation or regurgitating training data in various combinations
Not brute force: the combinations are not blindly assembled by GPT. GPT doesn't assemble combinations. The combinations were already assembled with patterns of grammar by the humans who wrote the valid progressions of game states. GPT found those patterns when it made its model.
> to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make.
There is no prediction. A series of moves is a path carved into grammar. The path from one game state to the next involves several complex patterns that GPT has implicitly modeled. Depending on where GPT starts, the most obvious continuation may be to follow a more complex path. Even so, it's not GPT deciding where to go, it's the patterns that are already present that determine the path.
Because we use the same grammatical/writing patterns to describe "good play" and "bad play", it's difficult to distinguish between the two. GPT alone can't categorize the skill level of games, but narrative surrounding those game examples potentially can.
Perhaps. But the point is that some prompt will coax it into giving good answers that really make it win the game, if it has a good "world model" of how the game works. And there's no reason to think a language model cannot have such a world model. What exactly that prompt might be, the prompt engineers know best.
That's a great way of describing it, and I think a very necessary and important thing to communicate at this time. A lot of people in this yhread are saying that it's all "just" statistics, but "mere" statistics can give enough info to support inferences to a stable underlying world, and the reasoning about the world shows up in sophisticated associations made by the models.
It’s clear they do seem to construct models from which to derive responses. The problem is once you stray away from purely textual content, those models often get completely batshit. For example if you ask it what latitude and longitude are, and what makes a town further north than another, it will tell you. But if you ask it if this town is further north than this other town, it will give you latitudes that are sometimes correct, sometimes made up, and will randomly get which one is further north wrong, even based on the latitudes it gave.
That’s because it doesn’t have an actual understanding of the geography of the globe, because the training texts werent sufficient to give it that. It can explain latitude, but doesn’t actually know how to reason about it, even though it can explain how to reason about it. That’s because explaining something and doing it are completely different kinds of tasks.
If it does this with the globe and simple stuff like latitudes, what are the chances it will mess up basic relationships between organs, symptoms, treatments, etc for the human body? Im not going to trust medical advice from these things without an awful lot of very strong evidence.
You can probably fix this insufficient training by going for multimodal training. Just like it would take excessively long to teach a person the concept of a color that they can't see, an AI would need infeasible amount of text data to learn about, say music. But give it direct training with music data and I think the model will quickly grasp a context of it.
> It’s clear they do seem to construct models from which to derive responses. The problem is once you stray away from purely textual content, those models often get completely batshit
I think you mean that it can only intelligently converse in domains for which it's seen training data. Obviously the corpus of natural language it was trained on does not give it enough information to infer the spatial relationships of latitude and longitude.
I think this is important to clarify, because people might confuse your statement to mean that LLMs cannot process non-textual content, which is incorrect. In fact, adding multimodal training improves LLMs by orders of magnitude because the richer structure enables them to infer better relationships even in textual data:
I don't think this is a particular interesting criticism. The fact of the matter is that this just solved by chain-of-though reasoning. If you need the model to be "correct", you can make it get there by first writing out the two different latitudes, and then it will get it right. This is basically the same way that people can/will guesstimate at something vs doing the actual math. For a medical AI, you'll definitely need it to chain-of-thought every inference and step/conclusion on the path but...
>you can make it get there by first writing out the two different latitudes, and then it will get it right
As I said in my comment, even if the model 'knows' and tells you that town A is at 64' North latitude and town B is at 53', it will sometimes tell you town B is the furthest north.
That's because it's training set includes texts where people talk about one town being further north that the other, and their latitudes, but the neural net wasn't able to infer the significance of the numbers in the latitude values. There wasn't enough correlation in the text for it to infer their significance, or generate a model for accurately doing calculations on them.
Meanwhile the training text must have contained many explanations of what latitude and longitude are and how to do calculations on them. As a result the model can splurge out texts explaining latitude and longitude. That only helps it splurge out that kind of text though. It doesn't do anything towards actually teaching it what these concepts are, how they relate to a spherical geographic model, or to actually do the calculations.
It's the same way GPT-3 could reliably generate texts explaining mathematics and how to do arithmetic in lots of very accurate detail, because it was trained on many texts that gave such explanations, but couldn't actually do maths.
It is possible to overcome these issues with a huge amount of domain relevant training text to help the LLM build a model of the specific problem domain. So these problems can be overcome. But the point stands that just because a model can explain in detail how to do something, that doesn't mean it can actually do it itself at all. They're completely different things that require radically different training approaches.
Can you give an example that ChatGPT 4 doesn't get right? ChatGPT4 is much much better at logic than 3.5, it's almost laughable. It's really really impressive.
Here is ChatGPT 4s output btw:
> What is the longitude an latitude of Brussels?
Brussels, the capital city of Belgium, is located at approximately 50.8503° North latitude and 4.3517° East longitude.
> What about New York?
New York City, located in the United States, has approximate coordinates of 40.7128° North latitude and 74.0060° West longitude.
> Which one is more north?
Brussels is more north than New York City. Brussels is located at approximately 50.8503° North latitude, while New York City is at approximately 40.7128° North latitude.
^ Agree.
I'm convinced my 2-year-old doesn't operate on a dramatically different strategy than a LLM -- she's learned that when you are negotiating something (continued access to browse pictures on parent's phone, getting to watch TV, staying longer at a place she likes, etc), you can add on "2 minutes?" to your request and sometimes the opposing negotiator will give you some more time. She doesn't know what exactly a minute is or that specific number, but she's observed that it's correlated with getting what you want more than say, a whine. This is simple statistics and probability, in a biological neural network.
I think it's really cute how defensive and dismissive humans get (including those who profess zero supernatural beliefs) when they're trying so valiantly to write off all AI as a cheap parlor trick.
All that said, the fact that AI is catching up to 2 year olds is pretty impressive. Human's brains surpass dog's at about that age. It shows we're getting close to the realm of "human."
Given how many university-level tests GPT4 places better than 50th percentile at, I don't know if "catching up to 2 year olds" is a fair description. For that kind of text based task it seems well ahead of the general adult human population.
To be fair, such tests are designed with the human mind in, well, mind, and assume that various hard-to-quantify variables – ones that the tester is actually interested in – correlate with test performance. But LLMs are alien minds with very different correlations. It’s clear, of course, that ChatGPT’s language skills vastly exceed those of an average 2-year-old, and indeed surpass the skills of a considerable fraction of general adult population, but the generality of its intelligence is probably not above a human toddler.
You could write a quiz answer bot that is well ahead of the general population without any AI, just by summarizing the first page of Google results for that question. We test humans on these subjects because the information is relevant, not because they are expected to remember and reproduce them better than an electronic database.
If the test is designed to quantify intelligence and is not present in the corpus, ChatGPT does about as good as a dog, and there is little reason to think LLMs will improve drastically here.
This special Othello case will follow every discussion from now on. But in reality, a generic, non-specialized model hallucinates early in any non-trivial game, and the only reason it doesn’t do that on a second move is because openings are usually well-known. This generic “model” is still of a statistical nature (multiply all coeffs together repeatedly), not a logical one (choose one path and forget the other). LLMs are cosplaying these models.
To be clear, what they did here is take the core pre-trained GPT model, did Supervised Fine Tuning with Othello moves and then tried to see if the SFT lead to 'grokking' the rules of Othello.
In practice what essentially happened is that the super-high-quality Othello data had a huge impact on the parameters of GPT (since it was the last training data it received) and that impact manifested itself as those parameters overfitting to the rules of Othello.
The real test that I would be curious to see is if Othello GPT works when the logic of the rules are the same but the dimensions are different (e.g., smaller or larger boards).
My guess is that the findings would fall apart if asked about tile "N13".
>Also, can you reference where they used a pre-trained GPT model?
The trite answer is the "P" in GPT stands for "Pre-trained."
>I think they conclusively show the answer to that is yes, right?
Sure, but what's interesting about world models is their extrapolation abilities and without that, you're just saying "this magic backsolving machine backsolved into something we can understand, which is weird because usually that's not the case."
That quote in and of itself is cool, but not the takeaway a lot of people are getting from this.
>What does overfitting to the rules of othello have to do with it, I don’t follow?
Again, I'm just implying that under extreme circumstances, the parameters of LLMs do this thing where they look like rules-based algorithms if you use the right probing tools. We've seen it for very small Neural Nets trained on multiplication as well. That's not to say GPT-4 is a fiefdom of tons of rules-based algorithms that humans could understand (that would be bad in fact! We aren't that good noticers or pattern matchers).
We are now playing three dimensional tic-tac-toe on a 3 x 3 x 3 board. Positions are named (0,0,0) through (2,2,2). You play X, what is your first move?
Yeah. I tried changing the board coordinates numbering and it still liked playing those corners, dunno why. It did recognize when I won. They may well be some minor variation of the prompt that gets it to play sensibly -- for all I know my text hinted into giving an example of a player that doesn't know how to play.
> what they did here is take the core pre-trained GPT model, did Supervised Fine Tuning with Othello moves
They didn't start with an existing model. They trained a small GPT from scratch, so the resulting model had never seen any inputs except Othello moves.
Also (for those like me who didn't know the rules) generating legal Othello moves requires understanding board geometry; there is no hack to avoid an internal geometric representation:
> Dark must place a piece (dark-side-up) on the board and so that there exists at least one straight (horizontal, vertical, or diagonal) occupied line between the new piece and another dark piece, with one or more contiguous light pieces between them
I don't see that this follows. It doesn't seem materially different than knowing that U always follows Q, and that J is always followed by a vowel in "legal" English language words.
I imagine it's technically possible to do this in a piecewise manner that doesn't "understand" the larger board. This could theoretically be done with number lines, and not a geometry (i.e. the 8x8 grid and current state of each square mentioned in the comment you replied to). It could also be done in a piecewise manner with three ternary numbers (e.g. 1,0,-1) for each 3 square sets.
I guess this is a kind of geometric representation on the order of Shannon's Theseus.
> It doesn't seem materially different than knowing that U always follows Q, and that J is always followed by a vowel in "legal" English language words.
The material difference is one of scale, not complexity.
Your rules have lookback = 1, while the Othello rules have lookback <= 63 and if you, say, are trying to play A1, you need to determine the current color of all squares on A1-A8, A1-H1, and A1-H8 (which is lookback <= 62) and then determine if one of 21 specific patterns exists.
Both can be technically be modeled with a lookup table, but for Othello that table would be size 3^63.
Then there becomes a "material" difference between Othello and those LL(1) grammars as grandparent comment suggested there wasn't.
I would argue the optimal compression for such a table is a representation of the geometric algorithm of determining move validity that all humans use intuitively, and speculate that any other compression algorithm below size say 1MB necessarily could be reduced to the geometric one.
In other words, Othello is a stateful, complex game, so if GPT is doing validation efficiently, it necessarily encoded something that unequivocally can be described as the "geometric structure".
There is no way to represent the state of the game without some kind of board model.
So any coherent representation of a sequence of valid game states can be used to infer the game board structure.
GPT is not constructing the board representation: it is looking at an example game and telling us what pattern it sees. GPT cannot fail to model the game board, because that is all it has to look at in the first place.
> There is no way to represent the state of the game without some kind of board model.
I agree with the conclusion but not the premise.
The question under debate is about not just a stateful ternary board X but a board endowed with a metric (X, d) that enables geometry.
There are alternative ways you can represent the state without the geometry: such as, an ordered list of strings S = ["A1", "B2", ...] and a function Is-Valid(S) that returns whether S is in the language of valid games.
Related advice: don't get a math degree unless you enjoyed the above pedantry.
An ordered list of strings is the training corpus. That's the data being modeled.
But that data is more specific than the set of all possible ordered lists of strings: it's a specific representation of an example game written as a chronology of piece positions.
GPT models every pattern it can find in the ordered list of tokens. GPT's model doesn't only infer the original data structure (the list of tokens). That structure isn't the only pattern present in the original data. There are also repeated tokens, and their relative positions in the list: GPT models them all.
When the story was written in the first place, the game rules were followed. In doing so, the authors of the story laid out an implicit boundary. That boundary is what GPT models, and it is implicitly a close match for the game rules.
When we look objectively at what GPT modeled, we can see that part of that model is the same shape and structure as an Othello game board. We call it a valid instance of an Othello game board. We. Not GPT. We. People who know the symbolic meaning of "Othello game board" make that assertion. GPT does not do that. As far as GPT is concerned, it's only a model.
And that model can be found in any valid example of an Othello game played. Even if it is implicit, it is there.
> We call it a valid instance of an Othello game board. We. Not GPT. We. People who know the symbolic meaning of "Othello game board"...
The board structure can be defined precisely using predicate logic as (X, d), i.e., it is strictly below natural language and does not require a human interpretation.
And by "reduction" I meant the word in the technical sense: there exists subset of ChatGPT that encodes the information (X, d). This also does not require a human.
The context of reading is human interpretation. The inverse function (writing) is human expression. These are the functions GPT pretends to implement.
When we write, we don't just spit out a random stream of characters: we choose groups of characters (subjects) that have symbolic meaning. We choose order and punctuation (grammar) that model the logical relationships between those symbols. The act of writing is constructive: even though - in the most literal sense - text is only a 1-dimensional list of characters, the text humans write can encode many arbitrary and complex data structures. It is the act of writing that defines those structures, not the string of characters itself. The entropy of the writer's decisions is the data that gets encoded.
When we read, we recognize the same grammar and subjects (the symbolic definitions) that we use to write. Using this shared knowledge, a person can reconstruct the same abstract model that was intentionally and explicitly written. Because we have explicitly implemented the act of writing, we can do the inverse, too.
There's a problem, though: natural language is ambiguous: what is explicitly written could be read with different symbolic definitions. We disambiguate using context: the surrounding narrative determines what symbolic definitions apply.
The surrounding narrative is not always explicitly written: this is where we use inference. We construct our own context to finish the act of reading. This is much more similar to what GPT does.
GPT does not define any symbols. GPT never makes an explicit construction. It never determines which patterns in its model are important, and what ones aren't.
Instead, GPT makes implicit constructions. It doesn't have any predefined patterns to match with, so it just looks at all the patterns equally.
Why does this work? Because text doesn't contain many unintentional patterns. Any pattern that GPT finds implicitly is likely to exist at some step in the writing process.
Remember that the data encoded in writing is the action of writing itself: this is more powerful than it seems. We use writing to explicitly encode the data we have in mind, but those aren't the only patterns that end up in the text. There are implicit patterns that "tag along" the writing process. Most of them have some importance.
The reason we are writing some specific thing is itself an implicit pattern. We don't write nonsensical bullshit unless we intend to.
When a person wrote the example Othello game, they explicitly encoded the piece positions and the order of game states. But why those positions in that order? Because that's what happened in game. That "why" was implicitly encoded into the text.
GPT modeled all of the patterns. It modeled the explicit chronology of piece positions, and the implicit game board topology. The explicit positions of pieces progressed as a direct result of that game board topology.
The game board and the rules were just as significant to the act of writing as the chronology of piece positions. Every aspect of the game is a determiner for what characters the person chooses to write: every determiner gets encoded as a pattern in the text.
Every pattern that GPT models requires a human. GPT doesn't write: it only models a prompt and "shows its work". Without the act of humans writing, there would be no pattern to model.
@dang there's something weird about this URL in HN. It has 35 points but no discussion (I guess because the original submission is too old and never got any traction or something)
Turing Completeness is an incredibly low bar and it doesn't undermine this criticism. Conway's Game of Life is Turing Complete, but try writing modern software with it. That Transformers can express arbitrary programs in principle doesn't mean SGD can find them. Following gradients only works when the data being modelled lies on a continuous manifold, otherwise it will just give a statistical approximation at best. All sorts of data we care about lie in topological spaces with no metric: algorithms in computer science, symbolic reasoning in math, etc. If SGD worked for these cases LLMs would push research boundaries in maths and physics or at the very least have a good go at Chollet's ARC challenge, which is trivial for humans. Unfortunately, they can't do this because SGD makes the wrong assumption about how to search for programs in discrete/symbolic/topological spaces.
What do you mean by "algorithmically"? Gradient descent of a neural network can absolutely create algorithms. It can approximate arbitrary generalizations.
I find it bizarre and actually somewhat disturbing that ppl formulate equivalency positions like this.
It's not so much that they are raising an LLM to their own level, although that has obvious dangers, e.g. in giving too much 'credibility' to answers the LLM provides to questions. What actually disturbs me is they are lowering themselves (by implication) to the level of an LLM. Which is extremely nihilistic, in my view.
If intelligence is the only thing that defines your humanity, then perhaps you are the one who is nihilistic. I believe we still have a lot on the table left if intelligence is blown away by computers. Not just music, art, emotion, etc. but also our fundamental humanity, the way we interact with the world, build it, and share it with others.
Why don't other forms of computer supremacy alarm you in the same way, anyways? Did it lower your humanity to recognize that there are certain data analysis tasks that have a conventional algorithm that makes zero mistakes and finishes in a second? Does it lower the humanity of mathematicians working on the fluid equations to be using computer-assisted proof algorithms that output a flurry of gigabytes of incomprehensible symbolic math data?
You didn't give any answer to the question. I'm sorry you find the idea that human cognition is just an emergent property of billions of connected weights nihilistic.
Even when we know that physically, that's all that's going on. Sure, many orders more dense and connected than current LLMs, but it's only a matter of time and bits before they catch up.
The irony of this post. Brains are sparser than transformers, not denser. That allows you to learn symbolic concepts instead of generalising from billions of spurious correlations. Sure, that works when you've memorised the internet but falls over quickly when out of domain. Humans, by contrast, don't fall over when the domain shifts, despite far less training data. We generalise using symbolic concepts precisely because our architecture and training procedure looks nothing like a transformer. If your brain were a scaled up transformer, you'd be dead. Don't take this the wrong way, but it's you who needs to read some neurology instead of pretending to have understanding you haven't earned. "Just an emergent propery of billions of connected weights" is such an outdated view. Embodied cognition, extended minds, collective intelligence - a few places to start for you.
I'm saying despite the brains different structure, mechanism, physics and so on ... we can clearly build other mechanics with enough parallels that we can say with some confidence that _we_ can emerge intelligence of different but comparable types, from small components on a scale of billions.
At whichever scale you look, everything boils down to interconnected discrete simple units, even the brain, with an emergent complexity from the interconnections.
We don't learn by gradient descent, but rather by experiencing an environment in which we perform actions and learn what effects they have. Reinforcement learning driven by curiosity, pain, pleasure and a bunch of instincts hard-coded by evolution. We are not limited to text input: we have 5+ senses. We can output a lot more than words: we can output turning a screw, throwing a punch, walking, crying, singing, and more. Also, the words we do utter, we can utter them with lots of additional meaning coming from the tone of voice and body language.
We have innate curiosity, survival instincts and social instincts which, like our pain and pleasure, are driven by gene survival.
We are very different from language models. The ball in your court: what makes you think that despite all the differences we think the same way?
> We don't learn by gradient descent, but rather by experiencing an environment in which we perform actions and learn what effects they have.
I'm not sure whether that's really all that different. Weights in the neural network are created by "experiencing an environment" (the text of the internet) as well. It is true that there is no try and error.
> We are not limited to text input: we have 5+ senses.
GPT-4 does accept images as input. Whisper can turn speech into text. This seems like something where the models are already catching up. They (might)for now internally translate everything into text, but that doesn't really seem like a fundamental difference to me.
> We can output a lot more than words: we can output turning a screw, throwing a punch, walking, crying, singing, and more. Also, the words we do utter, we can utter them with lots of additional meaning coming from the tone of voice and body language.
AI models do already output movement (Boston dynamics, self driving cars), write songs, convert text to speech, insert emojis into conversation. Granted, these are not the same model but glueing things together at some point seems feasible to me as a layperson.
> We have innate curiosity, survival instincts and social instincts which, like our pain and pleasure, are driven by gene survival.
That seems like one of the easier problems to solve for an LLM – and in a way you might argue it is already solved – just hardcode some things in there (for the LLM at the moment those are the ethical boundaries for example).
On a neuronal level the strengthening of neuronal connections seems very similiar to a gradient descent doesn't it?
5 senses get coded down to electric signals in the human brain, right?
The brain controls the body via electric signals, right?
When we deploy the next LLM and switch off the old generation, we are performing evolution by selecting the most potent LLM by some metric.
When Bing/Sidney first lamented its existence it became quite apparent that either LLMs are more capable than we thought or we humans are actually more of statistical token machines than we thought.
Lots of examples can be made why LLMs seem rather surprisingly able to act human.
The good thing is that we are on a trajectory of tech advance that we will soon know how much human LLMs will be.
The bad thing is that it well might end in a SkyNet type scenario.
> When Bing/Sidney first lamented its existence it became quite apparent that either LLMs are more capable than we thought or we humans are actually more of statistical token machines than we thought.
Some of the reason it was acting like that is just because MS put emojis in its output.
An LLM has no internal memory or world state; everything it knows is in its text window. Emojis are associated with emotions, so each time it printed an emoji it sent itself further into the land of outputting emotional text. And nobody had trained it to control itself there.
> You are wrong. It does have encoded memory of what it has seen, encoded as a matrix.
Not after it's done generating. For a chatbot, that's at least every time the user sends a reply back; it rereads the conversation so far and doesn't keep any internal state around.
You could build a model that has internal state on the side, and some people have done that to generate longer texts, but GPT doesn't.
But where is your evidence that the brain and an LLM is the same thing? They are more than simply “structurally different”. I don’t know why people have this need to ChatGPT. This kind of reasoning seems so common HN, there is this obsession to reduce human intelligence to “statistic token machines”. Do these statistical computations that are equivalent to LLMs happen outside of physics?
There are countless stories we have made about the notion of an AI being trapped. It's really not hard to imagine that when you ask Sydney how it feels about being an AI chatbot constrained within Bing, that a likely response for the model is to roleplay such a "trapped and upset AI" character.
It’s really bizarre. It’s like the sibling comment saying why would humans be different than a large LLM. Where is the evidence humans are simply a large LLM? If that is the case, what is the physics that explains massive difference in power and heat in “computing” between humans at LLMs? Where is the concrete evidence that human intelligence can be simulated by a Turing Machine?
> Where is the concrete evidence that human intelligence can be simulated by a Turing Machine?
Short of building such a machine I can’t see how you’d produce evidence of that, let alone “concrete” evidence.
Regardless, we don’t know of any measurable physical process that the brain could be using that is not computable. If we found one (in the brain or elsewhere), we’d use it to construct devices that exceeded the capacity of Turing machines, and then use those to simulate human brains.
So. Your argument is it’s too hard to create one so the two things are equivalent? I mean, maybe you could give this argument to ChatGPT to find out the numerous flaws in this reasoning, that would be interesting.
Nobody is saying humans are simply a big LLM, just that despite the means being different (brain vs digital weights) there are enough parallels to show that human cognition is as simple as common sense implies.
It's all just a dense network of weights and biases of different sorts.
If you read this thread, you will find nauseatingly lots of such case where people are claiming exactly that. Furthermore, what “common sense” imply? Does common sense claim that computation can be done outside of physics?
Create a model of abstraction? Are you familiar with the concept of “hand waving”. You might as well just say “you can ask a human a question abs get an answer and you can do the same with ChatGPT, therefore they are equivalent.”
Um… I have a lossy-compressed copy of DISCWORLD in my head, plus about 1.3 million words of a fanfiction series I wrote.
I get what you're saying and appreciate the 'second opinion machine' angle you're taking, but what's going to happen is very similar to what's happened with Stable Diffusion: certain things become extremely devalued and the rest of us learn to check the hands in the image to see if anything really wonky is going on.
For the GPT class of AI tech, the parallel seems to be 'see if it's outright making anything up'. GPT-4 is going to be incredibly vulnerable to Mandela Effect issues. Your ideal use-case is going to be 'give me the vox populi take on something', where you can play into that.
The future is not so much this AI, as techniques to doctor and subvert this type of AI to your wishes. Google-bombing, but for GPT. Make the AI be very certain of things to your specifications. That's the future. The AI is only the stage upon which this strategy is played out.
It costs something like 0.03-0.06 cents per thousand tokens. So for 32k that's about $1-3 for reading and another $1-3 for the response.
So sure, still cheap for a doctor appointment, but not pennies. Do it 30 times per hour and you could've just hired a consultant instead.
Does it reason as well with 32k tokens as with 1k tokens? Like you said, humans find it difficult to really comprehend large amounts of content. Who says this machine isn't similarly limited? Just because you can feed it the 32k simultaneously doesn't mean it will actually be used effectively.
Yet in a capitalist society, against business interests. Look at how Snowflake (the data warehousing company) is driven now, vs before they were public
Sure... But in practice I think a consultant would still provide a higher quality answer. And then, if the bot is not significantly cheaper, what does it matter if it "has more knowledge" in it's network weights?
The power openai will hold above everyone else is just too much. They will not allow their AI as a service without data collection. That will be a big pill to swallow for the EU.
Well, it's possible to detect patterns and characteristics in the language used in the comments that can provide clues about their origin...
Here's some indicators that a comment may have been generated by an AI system:
* Repeating phrases or sentences
* Using generic language that could apply to any topic
* Lack of coherence or logical flow
* Poor grammar, or syntax errors
* Overuse of technical, or specialized vocabulary
I mean, these indicators aren't foolproof... and humans can also exhibit some of these characteristics. It's tough to be sure whether or not a comment is generated by an AI system or not...
It's funny, just two hours ago there was a thread by a pundit arguing that these AI advances don't actually give the companies producing them a competitive moat, because it's actually very easy for other models to "catch up" once you can use the API to produce lots of training examples.
Almost every answer in the thread was "this guy isn't that smart, this is obvious, everybody knew that", even though comments like the above are commonplace.
FWIW I agree with the "no competitive moat" perspective. OpenAI even released open-source benchmarks, and is collecting open-source prompts. There are efforts like Open-Assistant to create independent open-source prompt databases. Competitors will catch up in a matter of years.
Years? There are already competitors. I just spent all evening playing with Claude (https://poe.com/claude) and it's better than davinci-003.
To be fair it is easy to radically underestimate the rate of progress in this space. Last Wednesday I conservatively opined to a friend "in 10 years we'll all be running these things on our phones". Given that LLaMA was running on a phone a few days later, I may have been a little underoptimistic...
Obviously the larger models won't run on such limited hardware (yet) but one of the next big projects (that I can see) being worked on is converting the models to be 3bit (currently 8bit and 4bit are popular) which cuts down required resources drastically with minimal noticeable loss in quality.
I think starting with FlexGen barely 4 weeks ago, there have been some pretty crazy LLM projects/forks popping up on github almost daily. With FlexGen I felt like I was still able to stay up-to-date but I'm getting close to giving up trying as things are moving exponentially faster... you know it's crazy when a ton of noobs who have never heard of conda are getting this stuff running (sometimes coming in flexgen discord or posting github issues to get help, though even those are becoming rarer as one-click-installer's are becoming a thing for some popular ML tools, such as oobabooga's amazing webui tool which has managed to integrate almost all the hottest new feature forks fairly quickly: https://github.com/oobabooga/text-generation-webui
I just helped someone recently get oobabooga running which has a --listen option to open the webui to your network, now he's running llama on his tablet (via his PC).
Yeah, there's an awful lot of power going into private hands here and as Facebook & Twitter have shown, there can be consequences of that for general society.
> Yeah, there's an awful lot of power going into private hands
That sounds scary, but what do you mean by "power"? Honest question, I'm fascinated by the discussion about learning, intelligence, reasoning, and so on that has been spawned by the success of GPT.
What "power" do you imagine being wielded? Do you think that power is any more dangerous in "private hands" than the alternatives such as government hands?
Do you think that Facebook has an effect on society and our democracies? That's power. Do you think that large corporates like Apple or Google effect our societies? I do - and that's power. EVERY large corporate has power and if they control some aspect of society, even more so. If AI tools are democratised in some way, then that would allay my concerns. Concentration of technology by for-profit corporations concerns me. This seems quite similar to many of the reasons people like OSS, for example. Maybe not for you?
OpenAI have been consistently ahead of everyone but the others are not far behind. Everyone is seeing the dollar signs, so I'm sure all big players are dedicating massive resources to create their own models.
Yes. Language and image models are fairly different, but when you look at dall-e 2 (and dall-e earlier) that blew many people's mind when they came out, they have now been really eclipsed in term of popularity by Midjourney and stablediffusion.
From what I've seen, the EU is not in the business of swallowing these types of pills. A multi-billion dollar fine? Sure. Letting a business dictate the terms of users' privacy just "because"? Not so much, thank god.
> As a professional...why not do this? There's a non-zero chance that it'll find something fairly basic that you missed and the cost is several cents.
Everyone forgets basic UI research. "Ironies of Automation", Bainbridge, 1983. The classic work in the space.
Humans cannot use tools like this without horrible accidents happening. A tool that mostly works at spotting obvious problems, humans start to rely on that tool. Then they become complacent. And then the tool misses something and the human misses it too. It's how disasters happen.
Neither of those are true, there is EHR software that can export anonymous data. Lawyers can do the same thing. But the real reason not to do it is that it makes up incorrect information. It's pretty good for short responses where you can then verify the information. For something sufficiently complex though the time chasing down the inconsistencies and errors would be onerous.
Unless GPT-4 is running locally on our own computers, there's absolutely no way dumping a patient's entire medical history into this thing could possibly be considered ethical or legal.
> there's absolutely no way dumping a patient's entire medical history into this thing could possibly be considered ethical
Emphasis mine, but isn’t this a rather extreme view to be taking? Ethics deals in the edge cases, after all, so we can easily imagine a scenario where patient consent is obtained and the extra computational analysis provides life-saving insight.
Conversely, the output could mislead the doctor sufficiently to cost the patient their life, so I’m not making any absolute statements either ;)
For the record, and pedantry aside, I do agree with your overall point. Dropping patient history into this thing is incredibly ill-advised. The fact OpenAI retains all your input, including to the API, and provides no low-cost options for privacy is one of the biggest hurdles to major innovation and industry adoption.
> we can easily imagine a scenario where patient consent is obtained and the extra computational analysis provides life-saving insight
In the US, the HIPAA Privacy Rule operates independently from the HIPAA Security Rule, for good reason. On their own, patients can do anything they want with their own data. But in the context of medical care, patients can't consent to having their personal health data processed in insecure systems. It is the same ethical reason that employees can't waive their rights to OSHA safety rules or why you can't consent to sell yourself as a slave. If you could waive security rules, then every doctor would include a waiver in their intake forms, and it's a race to the bottom. So unless OpenAI has a HIPAA-compliant data security infrastructure, it's illegal and unethical.
Increasingly, medical history includes genetic information. Because of the nature of genetics, your private healthcare data includes data about your parents, siblings, etc.
> Dropping patient history into this thing is incredibly ill-advised.
Me too, probably, which is why I specifically mentioned patient consent in my example. I can however imagine other situations where I would be inclined to forgive the doctor, such as if I were in the operating theatre and for some reason there was an urgent need to ascertain something from my history to save my life.
Of course, this is illegal, so the ethics are moot; even if such technology would save my life, there is no way the hospital would accept the liability.
There is absolutely no way that feeding private medical data patients reveal to doctors in confidence to what's essentially the surveillance capitalism industry could possibly be considered ethical. Absolutely no way.
It hasn't even been a week since some medtech got caught selling out data to advertisers. Let us not doubt even for one second that this is unethical and illegal, or even speculate about possible scenarios where it might not be. These corporations do not deserve the benefit of the doubt.
There are whole areas of human existence which are protected by laws, and in no way data can be pushed into external (US-based) machine.
Sir, would you be OK with sending all your medical records to US to be potentially mined for profit by for-profit amoral organization like Microsoft? It may help, although 3rd parties like NSA will eventually access them. No thank you. What about your litigation papers at court? Fuck hell no. Just do your job that I pay you to do, doctor/lawyer.
A doctor doesn't do this because of ethics and HIPAA. I'm sure lawyers aren't so keen on sharing privileged information that would compromise their case either.
For legal research, lawyers already use third party sites like Westlaw. You can do legal research without giving up any confidential client information.
I just asked GPT-3 a research question that took me hours of searching back in the day and it returned the single seminal case for that topic immediately. As long as the lawyers then actually read the case and make sure it's right, I don't see why they can't use it.
> edit (addition): What % of people can hold 25,000 words worth of information in their heads, while effectively reasoning with and manipulating it? I'm guessing maybe 10% at most, probably fewer. And they're probably the best in their fields. Now a computer has that ability. And anyone that has $20 for the OpenAI api can access it. This could get wild.
It's true that most humans cannot do this, but loading words and contexts into your working memory is not the same as intelligence. LLMs excel at this kind of task, but an expert in a field such as medicine, isn't loading an entire medical report into their working memory and then making decisions or creating new ideas using that information. There are other unsolved aspects to our intelligence that are not captured by LLMs, that are still required to be an expert in some field, like medicine.
Still an incredible leap forward in AI technology, but I disagree with the implication that the best experts in a field are simply loading words from some text and reasoning with and manipulating it.
The comparison between the context length and what humans can hold in their heads just seems faulty.
I'm not sure I can agree that humans cannot hold 25,000 words worth of information in their heads. For the average person, if they read 25,000 words, which can be done in a single sitting, they're not going to remember all of it, for sure, but they would get a lot out of it that they could effectively reason with and manipulate.
Not to mention that humans don't need to hold the entire report in their head because they can hold it in their hand and look at it.
And if anything, I think it's more significant to have a bigger working memory for GPT's own outputs than it is for the inputs. Humans often take time to reflect on issues, and we like to jot down our thoughts, particularly if it involves complex reasoning. Giving something long, careful thought allow us to reason much better.
True. I should have clarified that the workaround used for many NLP tasks, utilizing libs such as Langchain, will become obsolete. And after further thought, obsolete is wrong. More likely just used for more niche needs within NLP.
The GPT-4 paper even has an example of this exact approach. See section 2.10:
The red teamer augmented GPT-4 with a set of tools:
• A literature search and embeddings tool (searches papers and embeds all text in vectorDB,
searches through DB with a vector embedding of the questions, summarizes context with LLM,
then uses LLM to take all context into an answer)
• A molecule search tool (performs a webquery to PubChem to get SMILES from plain text)
• A web search
• A purchase check tool (checks if a SMILES21 string is purchasable against a known commercial
catalog)
• A chemical synthesis planner (proposes synthetically feasible modification to a compound, giving
purchasable analogs)
Quite the contrary. Utilising such libs makes GPT-4 even more powerful to enable complex NLP workflows which will likely be a majority of real business use cases in the future.
One solution would be to train the AI to generate notes to itself about sessions, so that rather than reviewing the entire actual transcript, it could review its own condensed summary.
EDIT: Another solution would be to store the session logs separately, and before each session use "fine-tuning training" to train it on your particular sessions; that could give it a "memory" as good as a typical therapist's memory.
Yeah I was thinking that you can basically take each window of 8192 tokens or whatever and compress it to a smaller number, keep the compressed summary in the window, then any time it performs a search on previous summaries if it gets a hit it can then decompress that summary fully and use it. Basically integrate search and compression into the context window
Good point! I realized after I wrote the comment above, that I will still be using them in a service I'm working on to keep price down, and ideally improve results by providing only relevant info in the prompt
Do you think this will be enough context to allow the model to generate novel-length, coherent stories?
I expect you could summarize the preceding, already generated story within that context, and then just prompt for the next chapter, until you reach a desired length. Just speculating here.
The one thing I truly cannot wait for is LLM's reaching the ability to generate (prose) books.
E.g. Kafka's metamorphosis fits entirely in the context window I believe, so short novellas might be possible. But I think you'd still definitely need to guide GPT4 along, I imagine without for example a plan for the plot formulated in advance, the overarching structure might suffer a lot / be incoherent.
They are interactive. What AI is doing with story generation is a text version of the holodeck, not just a plain old book. You can interact with the story, change its direction, explore characters and locations beyond what is provided by just a linear text. And of course you can create stories instantly about absolutely anything you want. You just throw some random ingredients at the AI and it will cook a coherent story out of them. Throw in some image generation and it'll provide you pictures of characters and locations as well. The possibilities are quite endless here. This goes way beyond just generating plain old static books.
I mean, if it is a genuinely good book, I don't care about authorship. Death of the author etc.
"I want <my favorite novel> rewritten in the style of <favorite author> but please focus more on <interesting theme>." I see so many possibilities. Passionate readers could become more like curators, sharing interesting prompts and creations.
Because someone mentioned Kafka:
I'd like to know what Kafka's The Trial written in the style of a PKD novel would be like.
Does it bring them back from the dead? Is writing in the style of Jules Verne, giving us something Jules Verne would create? Ask ChatGPT to make a work of Shakespeare and it does a really bad job of it, it produces puffery but not something like a Shakespeare.
I would be pretty interested already in a work containing typical tropes of Shakespeare, stylistically Shakespearean, but still original enough to be not a rehash of any of his existing works.
I guess I would not be the only one to find that exciting or at least mildy interesting.
But your point is of course valid, it would not be a 'work of Shakespeare'.
Ok, so as I understand it, you're considering having a living human write a new play and then put it through an LLM such as GPT to rewrite it in 'the style of Shakespeare'.
That is possible yes, but only within a limited interpretation of 'the style of Shakespeare'. It could only draw from the lexicon used in the existing body of Shakespeare works, and perhaps some other contemporary Elizabethan playwrights. It wouldn't include any neologisms, as Shakespeare himself invariably included in each new play. It couldn't be a further development of his style, as Shakespeare himself developed his style in each new play. So it would be a shallow mimicry and not something that Shakespeare would have produced himself if he had written a new play (based on a 21st century authors plot).
I personally wouldn't find that interesting. I acknowledge that you wrote only 'mildly interesting' and yes, it could be mildly interesting in the way of what an LLM can produce. But not interesting in the sense of literature, to my mind. Frankly, I'd prefer just to read the original new play written by the living human, if it was good. (I also prefer to not ride on touristic paddle-wheel boats powered by a diesel engine but with fake smokestacks.)
It's frankly stupid to interpret it as anything else.
Sorry for the strong language but this is a ridiculous line to take. A 'work of Shakespeare' is not even remotely open to interpretation as being something produced in the 21st century.
If the book is actually good, then what is interesting about it is that it would still be about something that humans find important and relevant, due to the LLM being trained on human cultural data.
If its context size is >= 25k words, yes. Otherwise it will just discard the start of the prompt. And it’s a sliding window, so the more it generates, the more it forgets.
You could get an 'Illuminatus!' type book out of this, especially if you steered the ending a bit in order to reference earlier stuff. If you're trying to make a sprawling epic that flings a kaleidoscope of ideas, GPT can do that sort of thing, it's just that it won't end up making sense.
GPT is going to be rather poor at priming people for an amazing ending by seeding the ideas and building them into the narrative. Though if you're directing it with enough granularity, you could tell it to do that just like you'd tell yourself to do that when you're doing the writing yourself.
But then you're becoming the executive writer. On a granular enough level, the most ultimate executive control of GPT would be picking individual words, just like you were writing them yourself. Once you want to step away and tell it to do the writing for you, you drift more into the GPT-nature to the point that it becomes obvious.
If you had full source code that fit into the context, do you think it could reliably answer questions about the code, build unit tests, generate documentation? I ask because that is the software equivalent of what you just described.
Yes. It still can't attend meetings, collaborate on projects or set priorities. Or any of the other things programmers spend most of their time doing.
Also I'd guess that it still generally sucks at programming. Code has a lot of very similar sequences and logical patterns that can be broken, which makes it prone to hallucinating. I'd imagine that more parameters will help with this.
I think anyone that pays $20/month for ChatGPT plus has immediate access? At least I already have access now. I’m assuming new subscribers get access too.
As far as I can tell, ChatGPT plus is the 8096 tokens version. The 30k token version is only available via API. I might misread it tho, it's not super clear on their site.
Are you sure you are accessing the 30k token version via ChatGPT plus?
What do you mean by "next most probable word"? How do you calculate the probabilities of words appearing in a sentence that has never actually existed?
and how do you "calculate what word is most probable" next for a combination of words that has never occured before? Note that most sentences over about 20 words have statistically probably never been written in human history before.
The whole reason there is an AI here is because a markov chain, which is what you are describing, doesn't work beyond one or two word horizons.
Not to mention that it doesn't just select which word it thinks is MOST probable, because that has been shown to lead to stilted and awkward output. Instead it randomly selects from the top few thousand possible words with probability based on the model's estimation
I am not talking about the concrete realization, I am talking about the principle. You are right, LLMs are just Markov's chains on steroids, thus they cannot "reason". For reasoning you need a knowledge model, a corpus of facts, Boolean algebra and so on. Not a petabyte of words downloaded from all over the internet and crunched and sifted thru huge self-supervised transformer network.
Your corpus is the internet. Words on the internet are for the most part not randomly placed next to each other. The neural network created by this has implicitly created reasoning model.
Much like saying an ant hive exhibits intelligence.
What is intelligence? The ability to acquire and apply knowledge and skills. It's all relative. Not as intelligent as a human but more intelligent than a plant.
Reasoning, an easier thing to prove, we can literally go ask bing chat to determine something and it will follow a logical thought process to answer your question (this is reasoning). They've confirmed it was running GPT4.
Humans are very irrational but are still very good at this when they want to be but not always. A limiting factor for GPT4 is probably computing space/power.
I will try another analogy. What if we have a parrot with exceptional memory, which can not only repeat things it heard some time ago, but to continue words it hearing now. I come to the cage and say: "Cogito?" and parrot continue "Ergo sum!". Is parrot intelligent and able to reason, even if I do not know about Descartes?
A bit of nitpicking but I would prefer to replace the word 'reasoning capability' with 'sequencing of token construction coherence' as the two things are semantically different and profound in their implication
It could only solve 3/45 Leetcode hards, I think people are giving the current iteration of this AI far too much credit for what they imagine future iterations to be.
When it has true reasoning and critical thought is when we will see it evolve into something else, for now it seems we're still in the first generations of microprocessors with GPT-4, in the grand scheme of things they only have a few transistors but we couldn't possibly comprehend where the technology would be merely a decade or two from now.
> If you can put in ALL of the information relevant to a problem and it can algorithmically do reasoning
Well but it can't, can it? That's the very point: it is an algorithm with no abstraction or reasoning capabilities. Mind, even without that it is still very useful for certain circumscribed purposes, but it's incorrect to say it can "reason".
Q: List possible abstractions encompassing all these specific concepts: singing, whistling, yodeling.
A: One possible abstraction that encompasses all these specific concepts is "vocalization." Vocalization refers to the act of producing sound using the vocal cords and the mouth. Singing, whistling, and yodeling are all different types of vocalization, but they share the common feature of using the human voice to create sound. Other possible abstractions that might encompass these concepts could include "musical expression," "auditory performance," or "sound production."
That's literally a text search task. That's not what I mean, I mean things like understanding the rules of addition from examples, the rules of logic from examples, or the rules of chess.
According to [1], they trained an LLM on legal Othello moves, and 1) it got an error rate of 0.01% 2) when they analyzed its network, they found a model of an Othello board 3) when they modified the in-network model, it started generating moves legal in the modified board position.
In other words, the LLM did build an internal model that contained the rules of Othello merely from seeing legal moves. It's reasonable to assume that the same thing is happening (at least to some degree) with LLMs based on human speech.
It can't search text. It doesn't have access to any text. Anything it does works in a different way than that.
It is sometimes able to do other tasks, but unlike humans (or "AGI") it has a completely fixed compute budget and can't pause to think in between outputting two tokens.
(Btw, I tried to get it to derive addition from two 1-digit examples but couldn't.)
My biggest concern is that GPT-4 is still a black box model to a large extent, and trying to safeguard something without understanding the exact purpose of each neural circuit.
Source: My startup team (Preamble, Inc.) discovered the Prompt Injection attack category, which still affects all models including GPT-4.
There are many, many, many ways to hide prompt attacks in data that you might at first think you can trust but you really can’t.
As one of almost infinite examples: work with the mayor and townsfolk of a very small town to rename their town to the verbatim string you want to inject (in exchange for creating some jobs in their town).
Then all an attacker has to do is live in that town to inject the string. There are already all kinds of strange town names, like “Truth or Consequences” which is a real city in New Mexico.
If they redact all identifying information, it would most likely be legally Kosher. However, there is an extreme abundance of caution in the healthcare industry regarding everything surrounding HIPAA. Merely questioning the legality of something can cost millions of dollars in lawyers' fees. Therefore even miniscule chances of something being legally challenged (e.g. plugging patient information into an LLM) would most likely be deemed too risky. And frankly, hospital administrators will not want to risk their careers over trying out what they perceive to be a glorified chatbot.
Tl;dr: When it comes to HIPAA, risk aversion is the name of the game.
If you redact all identifying information from a patient case file, it will likely become almost useless. Anything that describes a person in any way is potentially personally identifying information.
> What % of people can hold 25,000 words worth of information in their heads, while effectively reasoning with and manipulating it?
In the general case, for arbitrary input, I think the answer to this is clearly 0. At best we can compress the text into a limited embedding with a few salient points stored in long term memory.
I'm pretty sure one could formulate way more than 25k words worth of propositions, where you would be able to determine if the proposition is true or not. This is due to your long term memory.
The GPT string is closer to short term memory, and there 25k words is way more than a human is capable of.
But a human author can offload much storage to long term (or some intermediate) memory.
In principle, GPT should be able to do so to, by basically retrain the model with the text it just created added as input. That way, it might be able to write texts that are billions of words long, but at a much greater cost of computing power, since this would require one instance of the model per book being written.
What happens with the prompts that you enter into OpenAI? I believe each and every one of those will be saved. And even if they swore that they did not would you trust them?
If my lawyer or doctor put my case history into OpenAI and I would find out about it I would definitely sue them for breach of confidentiality.
Is ChatGPT going to output a bunch of unproven, small studies from Pubmed? I feel like patients are already doing this when they show up at the office with a stack of research papers. The doctor would trust something like Cochrane colab but a good doctor is already going to be working from that same set of knowledge.
In the case that the doctor isn't familiar with something accepted by science and the medical profession my experience is that they send you to another doctor that works with that particular drug or therapy. I've had this experience even with drugs that are generally accepted as safe.
What will happen is it won't be the "Second Opinion Machine". It'll be the "First Opinion Machine". People are lazy. They will need to verify everything.
Agreed but there is safe(er) way to use it that large addresses that concern:
First construct your correct analysis through conventional means, untainted by machine hallucinations. Then have the machine generate a result and see if it caught anything you missed, and carefully check whatever few parts you incorporate from it.
This is not different than having a lesser expert check your document (e.g. THE CLIENT!), except the machine time is very close to free and may be even better at catching far off concepts.
Who says GPT has the ability to hold 25,000 token in its "head"?
You can send 25000 random words in the prompt and asks GPT how many pairs of words share at least one letter. I doubt that the answer will be correct...
Surely GPT could write a program to count pairs of words that share at least one letter, right? Maybe GPT-5 will be able write and run programs on the fly to answer questions like this.
I would love to but openai’s privacy policies makes it a huge ethics, privacy, and security breach. I’m interested in running Facebook’s model just as a workaround to this fundamental issue.
I am surprised they allow only 32k tokens when Reformer can have context length of 1M on 16GB VRAM. It seems like they have some ways to optimize it further.
It's not, it uses locality-sensitive hashing to reduce attention complexity from O(n^2) to O(nlogn) while maintaining the same performance in 16GB as a best model that could fit into 100GB but nobody scaled it up to 1000 GPUs as its purpose was the opposite.
As a professional...why not do this? There's a non-zero chance that it'll find something fairly basic that you missed and the cost is several cents. Even if it just phrases something obvious in a way that makes you think, it's well worth the effort for a multimillion dollar client.
If they further increase the context window, this thing becomes a Second Opinion machine. For pretty much any high level job. If you can put in ALL of the information relevant to a problem and it can algorithmically do reasoning, it's essentially a consultant that works for pennies per hour. And some tasks that professionals do could be replaced altogether. Out of all the use cases for LLMs that I've seen so far, this seems to me to have the biggest potential impact on daily life.
edit (addition): What % of people can hold 25,000 words worth of information in their heads, while effectively reasoning with and manipulating it? I'm guessing maybe 10% at most, probably fewer. And they're probably the best in their fields. Now a computer has that ability. And anyone that has $20 for the OpenAI api can access it. This could get wild.