If anyone needed to be convinced that LLMs are not accurate models of the human brain they need look no further than these numbers. The smallest model under discussion requires "more books than are in the Kindle store on Amazon U.S." Humans obviously acquire language proficiency with a lot less input than that.
Yet we learning from 18 hours of ultra high resolution video per day (x2), along with 18 hours of reinforcement learning, along with 18 hours of audio and 18 hours of nerve stimulus data.
This is assuming we are learning nothing during sleep, which probably isn't true.
By the time a person is 21 years old, they have been trained on at least 1 petabyte of data.
By two years old, about 125TB of data. It makes LLMs look quite good in comparison.
And by far most of that data is completely meaningless. You just can't naïvely compare that with LLMs where all the data is curated and having meaningful content.
The majority of the data is a web crawl. It's not curated in the slightest and includes the whole of reddit.
In contrast the data we collect through our nervous system is rich and meaningful and far deeper than just raw text. We can even manipulate the environment as we learn to facilitate faster learning eg. pick up a ball and throw it, rather than just watch videos of balls being thrown.
The web crawling is curation, as even the most mundane entries into the dataset will still have a significant weight on the model, even if it's small. What separates the brain from any statistics-driven model is how well it swims through an ocean of information to find the relevant details, tossing out what's left, and filling in gaps with intuition. Sure, its messy process, and mistakes get made, but that's in stark contrast to the very trim way that ML models run.
By no means rigorous, an estimate for the number of neurons is around 10^10 [0] with an average number of connections estimated at around 8000 [1].
Call 10^10 \approx 2^40 for convenience, and 8000 \approx 2^13, which gives us a 2^53 entropy estimate, or about a petabyte of information as an estimate of what the human brain can store (discounting more exotic theories of memory stored in DNA or some such).
Their communication is much more complex too, and neither they nor their connections are static.
We don't understand how they work at the subatomic level simply because human understanding of the subatomic world is not complete, but even just at the atomic level a single neuron is massively more complex than anything humans have created.
Going up to the molecular level, even that is staggeringly more complex than the incredibly simple abstractions that make up a neural net.
Is what happens in the brain at the molecular, atomic, or subatomic levels relevant or necessary to intelligence and consciousness? We just don't know yet, but we do know all of that is far more complex and very different from the simple abstractions that are used for neural nets and LLMs.
The back of a napkin calculations in this thread don't even begin to do justice to the tremendous amount of "calculation" or "storage" that happens in the human brain.
> Yet we learning from 18 hours of ultra high resolution video per day (x2)
Our object recognition is trained pretty quickly. And we sleep (certainly as a child) more than 6 hours per day. But we don't learn much from just looking at pictures.
> 18 hours of reinforcement learning
You made that up.
> It makes LLMs look quite good in comparison.
So, exposing an LLM to a lot of video will make them understand language?
The poster above for some reason though encoded video is a good measure, which does not make sense actually. Brain gets raw video. With raw video it is about 100PB per year, > 200TB per day.
Even sound alone (uncompressed CD quality stereo) is 3TB/year.
Most of that data is the same all day e.g vision. There’s very little difference between frames.
The significant thing is the reinforcement. Without it you have to resort to the openai brainlet style 10tb of text training.
If their model had reinforcement built in then you could just plop it in front of a person or on reddit and it would rapidly self-learn on a fraction of the data. Their training model relies on it learning purely based on observations rather than interaction which is inefficient.
One argument (that I don't necessarily buy) is that inputs from sensory perception play a major role in building a language model in humans. A human might not read TB of books to learn language, but if you put them in a sensory deprivation tank that only displayed a massive series of Unicode characters it would take forever for us to learn language. If we did the same for an ML model, perhaps those non-language training materials would help.
(That said, DL architectures are obviously wildly different from how the human brain works. E.g. backprop is physically impossible.)
Yep totally agreed. One of the things I'm excited to see going forward is multimodal models (trained e.g. on text + video + audio + images).
I'm sure there's a lot more to it than this, but maybe one factor that makes humans a lot more data efficient is the multimodal input we receive.
If that's the case, imagine how much better things could get when we train with all the videos, podcasts, radio etc in the world, in addition to all the text out there!
Nature is good reference point but it's not by any means optimal. One example may be human eye - we can make cameras orders of magnitude better than eyes created by evolution.
You can look at it the other way around - dopamine and other neuro transmitters as poor approximation of backpropagation. It has many flaws for example tight harmful loops ie. addictions.
Majority of brain work is ignoring irrelevant information (attention) and small scale hallucinations (we don't see world as is but slightly hallucinated to keep it stable - ie. they way brain processes blinking <<turns off>>, you can peek at those nuances with ie. optical illusions etc).
One of missing bits in neural nets may be reusing its output as input (embedded in inference itself, not poor mans re-prompting).
Once it's sorted out I'd argue the performance will skyrocket and give opportunity to massive optimisations ie. embedding things like known functions - imagine brain which has known, very narrowed, available functions at its disposal - all mathematical functions on numbers, logic, optimal sorting etc. Imagine if as thinking human you'd have access to accurate functions - the sky is a limit.
“Nature is good reference point but it's not by any means optimal. One example may be human eye - we can make cameras orders of magnitude better than eyes created by evolution.”
This is only true when considering single performance axis like pixel resolution. When you consider the corpus of power efficiency, jitter resolution enhancement, dynamic contrast, performance per volume, etc. We aren’t close to building something as capable.
> all mathematical functions on numbers, logic, optimal sorting etc
Bayes formula as a built in primitive. You don't need to know much of statistics to see how limited humans are at processing information because estimating posterior updates is so expensive for them.
Thinking in terms of raw probabilities would be very alien to most humans, but could easily be technically superior for making plans.
Wasn’t she ‘boot-strapped’ by having had both for the first 19 months of her life? - Presumably this gave her an internal model of the world on to which she could map touch perceptions and language concepts.
1. You're mischaracterizing the parent's comment, to such a degree that it seems willful.
2. This tendency to police other people's behavior, telling them what they should or should not do, does not help your cause. It puts people on the defensive. No one, especially adults, like to be told what, or what not, to do unless it's coming from a well-qualified lawyer.
Agreed, i think the "telling other people what to do" thing is something i haven't seen brought up enough. Seems like the whole current social media is built on defending some group by telling other people that they're bad people for speaking the way they always have and that they must change themselves to fit your opinions, otherwise they're forever doomed to be "bad people". The bad person part is implicit in the command. When did people get so comfortable commanding others and demeaning their thoughts and agency?
I can't believe you are serious. It is pretty clear what the context is here and I didn't mince my words. You are policing MY behavior and you should stop telling people how to be anti-ableist.
Having a model of the world (predominantly from vision, but also hearing and touch) surely does wonders for language acquisition as it provides the learner a backdrop against which to infer the meaning of speech. When an infant hears “look, a dog” and “look, a cat” in two separate instances, their eyes alone provide enough information to infer, at least at a high level, the meanings of “look, a”, “dog”, and “cat”.
It's pretty clear that humans, unlike LLMs, use external sensory data (the only external data we have, when you cut through it all) when they produce speech, as evidenced by the fact that they don't speak falsehoods that don't mesh with their internal data model. LLMs have such a weak model of reality -- it's whatever “sounds right” -- that they speak falsehoods all the time. The only way to give an LLM sensory data would be to encode every sensory experience people have into text.
> It’s pretty clear that humans, unlike LLMs, use external sensory data […] when they produce speech, as evidenced by the fact that they don’t speak falsehoods that don’t mesh with their internal data model.
I don’t have access to any other humans internal data model, but the indirect evidence I do have suggests that they do, in fact, speak falsehoods that don’t mesh with their internal data model for a variety of strategic purposes.
It's all a combination of both explicit (traditional software that's faster/has better specific accuracy but is brittle) and implicit (machine learning/etc where fuzziness is better) optimized hardware (wetware?)/software all up and down the biological stack, so to speak.
Right. The nature trained all the nervous systems with genetic algorithms for millennia. Individuals rather fine tuned their own with natural noise and dropoffs and natural feedback. Fine tuning our brains are rather expensive.
There are a few innate reflexes but you lose them quickly and they are pretty basic. Our brains have way way more neurons/parameters than these models and are still not overfitting... the issue lies in the learning method.
It's not just about reflexes though I would hardly classify reflexes as "lost quickly". That's just not true.
It's deeper than that. Our brains are optimized for a lot general human functions. Learning language is one of them. There's a whole section of the brain dedicated to it and other things.
There's also a lot of vital biological information encoded in DNA/RNA.
We are not even close to starting from scratch.
Not as strong an argument as you think as you're forgetting humans have 5 senses, are embodied, and millions of years of tuned weights from natural evolution codified into our brain structure.
Brain is nothing like attention networks at least in terms of dynamics and individual neuron complexity, my wild ass guess; there is not enough space in DNA to store even 10% of brain connections.
Cluster density, cluster location, eagerness to connect, distribution of cell types, etc. are all coded in DNA and contribute to creating "initial weights". Neuron "weights" need not be stored completely uncompressed.
Obviously newborns need to develop and take in stimulus before they "know" anything, so the initial conditions are not sufficient, but they are obviously necessary to make the limited learning useful.
It always seems naive to compare neural structures and functions to artificial networks, dynamics are simply not there, even single neural would require multi layered network (if not mistaken 7 layer network, and very long training time), it is fascinating how brain self-(re)organizes through out life time from small number of examples here and there.
As far my understanding goes to very large degree it is unknown how to model such dynamics, where it would be possible to start with no spiking neurons and evolve effective/stable learning behavior from small number of examples/experiences.
I accept this possibility, but because we don't understand, seems wrong to assume any other path, furthermore, there is some evidence that intelligence/cognition is convergent in neural structure and function, in addition to that most ANN are inspired from studying real neurons.
"Cluster density, cluster location, eagerness to connect, distribution of cell types, etc. are all coded in DNA and contribute to creating "initial weights""
Are they coded in DNA, or do epigenetic factors, the environment cells grow in, adjacent cells, their interaction, etc, play a role here as well?
That would depend on what you mean by "connection". The human genome has approximately 200GB of data in it. And this data is highly compressed as it takes 3 different layers of decompression before it gets to usable form.
Also, since you asked, I would like to mention protein interactions with cellular components and tissues. These can be argued as just proteins interacting with each other but the extreme complexity of these higher levels and the unique phenotypes arising from them make me think they are deserved to be treated as another layer of "data decompression".
A fruit fly has no meaningful learned intelligence yet is more capable than any trained neural net. It’s just a small bundle of nerves laid out according to the model encoded in their DNA. It was trained through millions of years of genetic algorithms, and is fully encoded in the DNA, along with the entire ability to specify the entire physical structure of its body. Why is it surprising our brain has instinctual ability encoded in its structure stemming from our DNA, evolved over the history of life?
Not sure about the 10% point. You could imagine that we got “lucky” and a handful of simple proteins manage to produce the brain (or at least enough of the brain’s connections that the rest can be filled in during infancy via sensory input). Or put another way, maybe evolution optimized for brains that are incredibly adept while only taking a small number of codons (relative to the number of neurons) to describe. The fact that we've managed to l describe the universe through a small set of laws shows that the laws of the universe require much less space to encode than the universe itself, so it's theoretically possible.
We have managed to describe a small subset of the universe through a small set of laws. The current incompatibility of gravity and QM and the resulting dark energy/matter explanation means we can’t see or interact with most of the universe.
They aren't accurate models of the human brain just as airplanes are not flapping their wings and cars don't walk on legs. Our brain's way of achieving intelligence is not the only one out there, even if LLMs aren't there yet.
Yes, we use now magnitudes more oil than we used to 120 years ago. But back then oil was used for cooking and lamps, while now it is used for so many more things. Same goes for data. It is the new oil :).
Fair point, but accurate versus efficient are different. While humans do acquire language much more efficiently, these models "know" more than a human about most things - in that if you asked it to describe each of the 100 most popular subjects or books it would have no trouble doing so (in a dozen languages no less). So I don't think it's quite an apples to apples comparison.
Twitter is a madhouse, but as for HN, the closest I've seen anyone come to that here is claiming that the difference in underlying structure can/does lead to emergence of the same phenomena.
That argument is utterly unconvincing, as you've conveniently left out the millions of years of "training" and "fine tuning" that has occurred through the evolutionary process.
It could be argued based on the extreme difficulty we're having in teaching animals human language, even primates. We can create a semiotic system and get a chimpanzee to communicate through it, but it won't learn English no matter how hard we try.
I think that's generally accepted to be that humans have a very large language center in the brain; IE: and inductive bias towards learning language (and I think there is something about our mouth and tongue being able to make specific sounds?)
It doesn't say anything about that knowledge being precoded.
Isn't a language center by itself precoded human toward the acquisition of human language (which has co-evolved to make use of our language center)? I don't understand how it could be any other way.
Your argument is utterly unconvincing too, a human left alone with no parents or way of learning can basically do nothing at all, i.e. what we start with isn't some trained neural net at all, it's an empty framework with no data. The OP is correct in that we are several orders of magnitude more efficient at learning than ANNs.
The fact that human brains are capable of learning things that animal brains cannot, is a clear indicator that it is already "trained" to a significant degree.
Human "learning" seems to be much more analogous to "fine tuning and memory storage/retrieval", than actual training.
Why can't a macaque teach a calculus class? Why can't its "empty" brain be taught to do something like that?
Try raising a child with no inputs and see what it ends up doing. We don’t start “trained” at all. We just start with a neutral network architecture with no data. The difference between us and other animals is the number of neurons and the network complexity, but it doesn’t start trained to any degree. Humans do not start with any ability to talk, reason, identify objects etc.
While I agree with the conclusion, I don't think your argument actually supports it. Humans for the most part don't acquire language through books until they're mostly already fluent in their native language.
These LLMs are also trained on arbitrary books. When acquiring new languages, we use educational materials that are specifically created to facilitate an understanding of language. Not just arbitrary books in random order.
Why do they have to be accurate models? They just have to be possible to construct and have to perform. The construction method is irrelevant beyond feasibility.
I don't care if my model needs an exabyte of RAM if I can just go and buy that much RAM one day.
True but the machinery (is the brain) that enables humans to acquire/process/understand that knowledge is a lot more complex and went through years of environmental conditioning?
No totally. Not an ml guy by long shot. My understanding is llms are foundational models which iiuc are "base" models representing all human knowledge until some point in time. Now I'd argue that you don't have to count all of evolution but just the point from when you were born. I'd say the all the sensory experiences + knowledge input from the day a brain is activated (DOB) would be in the order of PBytes if not more?
Yes, but that environmental conditioning did not involve training on the contents of the Kindle library. And the results of that training are encoded in under a gigabyte.
For the record, I'm not saying LLMs are not a huge step forward. They are. But they are not -- and cannot be -- the whole answer.
If you mean a lot of accuracy, that's obvious and doesn't really need argument. And this new fact doesn't change the argument.
If you mean a more moderate amount of accuracy, this isn't proof either way. Human brains take in less text but they put a lot more processing into it.
I’d note that the models don’t just acquire language facility, but the ability to use language with alacrity about almost any subject. There’s no human alive that is remotely as capable, and most I’ve met hallucinate more.
True, but humans have a 4B element text book downloaded into each cell and some of that stuff encodes for a machine that already has a lot of language proficiency built in from day #1. I've always wondered how much of that genetic code is 'soft' in the sense that it pre-sets certain bits in the brains that it generates. Sort of a boot-strap ROM for a human.
