Everyone here seems totally lost on the physics connection. Suppose you have a box of atoms, each atom can be in one of two states, a low energy E1 and a high energy E2. If the box has a temperature T, then the probability that any atom is in state E1 is e^(-E1/kT) / [ e^(-E1/kT) + e^(-E2/kT) ], and similar for E2. As you lower the temperature most of the atoms gravitate towards the lower energy state E1, and as you raise the temperature they gravitate towards a 50/50 mix of E1 and E2.
Since you bring up physics, this has a name: the Maxwell-Boltzmann distribution. You might have trouble getting physicists to describe particles as “gravitating” to it, though — the particles are doing their own thing, and this is the resulting probability distribution.
But this is only for distinguishable particles. If you have a bunch of indistinguishable particles, you get the Fermi-Dirac distribution or the Bose-Einstein distribution, depending on whether they are fermions or bosons.
You can find all of these distributions on Wikipedia.
Fun fact: not only you get closer to a 50/50 mix of E1 and E2 (increasing temperature) but if you continue to add energy to the box you will find yourself with more atoms in the E2 state than in the E1 state.
The temperature will go from infinity to -infinity and as you keep adding energy you will approach zero temperature from the left (increasing temperature). The zero value is reached when the energy of the system can no longer be increased and all the atoms are in the E2 state.
How are you defining temperature? I assume it's not average kinetic energy of the particles. Is it that definition I learned once upon a time where T = d entropy /d energy? Is this a useful definition of temperature if it leads to this scenario?
Yes, this is in accordance with the definition of temperature as d entropy / d energy, which is the more generally-applicable definition. It's easy to see from this that temperature becomes negative if adding more energy into the system causes the entropy to decrease. In most systems that doesn't happen because there are always an increasing number of higher-energy states that open up, but in a very carefully constrained system, the added energy forces the system into a smaller number of excited states away from a higher-entropy ground state. A real-world example of this is a laser's population inversion.
A system at finite negative temperature is actually considered "hotter" than a normal system at any positive finite temperature; if you put them in thermal contact, heat will flow in the direction that increases entropy, which you get by taking energy out of the negative-temperature inverted system and adding it into the ordinary positive-temperature system. This increases the entropy of both systems.
The definition where temperature is the "average kinetic energy of the particles" is a special case, and it only really works when that energy is evenly distributed over all degrees of freedom. For example, you wouldn't consider an icy comet to be at a high temperature just because it's moving quickly, even though its particles have a great deal of kinetic energy on average!
The two-state system was the first example I learned in undergrad stat mech, and really helped me understand the first-principles definitions of entropy, temperature, etc. My parent comment was more for the casual HN reader, but if you really dig into the two state system, negative temperatures aren't all that weird.
The first thing to understand is microstates, which is just the number of ways a system can have a certain energy. Eg in a two-state system with ten particles and energies +/-(E/2), there's one microstate where the energy is -5E (all negative), ten states with -4E (one elevated), etc. Then entropy is just the log of the # of microstates, which is much easier to deal with, since microstates tend to behave exponentially. Eg entropy(E=-10) is log(1)=0, entropy(E=-9) is log(10), etc.
Then temperature like you said is d(entropy) / d(energy). Two systems with different temps brought into contact will exchange energy until the temps are equal, since this configuration maximizes entropy.
The two-state system can have negative temperatures since entropy starts decreasing with energy once more than half of them are in the higher-energy state. This can't happen in more familiar scenarios (eg ideal gas, blackbody, etc) since usually entropy always goes up with energy.
Just because there is a physics connection doesnt make this a good name for the parameter.
I do understand the benefit of not having 10 different names for the same concept under different scenarios, however. Even if that name isnt the best.
But note that, even in the physics scenario, temperature isnt really the name providing the most intuition either. You need to be aware of the connection between higher temperature -> higher excitability/mobility to begin with, for it to make sense; and temperature isnt the only way to modify this underlying excitability in the first place.
> A higher temperature θ “excites” previously low probability outputs.
I found this misleading, because the low probability outputs were already excited. Consider that the first softmax formula has no theta in the denominator, so it’s equivalent to theta=1 (max temperature).
The way I explained temperature to a colleague last week was: temperature of one results in drawing from the posterior distribution, while temperature of zero gives the argmax that distribution. Intermediate temperatures get some blend between them.
But now I see that that hand-wavy description doesn’t explain what happens to the runner ups to the argmax, which this site beautifully does.
The metaphor is straightforward: higher temperature ~ higher entropy ~ less information ~ probabilities are further from certainty (0 and 1). I think the connection originally comes from Boltzmann machines and simulated annealing that were based off statistical physics.
Roughly analogous process to the temperature parameter in simulated annealing, I think. Low temperature equates to a greedy algorithm (i.e. always take the most likely next word fragment) - higher temps. increase the probability of taking a lower probability fragment.
I’m not sure if comparing it to the greedy algorithm is the correct way to think about it. Even after applying the temperature, if you use greedy decoding, you’ll still pick the output that has the highest probability. Where the concept of temperature is effective though is when you use a sampling method like sampling from a multinomial distribution, nucleus sampling, etc.
The analogy is apt. In SA, the temperature is used to make the distribution sharper or fatter. In this article, the temperature is used to make the distribution sharper or fatter.
To me, ML is tending towards religion. Religion in the way of blackbox processing. Seems no one can reason what happens within the network, the number of parameters, the number of layer, the choice of temperature, etc. Or it is just my ignorance. While the enlightenment nudges mankind toward science, could it be that the scientific breakthrough of ML is to knock us back into pre-enlightened age?
Per stephen wolfram [1]
> It’s worth emphasizing that there’s no “theory” being used here; it’s just a matter of what’s been found to work in practice.
>no one can reason what happens within the network
The formula of a neutral network is not hard to see. People would not be able to write code to do training or inference if they didn't know what is happening in the network.
>the number of parameters, the number of layer
You can measure the effectiveness of a neutral network to know how tweaking the number of parameters affects it. There are also restrictions like hardware or execution time that can place an upper bound on these aside from overfitting.
>the choice of temperature
Similarly, you can measure the quality of output to find the temperature that seems to work the best.
The theory is that it's all one big optimization problem. Where the values of the parameters of the NN aren't the only thing you can tweak.
> You can measure the effectiveness of a neutral network to know how tweaking the number of parameters affects it
We can throw darts to the board and measure how close we are to the bullseye. Then we can tune parameters like throwing force, elbow angel, etc. These parameters have physical meaning. We can reason that by throwing harder, the trajectory of the dart can be so and so affected, hence causing the dart to fall closer to or further from the bullseye. Regarding "People would not be able to write code to do training or inference if they didn't know what is happening in the network", I understand from an operation perspective what the neurons are doing, but I also don't get what they are doing, if that makes sense.
For example in each layer of a CNN, is it doing Fourier transform, low pass filtering, rotating, scaling or whatever? And even if they are, why does it work? Why does it give the result that we want? Can we reason that varying our throwing force will change the trajectory of the dart? Shouldn't we do that? Or we are happy with getting results of hot dog/not hot dog [1]. There lies my uneasiness with ML.
I agree with cycrutchfield's comment that practice outrun theory though. We will learn more as we progress.
>We can throw darts to the board and measure how close we are to the bullseye. Then we can tune parameters like throwing force, elbow angel, etc. These parameters have physical meaning. We can reason that by throwing harder, the trajectory of the dart can be so and so affected, hence causing the dart to fall closer to or further from the bullseye.
This is how Newtonian physics is derived. Similar to neural networks you create an approximation based off your observations. The actual reason why those approximations work is not necessary to be known to utilize Newtonian physics.
>in each layer of a CNN, is it doing Fourier transform, low pass filtering, rotating, scaling or whatever? And even if they are, why does it work? Why does it give the result that we want?
What the layers themselves are actually doing doesn't really matter and each one is just a part of a larger system. What is important is finding a set of weights that approximates a function that minimizes loss. People care about the ability to find this approximations. What exactly it is found is less of a concern.
>Or we are happy with getting results of hot dog/not hot dog
Neural networks are not limited to binary classification.
The model knows not where it is, but it knows where it isn't. By subtracting where it is from where it isn't it obtains a difference, or gradient. Gradient is used to generate corrective commands to drive the model from a position where it was to a position where it wasn't.
We don’t use neural networks because they’re a priori the most accurate models of physical processes. We use them because they have two interesting properties:
1) They’re arbitrary function approximators.
2) We have a great way of fitting them to data (backpropagation).
As an aside, it’s just an accident of nature (or is it?) that neural networks have passing similarity to biological systems.
Anyways, it doesn’t make much sense to try to reason about what a neural net is doing inside. What it’s doing is trying to approximate the function it was trained to. That’s it.
That quote is not about how the model works in general. It's saying that there's no theoretical or mathematical way to identify "the best" temperature. Which is obvious. What level of whimsy creates "the best" novel? It's subjective and noisy.
It has nothing to do with a poor misunderstanding of the model. Temperature is not that hard of a concept. A strong understanding leaves you understanding that such a parameter will be kind of fluffy. That's the nature of creative processes.
How different is this from medicine? When problems begin to contain irreducible complexity human reasoning starts to break. We can't test every possible state to determine which state is the best (in human reasonable energy use and timeframes). When this occurs we use heuristics and throw something at the wall to see what works.
Sometimes there just may not be a simpler theory of why something works. Or the 'simple' theory is still massively complex. Reality doesn't owe us an easy explanation.
Neurosciences are helping understand the brain better, but we don't much yet.
AI is developing more capability, while we are losing understanding of how it is working.
Convergence would happen at some point, where for a given feature/capability, we may have an equalish understanding of the brain and AI. Subsequently understanding of both may increase together.
This may rinse and repeat for more desirable features of the brain.
We may find that the brain does not have too much more to put into AI. What if the hard problem of consciousness is not actually is.
I get that temperature comes from the connection to physics but this seems like an overly complex explanation to a relatively simple formula. If you have a softmax and you multiply the input by some factor, if that factor forces the exponent to approach zero you’re approaching the uniform distribution, and if you make the exponent approach infinity you get the arg max. Curious if others feel the same way.
Are you suggesting that your explanation is more simple? (genuine question)
I find "temperature" quite intuitive as a metaphor in that context. It doesn't even require me to think of the formula to understand "the higher the temperature, the more it will create a stew of the results"
No. I'd say everything the model does is a hallucination.
Some writers tend to split ChatGPT oputput into "correct answers" and "Hallucinated answers". If this is what you are thinking when saying "Is this part of why ChatGPT hallucinates?", then no. The model never knows if what it is saying is correct. It has a probability about how likely the words are to follow eachother, and that is about it.
The temperature changes how these probabilities are calculated, making otherwise less probable words more probable or vice versa.
However, the model doesn't choose the most likely word every time. Instead it rolls a dice to see where it wants to go next. If Cheese is 80% and Cookie is 20% it won't necessarily choose Cheese. It rolls a die, and maybe the outcome lands on the less likely Cookie.
The roll of the dice happen _after_ temperature has affected the calculation of probabilities.
Maybe it is this roll you are thinking of as hallucination?
While it can be customary to trivialise this, the reality is that no "intelligence" is sure if what it's saying may be correct or constitute any kind of "knowledge". At this level a human can't claim to arbiter fact versus fiction.
Fundamentally, no one can be sure what they experience is not some kind of simulation, or a dreamlike state beneath some more real "reality". As such, assigning hierarchy to such a foundation lacking in tangible proof is arbitrary.
The point being, unless it's possible to conclusively and infallibly prove you are not hallucinating, you can't lay claim to having knowledge an AI is hallucinating, or not.
That line of argument swiftly devolves into the case for solipsism.
Generally, you may preface any Internet discussion with: first, assume there an independent reality that the author and the readers share. Also causation and the forward arrow of time is true. (It occurs to me that this sounds like prompt engineering.)
In other words, you can never conclusively determine you’re not hallucinating (c.f. Hume), but let’s pretend we’re not for the sake of discussion.
Unfortunately, it also swiftly devolves into the issue being discussed...
It's not possible to prove whether or not a model can hallucinate or even think, based upon the premise that we have arbitrarily defined what constitutes "thought" (and so also "hallucination").
Can you prove an AI can or can't "think" in a way close enough to us that it could be considered to have basic ("human") rights? What if we built it out of biological material? And would that make any difference? Why?
This leads to: how may we objectively prove that the exclusivity of biological "consciousness", as we've defined it, is correct?
How may we prove either we or an AI can think, hallucinate, or "be conscious", when we are unable to objectively define the terms in a provable way?
I think it may be impossible, and as I said - I don't think it's a trivial concern. We may soon be faced with an entity that demands we respect it, and we need to have worked through as much of that discussion as possible, prior.
> The point being, unless it's possible to conclusively and infallibly prove you are not hallucinating, you can't lay claim to having knowledge an AI is hallucinating, or not.
This is very much true for more irrational (or less deterministic) "thoughts" in certain languages. However, if I asked you to write JavaScript or Python to "prove" your thoughts on the answer, it would be a simple matter of running the code you output on a state machine to determine the truths of the query.
super-frog-of-adventure~> !calc 1+1
athletic-giraffe|> 2
super-frog-of-adventure~> !calc one plus one and output the answer
athletic-giraffe|> Writing code.
// filename: synth_calc_plus.js
// version: 0.1.1
// description: simple addition calculation
var num1 = 1;
var num2 = 1;
var answer = num1 + num2;
Sidekick(answer + " (units)", ", ");
athletic-giraffe|> 2 (units)
super-frog-of-adventure~>
> the reality is that no "intelligence" is sure if what it's saying may be correct or constitute any kind of "knowledge"
This is not true. An intelligence should almost by definition be able to recognize patterns, i.e. when things are similar or are the same. This is why we know that 1+1=2, since 1+1 and 2 are just different names for the same thing.
ChatGPT 'hallucinations' are a lack of this exact sort of consistency that makes it obvious that words are merely symbols and not representative of an actual thing. And how could it even know otherwise? ChatGPT is never exposed to things.
I'm certain in my knowledge of 1+1=2, but even if I wasn't, I know that 1+1=2 aren't just some ASCII symbols; but we're not entirely sure ChatGPT can be either of these.
> a words are merely symbols and not representative of an actual thing. And how could it even know otherwise? ChatGPT is never exposed to things.
An interesting argument, what does it mean for the disabled?
Eg, if a human had a fully functioning brain, but from birth could not sense anything except text sent to their brain, which they could only respond to with text, would this mean they're equivalent to a language model?
> but we're not entirely sure ChatGPT can be either of these.
Here you seem to agree with my point. We cannot be sure. Why?
> would this mean they're equivalent to a language model?
This is just a bad question. For one, we know that ChatGPT doesn't use neurological mechanisms. It has no proteins. It's not analogous. You're comparing meters to radians. I'm going to say no, a human without sensory organs is not just an LLM, not because of some complicated argument about how evolution imposes structure on the brain prior to any experiences it has, but because the premise is just bad.
> Here you seem to agree with my point. We cannot be sure. Why?
This is not the same type of 'sure' as in "we're not sure if there's life out there." It's 'sure' as in "we're not sure monkeys can build a bridge on their own."
Side note: when did it become standard rhetoric to say "you're agreeing with me now."? I'm obviously not agreeing with you. Do you think I need reminding? Or do you need reminding?
It's made me somewhat interested in having an informative argument to actively disinterested in anything other than telling you how wrong you are. Can we keep it to reddit please?
How can we confidently state we know ChatGPT doesn't use neurological mechanisms, when we don't yet completely understand our own neurological mechanisms, nor emergent mechanisms that may be at work within ChatGPT?
At the same time, we can already comprehensively model some biological neurological systems in their entirety, in silicon. This will continue and the human mind will not be immune to such progress, making the question important.
I see it as a question that cuts to the core of the matter - long-standing, unanswered philosophical questions:
What does it mean to "perceive"? Can a machine be conscious? Are our biological processes meaningfully differentiable from equivalents when mimmicked by a machine? Are our "perceptions" special? Or not?
Putting the point another way:
We can't be objective about "perception" from within the constraints of perception.
While we routinely "imagine" an objective reality that exists beyond our perception, that we believe our perceptions reveal, there is no way to conclusively prove that's actually happening. Possibly, ever.
(As for the side note: I genuinely interpreted it that way, hence the word "seem", I didn't think about whether or not you'd missed it. I too assumed a conversation precisely unlike reddit... my default assumption here.)
You can't "cut to the core of deep philosophical questions" when you don't know the basics. For example, this
> we can already comprehensively model some biological neurological systems in their entirety, in silicon.
is just patently wrong.
> How can we confidently state we know ChatGPT doesn't use neurological mechanisms, when we don't yet completely understand our own neurological mechanisms, nor emergent mechanisms that may be at work within ChatGPT?
How can we confidently state that the sun isn't alive when we don't ourselves know what life is? I mean, it's got different layers like a cell. It has a source of energy, a lifecycle and it eventually dies. The sun has been around for much longer than chatGPT so I think we should answer this question first.
> While we routinely "imagine" an objective reality that exists beyond our perception
I'll just say this - if you really want to believe you're a brain floating in space, why don't you live life like it's GTA? Because obviously none of it is real, right?
Look up 'solipsism', this isn't a new or for that matter important philosophical position. Unless you want to write about how much fun whatever video game you're playing is in lieu of reality.
> How can we confidently state that the sun isn't alive when we don't ourselves know what life is? I mean, it's got different layers like a cell. It has a source of energy, a lifecycle and it eventually dies.
Well, I've learned from your previous comment you're not agreeing with me, but here's another instance I'd have thought you might be.
These are exactly the kinds of questions we should pose and explore. Are such a massive, complex, long-lasting sequences of nuclear processes "alive" or potentially intelligent? Might they have a detectable consciousness?
What are we but a vast sequence of chemical and electrical processes? In that sense, not much different.
(Science has indeed so far rather arbitrarily defined "alive", and this shows in some of the more controversial categories of "life", such as viruses.)
> I'll just say this - if you really want to believe you're a brain floating in space, why don't you live life like it's GTA?
I've never stated that, I've posed a relevant, important philosophical question I think lies at the centre of the issue of whether or not an AI can be considered to "hallucinate". As was the topic. Your response seems to be "no", and that's fine.
Solipsism implies there is only the self. However, it is possible that we all have similar enough brains that have evolved for us to share a false reality together, like Donald Hoffman's recent theories. On that note I do often wonder about the sentience of stars, photons, everything in between. So chatgpt is not much of a stretch and is exciting in that context.
The thing is that words are supposed to be useful. If a word can't tell the difference between photons, stars, and everything in between, then it's not very useful anymore. Calling all of these things 'alive' would just defeat the meaning of the word - we can't resuscitate a rock, we can't talk to photons, feed an atom - all these things that we associate with being alive are just rendered inapplicable.
Again, do people need reminding of who agrees with who? What is the purpose other than to give points to each other? And I obviously wasn't agreeing with them, under any reading of the reply.
You say it never knows if what it is saying is correct, but in my experience it tells you when it does and doesn't know about a concept. For example in one thread I had it give me some random words from linear algebra, then in another thread I combined these into the nonsense phrase "decomposed orthogonal eigenvector division" and asked it if it was familiar with this, and if so to define it. It said it wasn't familiar with it and asked for more information. I'd say that shows it knows what it doesn't know.
I think it hallucinates in scenarios where it either doesn't "remember" the answer or there is no answer.
For example: since the text generation is sequential, based on context the highest probability continuation in a situation might be:
"A famous paper that discussed this phenomenon is"
However, when the network gets to this point it doesn't know the name of such a paper. Maybe it already knew that it doesn't know the paper's name but the above sentence was generated despite that, since everything else was lower probability or it got sampled due to higher temperature. In any case, it can't go back and delete the sentence so it must make up something. So it ends up going with the most plausible sounding (highest sample probability) continuation, which ends up being a "hallucination".
Basically hallucinations are produced when the network "writes itself into a corner" due to the sequential nature of text generation.
It can halluciate at low temp too. e.g. mice don't necessarily eat cheese; it's just a common stereotype of what mice eat.
High temp is more likely to lead it into ranting on an unrelated topic. You ask it a question on what the mouse eats. It answers the question and then starts saying what lizards eat, or goes on a rant about food chains.
I think hallucination is akin to encountering a problem on a test to which you don’t fully remember the right concept to solve it. You might attempt the problem with whatever little knowledge you can recall, but the answer is ultimately incorrect. Likewise, these LLMs exhibit such behavior when they hallucinate. I think what distinguishes the LLM’s hallucinations from humans is the confidence level. A human who is unsure of their answer might tell you that they’re not certain or they don’t know. LLMs like chatgpt, however, will just confidently make stuff up.
My favorite prompt to try so far to play with this effect is:
Picture yourself in a boat on a river With tangerine trees and marmalade skies
It might veer off into a scenario where you're on a romantic boat trip or talk about how it is its favorite Beatles song and how it remembers them going on Ed Sullivan. :)
The temp affects the "color" of the next word which sends it on a tangent just like a person -- especially somebody with ADHD or playing an improve game as opposed to lip reading, for example.
From what I've seen, low temp "only the most probable" output tends to be more repetetive and incoherent than medium-tempeature output, which is a bit of a paradox. I wonder why that is.
Look up at the stars: You can only see a few of them. These functions are like really twisted/complex lenses and mirrors you might imagine we have hoisted into view so that now the sky is completely white (to our eyes) and we can see every star in the sky.
Now without knowing the precise shape and position of that contraption, because all those stars really flicker at different rates, you can with a very fast camera recognise them by their flicker and note that the same star is actually visible in multiple "places" in the sky. You might if it is useful to you consider a star being "close" to many different stars in the sky -- as distorted by those lenses and mirrors -- simply by how long you watch the flicker.
It is in this way, that I imagine "low temperatures" producing incoherence like the fast camera, not waiting long enough to see how many better (closer!) options there might be, seeing only the brightest (closest to us! cookie!) stars, and not waiting for all the other options (cheese, baguette, etc) to make into a longer exposure.
Or the kinds of things you can meaningfully say after studying the Internet for (any random) week is probably going to be more repetitive and less coherent than the kinds of things you can meaningfully say after studying the Internet for 20 years.
The reason is that actual human-written text generally doesn't follow the most probable next word rule, but rather there are occasional 'decisions' made which make the text unique and therefore more interesting/coherent. There are other sampling methods which try to avoid this issue like "Locally Typical" and "Nucleus" sampling.
Is there any centralized vernacular? I know what a hallucination is, but I frequently come across many terms I need to dig into with surrounding context to figure out. Which works but is not efficient.
From what I understand top_k is the number of words it generates to pick from, and the top_p is the percentage of the top words it will use as the next word, correct?
Machine learning models don't do well with math that has hard bounds at 1 and 0 (probability). So we use the log of the odds ratio instead. This function goes from negative infinity to positive infinity. And is mappable back to 0 and 1. So as you get closer to infinity log odds, you get closer to 100% certainty that your answer is A.
Yeah -- the choice of temperature corresponds to the base of the logarithm. When you think of probabilities in the log-odds representation, a bayesian update means adding an increment. https://arbital.com/p/bayes_log_odds/
Basically, logits are the raw outputs of the last linear layer of a neural network just before the softmax(for multi-class classification tasks) or the sigmoid(for binary classification tasks) is applied to the outputs.
Logits are the raw predictions of the model that you then need to normalize to turn into probabilities (that sum to 1). You can do that is with the softmax function.
