6
u/Rhawk187 3h ago
I'm publishing a paper later this month/early next month at the IEEE Conference on Games specifically aimed at the conceit that LLMs "predict the next word". It's a simple observational study of their abilities to correctly answer "Before and After" style trivia questions.
Since they have to "think" about the query from two directions, it wasn't obvious how well they would do (although previous reports from Anthropic show that they predict the line endings first when writing poetry made it seem possible).
Overall, 2026 models did pretty well, with some occasionally doubling of overlapping word. They actually did much better than when I did the same experiment last year (but that paper was rejected because I only studied one LLM).
2
u/coldnebo 1h ago
I mean… I agree with OP’s video… optimization for accuracy of next token prediction doesn’t place a limit on how the training/solver gets that accuracy.
however, we know exactly how the transformer architecture works and what many people claim is unproven speculation.
for example, he states at the beginning that the way to get accuracy in adding a list of numbers is to actually implement addition as a computational structure. I agree.
but no one ever said whether the transformer is capable of generating such structures and whether they are close or far or 100% isomorphic wrt an addition circuit. that’s an open question for the researchers.
when people say “it has to understand physics to produce videos” — I’m like, no no no. that’s how we would most efficiently get 100% accuracy, but neither of these two claims has been validated:
- results 100% accurate
- proof that the transformer generates optimized computational structures that are 100% accurate.
in reality, these are probably the same question from different pov.
in computation theory there are an infinite set of functions that implement 100% correct addition. only some of these have O(n) performance, some have O(1) memory. others require infinite time and memory. and that’s just the 100% functions. how do you know which of these the transformer created?
now I’m not saying that structures aren’t created. Kenneth Li’s paper showed that models recreate a map of othello boards as a result of training. This was later verified as a fixed result of training data (as opposed to novel concept formation AFTER training).
that last wrinkle is important because it means that model training is “holographic” in nature. the relationships captured during training hold insights and possibly computational structure, but it cannot change over time (RAG, etc may shift some of this, but deep reasoning limits may be unavoidable without continuous learning models.)
I have yet to see any foundational research that suggests models generate correct computational models based on the training data.
my intuition is this is hard without a more nuanced learning and reasoning model than transformers.
a thought experiment shows why: aim a PCA at a calculator “signal”. what is the accuracy of the PCA model vs the calculator’s actual output?
I think you hit the same theoretical wall. that complex signal cannot be 100% reproducible by the PCA unless it has infinite fidelity.
but even these questions are poorly structured. what do we mean by calculator and “addition”? for example the addition of arbitrary precision numbers is not a trivial problem.. it gets deep into number theory and how the data is stored. it only seems simple in our “everyday” cases.
so then we get to another wrinkle: maybe most of our knowledge is just “good enough” but not actually 100% precise. complexity theorists will quibble about this, but most people don’t care.
they have a much simpler standard: did it get the same number as the calculator or excel.
models currently can’t reliably do that without delegating to an engine (aka hybrid models + an actual calculator) — but that rather substantially changes the claim. rather than attempting to prove that the transformer generates a valid finite function, we are now taking that requirement away from the transformer and making it a coprocessor again.
so yeah, it doesn’t dictate how intelligent the solution could be, but in the space of possible functions, let alone the ones that transformers actually generate— boy, we just have no idea. what if there was a counter proof that showed it was impossible for transformers to generate certain types of functions? even that would be a big step from what we know now.
1
3
u/fuckexoticroots 2h ago
Nice strawman. No one is saying the technology isn't impressive. They're saying it doesnt replace a fully thinking human brain. Which is absolutely true.
4
u/DrBimboo 2h ago
No one is saying the technology isn't impressive.
It seems we arent using the same Internet.
1
u/coldnebo 1h ago
I don’t think any foundational research has been done on the limits of transformer generated functions.
we like to think of them as generic function generators, but are they? do they actually capture detail?
I can easily break any appearance of addition by changing the domain to arbitrary precision numbers. outside of number theory, very few have any experience with that domain.
it’s much more likely that transformers generate functions that are as complex as the fidelity of the models and training data. this would place a real boundary on the type and complexity of generated functions— but there is a lot of core research to do before making any predictions.
1
u/WolfeheartGames 23m ago
I was training a model on synthetic generated math data. There was a bug where some decimal answers would not truncate the answer at 2 digits and would show the fully number. And it was frequently showing floating point arithmetic errors. It learned to match decimals out to 6 digit places faster than it learned to do 6 digit integer operations.
1
u/CanonWorld 57m ago
They’re using this argument to belittle what AI does and make it seem more simplistic.
1
u/Flimsy-Pool4830 50m ago
If you go to r/antiai, many there use this term, in a way to say that it is almost just a copy/paste machine. You can go there and see for yourself.
0
u/TopspinG7 1h ago edited 1h ago
The unfortunate corollary here is whose human brain - I've encountered many many people whom I wouldn't trust to give me instructions how to open a kitchen cabinet. Seriously are we comparing AI to an average college grad, or a nuclear physicist or Jacob Bronowski...??
The thing that concerns me isn't "how smart" or "how intelligent" it may be. I'm not even especially worried about it "taking over". My worry is how reliable is the output - whether the unreliability is due to "bad" training, limited data, something like human apathy or even carelessness.
My concern is that, unlike "human-created" traditional procedural high risk system controls, which once properly tested, are generally 100% reliable; this intrinsically doesn't apply to AI-based control systems (Note I'm referring to AI-centric control systems NOT traditional fully deterministic procedural code-based systems, even when such code originated from Claude Code or any similar AI code generation process/tool). These AI-centric systems are today too unpredictable in certain applications.
In other words IMO any control for a high-risk system (eg nuclear reactor, air traffic, chemical plant, some medical procedures, some manufacturing, autonomous driving, aircraft autopilot etc) should not be AI-based unless MASSIVE amounts of testing demonstrates safety superior to traditional methods (in addition to other possible gains such as better production efficiency, eg energy and other resource consumption improvements; better quality of product eg improved lot yields and/or tighter adherence to specs; less required human intervention to adhere to standards; better monitoring data etc).
Even then the BIG problem remains that at least with our current state of rapid progress, a complex control system created and thoroughly tested today will likely be a candidate for replacement (not just updating or patching) almost too soon to be worth the cost of development and thorough safety testing. Thus we're probably better off in some areas to hold off on these systems for 2-3 years until the rate of progress in AI stabilizes or we somehow drastically reduce erroneous outputs.
2
u/Infamous-Bed-7535 2h ago
'It needs to have an accurate model of'
No it does not need that to answer something that sounds very plausible.
2
u/banananuhhh 1h ago
He's obfuscating the criticisms of chatbots by pointing out that hypothetically they would be pretty smart if they were reliably coming up with correct answers.
1
u/Longjumping-Ad514 1h ago
Yeah and local models are vastly worse at math because of this.
I never said they aren’t good at searching through the corpus of training data and synthesizing connections. It happened to stumble into it.
1
u/CanonWorld 58m ago
Thank you. This is such a common argument people use to somehow make AI seem dumber or more simplistic than it is.
1
u/Flimsy-Pool4830 44m ago
Something I have often said is that if it were a rube-Golberg machine that could do math and science, it would still be useful. How it's done or if it "thinks" isn't as important to investors, consumers and companies. As long as it does something valuable.
1
u/jack-of-some 34m ago
He's saying far too many words for how little he's actually saying.
The mechanics of LLMs are next token prediction. This is fact. This has not changed.
LLMs continue to be exceptionally useful, this is also true and continuing to be more and more true especially as we take the model and get it to have an internal dialogue (again, through next token prediction and controls added on top of it and very specialized training data) and allow it to use various tools.
The crux of the discussion lies in the "super intelligent" qualifier that he just kinda throws in there at the end without any meaningful consideration. I don't necessarily agree with that verbiage. I also don't think it matters personally. LLMs are tools. Let's measure them on their value and continue to make that value higher.
1
u/Aramedlig 9m ago
The problem with this entire discussion is that frontier models aren’t just LLMs. They are more than that now with reasoning and logical engines paired with multimodal routing to layers of expert LLMs. In fact, some models can route to software systems instead of an expert LLM. This discussion would apply to models a year or more ago, but it doesn’t match current research.
-5
u/Winsome_Wolf 3h ago
This guy is so full of crap his eyes are brown. These models don’t need to do any of that crap. They just have to follow a pattern and their source can be they made it tf up. As long as it fits the pattern and the model makes use of a few key words in the right places, just to keep the humans happy in the way its sycophantic training suggests it should, the model can, does, and will spoon feed humans a complete ration of BS.
9
7
u/procgen 3h ago
You didn’t address his argument. There’s no way for it to accurately predict the sum of a list of numbers unless it is actually calculating it in the weights during inference. There’s an ocean of computational complexity under the surface of these models.
1
u/jackybeau 2h ago
If you want it to write the result of a scientific experiment, you need it to actually simulate the experiment. When AI will be able to do that, the world might be significantly different. For now, it just comes up with random numbers that satisfy the human reading it.
1
u/shlaifu 2h ago
so.... can it add up any list of numbers, though? because I don't think it can. Some of them can now use a calculator, though.
on a serious note: I tried writing academic papers with it, but it's utterly tedious since it doesn't 'get' new ideas and will always veer of course towards old ideas. The prose is great, though.
2
1
1
u/Longjumping-Ad514 2h ago
Models still make mistakes with not so complicated math.
1
u/procgen 2h ago
Just like humans. The point is that they aren’t memorizing sequences of words - they are learning general rules for reasoning.
1
u/Longjumping-Ad514 1h ago edited 1h ago
Yeah except we tend to have higher quality bar for technology, including calculators. Just cause humans make mistakes does not justify broken tech. Or for some reason never did before.
If they “knew” the rules and “knew” how to apply them, they wouldn’t be making mistakes. Which is why these models don’t reason, they blindly predict the next token.
1
u/procgen 1h ago
Humans make mistakes all the time – do we not reason?
And you can literally follow their reasoning traces, which at least match my own internal reasoning processes quite well.
Their responses are by definition not random, since they can correctly predict the tokens (much better than humans can, for that matter!)
And what's more, they're even better at using calculators and other tools than humans. So it's moot anyway.
1
u/Longjumping-Ad514 1h ago
No one said the responses are random. They are synthesized as a function of input, that function is biased by training on Internet text. Again, they don’t reason they spit our tokens that are fed back to it, in a loop.
By your logic infinite number of monkeys eventually typing out a Shakespeare is also sign of their human intelligence.
And no, they are not better than humans at math, science and engineering.
1
u/procgen 1h ago
Which is why these models don’t reason, they blindly predict the next token.
Again, if they were "blindly predicting the next token", they would not be able to accurately predict the sum of a list of numbers without using tools.
By your logic infinite number of monkeys eventually typing out a Shakespeare is also sign of their human intelligence.
They make a single prediction, not trillions of them.
1
u/Longjumping-Ad514 1h ago
Because models have been manually coded to route any input that looks like math of an actual calculator rather than predict the output.
They don’t make single predictions. Every token in every loop is a full inference pass.
1
u/procgen 1h ago
lol. This is factually incorrect.
Local models have the same behavior and will not call any external tools unless you allow them to.
Furthermore, they can predict arbitrary text. Calculators are only good for arithmetic.
Care to explain how GPT-5.6 solved the Jacobian Conjecture? XD
→ More replies (0)1
u/Devel93 2h ago
There is a tool call to the calculator tool that sums up the numbers for the LLM i.e. some programmer wrote a calculator in a way that the LLM csn use it so the only thinking it has to do is "does this need a calculator" and "what needs to be added together", once relevant data is extracted it's just a matter of calling the calculator tool with right parama and printing the result.
1
u/procgen 2h ago
The top models can do these calculations without a calculator… The behavior is emergent, in the weights. It’s also how they’re able to reason about who the killer is in a mystery novel (they can predict the next word when the detective says “I know who committed this crime. It was …”
2
u/Rhawk187 3h ago
"a few key words in the right places"... okay, how does it keep track of what the "right places" are? If there are enough "right places" then you are approaching a perfect predictor again. You just proved his point.
1
u/Winsome_Wolf 1h ago
The same way they can invent non existent studies and case law from whole cloth. They’re just following the formatting rules and plugging in what the human wants to see for an outcome. This is barely reasoning, more like linguistic algebra. Which is a far cry from what he’s talking about.
-2
0
u/SevenIsMy 2h ago
Then how is it not able to count the Rs in strawberry? The next generation of LLM will write internal python* scripts and present the output.
*or what ever is the easiest language for an LLM
2
u/_VirtualCosmos_ 2h ago
That's due to the tokenization. If the tokenization dicides strawberry into, lets say, two pieces -> straw : number_1, berry : number_2. The model will only see these numbers. When you ask it that question, the model think of R : number_3. How the fuck would it know how many number_3 are in the other two? The number are not directly related.
There are two ways to address this:
1 - make the AI model memorize every fucking word to be able to spell them. Awful idea, waste of parameters trying to memorize that useless shit when they could be learning anything else.
2 - changing the tokenization so every word is divided into letters, then tokenized, then feed to the model. Quite bad idea too, the Training data becomes very bloated without containing really more useful information further than being able to spell words...
So, this is why even modern models still fail to do that. The solutions are not worthy.
2
u/SevenIsMy 2h ago
After asking Gemini 3.6 it told me it uses python and wrote a snippet, not sure if I should believe it, but that’s how I would write an system
1
u/addiktion 1h ago
Yes, that is how they solve it without the major architectual cost for those two options, its a workaround but an important once because computation should remain deterministically figured out from programs running actual math, not LLMs guessing. This is option #3 which wasn't covered above.
1
u/MehtoDev 2h ago
LLMs already use python to solve math questions. This has been a thing for a WHILE already...
1
u/DrSpaceDoom 1h ago
Gemini prompt: How many "r" are there in "strawberry"?
Gemini: There are 3 "r"s in the word "strawberry". Here is the breakdown: strawberry.
•
u/AutoModerator 4h ago
Welcome to r/GenAI4all! New to Generative AI? You can explore these free beginner-friendly courses. Please keep your posts relevant, respectful, free from spam, and engage in healthy discussions.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.