Sorry Paul, I should have define a LLM. First LLM stands for Large Language Model.
To me a LLM is a vast array like a 2 or 3 dimension table like we knew in the programs we use to write. But really, the LLM is so much more vast and complex. And the LLM is nothing more then numbers. Tokens are converted to a number and the tokens have a weighted value associated to them. This is a large over simplification and may not even be 100& correct.
Here i asked AI to take my wording and reword it to more accurate, but still at a high level of undersatanding.
Please don’t be for me. It’s a long time since I did any programming and now my research is only very simple stuff.
We organise events in our village and our former admin member did the posters in PowerPoint and converted them to jpg for me and the site. Our new replacement uses AI to generate the images but takes him much longer and appears more completed especially if details change such as hour or DJ to a group. Not been asked to do it myself but guess I would use libreoffice and its version of PowerPoint. Not AI
Well, the model (LLM) can and does make mistakes. But you have to remember the LLM has been trained on millions upon millions of sentences. The LLM can predict the next token for a word, part of a word, or part of a sentence with huge accuracy from the calculations the the highest possibility from all that training..
I type to an AI and ask it to complete the sentence.
“A cat is a”
So, after all the millions of examples the LLM was trained and the number of times it saw (feed tokens for) “A cat is” what was the most often next word and it was “pet”.
“A cat is a pet”
The next highest 4 tokens after cat was; cute, playful, furry, small.
A cat is cute.
A cat is playful.
A cat is furry.
A cat is small.
So that is how the model avoids mistakes or nonsense. It picks the next token (word) that appears most often.
Why didn’t the model come back and said 'A cat is round."
Because the probability is near zero. Not impossible but highly unlikely.
Answer form my understanding and a chat with AI. Example of 'A cat is" supplied by AI.
Often the tokens are subwords (‘tion’, ‘mis’, etc), so stringing together two subwords for example is much less likely to result in a misspelling than stringing together eight individual letters one by one. Like it’s easier to build a Lego train if you’ve been given two big pre-assembled pieces than having to build it bit-by-bit yourself. Plus the vast majority of the training data will be spelt correctly in the first place and models tend towards statistical regularity. If the correct spelling corresponds to higher-probability continuations, a decoder will usually pick it. If you increase temperature or allow sampling to be more exploratory, the decoder has a wider range of acceptable solutions to the next token, so you might see more spelling ‘mistakes’ per se.
Interestingly, I find that when I read AI generated fiction (typically not by choice!), you find that there are zero spelling errors, but you do get a lot of meaningless sentences… Words that are each correct in isolation and probably cluster close together in the training corpus, but when strung together are simply nonsense.
So in a large slab of text, like an ai book, there will be rare grammar errors or more correctly bad sequencing of tokens.
My guess is it is good at predicting the next token, not so good at predicting the next plus one, and so on.
or
does only ever predict ‘next’ then add that and loop around?
A very good question. I can see you have your thinking cap on. It also got me to thinking about how an AI responses to a question or input. I have been using 3 different AIs for different reason, but one was to see how the AI reacted. The 3 AIs I been using are Copilot, Gemini, and ChatGPT.
Over the past 3 months, I ave been using Copilot the most and have been setting rules and memory in it to response to me in the way I prefer. So taking you idea, I thought I would pose the simple “The cat is a” to all 3 AIs to see what the difference might be.
ChatGPT
Gemini response to The cat is a
Copilot
Different AI can give different responses. In this test the input an ambiguous text so different replies were expected. AIs response to a direct question or problem usually match pretty closely. That has been my experience so far.
We have talked a lot about LLM or model inside a modern AI. Here are the layers of a AI according to an AI. The LLM consists of items 2 thru 6 working as one unit.
And of course if you really want a good explanation of how AI doesn’t work, or other fun informative non-information, you can always consult Halupedia.com It is a genuine artificial idiot designed to produce hallucinations…
