A large language model generates text one piece at a time, predicting a likely next piece based on everything that came before, over and over until a full response emerges. That simple description undersells how much useful behavior comes from doing this prediction extremely well at enormous scale, but it's the actual mechanism underneath what looks like understanding.

The core mechanism: predicting the next token

Text gets broken into tokens, roughly word-sized or sub-word-sized pieces, and an LLM is trained to predict which token comes next given everything before it. At each step the model produces a probability for every possible next token, and one is selected (often, but not always, the most likely; a setting called temperature controls how much randomness is allowed). Generating a response means repeating this step, with each new token added to the context for the next prediction, until the response is complete. This is the entire generative mechanism, however sophisticated the output ends up looking.

Why this simple mechanism produces such capable behavior

During training, a model is exposed to enormous amounts of text and learns statistical patterns about language, facts, reasoning structures, and conventions. Predicting the next token well turns out to require the model to implicitly encode a great deal of real knowledge about how language and reasoning work. The model isn't retrieving stored sentences; it's generating new sequences from learned patterns, which is why it can respond coherently to inputs it has never seen verbatim.

Where the model's actual knowledge comes from

An LLM's knowledge comes from patterns in its training data, which has a fixed cutoff point. The model doesn't know about anything after that, and it has no live internet access unless it's connected to a tool that provides it. This is exactly why RAG exists: to give a model current or specific information it wasn't trained on, by retrieving it and providing it as part of the input rather than expecting the model to already know it.

Why this mechanism also explains the model's real limitations

Why models can state incorrect things confidently

Because the model is generating a statistically plausible continuation, not looking up a verified fact, it can produce a fluent, confident-sounding statement that's simply wrong (a hallucination) when the most plausible-sounding continuation happens not to be true.

Why context matters so much for output quality

Every prediction is conditioned on everything provided as context (the context window), so what's in that context directly shapes what the model generates. That's why how a prompt is written has such a large effect on output quality.

Why some models "think" before answering

Reasoning models generate intermediate reasoning steps before producing a final answer, using the same next-token mechanism to work through a problem step by step. This tends to improve accuracy on problems that genuinely benefit from multi-step reasoning.

A useful mental model
An LLM is best understood as an extremely capable pattern-completion engine, not a database with a search function or a mind with beliefs. That framing explains both why it's remarkably capable at language and reasoning tasks and why it can be confidently wrong in ways a database lookup never would be.

How we approach this

We design AI systems around what LLMs actually are: powerful pattern-completion engines whose output quality depends heavily on context and grounding, not omniscient databases. That shapes real decisions about when to use RAG, how to structure prompts, and where human review genuinely matters.