Introduction

When you type a question into a large language model (LLM) and a coherent answer appears almost instantly, a dense sequence of mathematical operations is running behind the scenes. The model does not “understand” words the way humans do, it converts text into vectors of numbers, computes the relationships between words through the attention mechanism, and then projects the result back into vocabulary space to predict the next word. This article dissects that entire pipeline in order, using a GPT-style (decoder-only Transformer) architecture as a concrete example with model dimension \(d_{\text{model}} = 768\), 12 attention heads, per-head dimension \(d_k = 64\), a feed-forward inner size of 3072, and a vocabulary of roughly 50,000 tokens.

The flow below traces a single input sentence until the model produces its first output token, then repeats the cycle autoregressively until the answer is complete.

Part 1: From Text to Numbers

Step 1: Tokenization

The input text is split into subword units and mapped to numeric IDs based on the model's vocabulary. Subword splitting, for example via Byte Pair Encoding, lets the model handle rare or unseen words without running out of vocabulary.

• Input: "Hello, how are you?"
• Output (illustrative): [2572, 3058, 2573, 102]

Step 2: Embedding Lookup

Each token ID is used to retrieve the corresponding row from the embedding matrix of size \(\text{vocab\_size} \times d_{\text{model}}\). The result is a dense vector representing the token's meaning.

• Output: 4 vectors, each 768-dimensional.
• Example: \(\mathbf{E}_1 = [0.5,\ 0.2,\ -0.3,\ \dots,\ 0.7]\)  (768 values).

Step 3: Adding Positional Encoding

A Transformer processes all tokens in parallel and has no built-in notion of order. Positional information is therefore added directly to the token embeddings before they enter the Transformer blocks. This is what lets the model know that “how are you” differs from “you are how”.

• Output: 4 initial hidden-state vectors, each 768-dimensional.

Key note. The embedding dimension equals the model's hidden dimension (\(d_{\text{model}} = 768\)). There is no projection that “expands” the embedding into a different dimension before the Transformer blocks, the 768-vector from Step 2 becomes the hidden state that every block processes in turn.

Part 2: The Heart of the Transformer: Self-Attention

Step 4: Query, Key, and Value Projection

Each hidden-state vector is projected into three distinct vectors, Query (Q), Key (K), and Value (V), by multiplication with trained weight matrices. Q, K, and V are the raw material of the attention mechanism.

• Input: 768-dimensional hidden states (4 tokens).
• Output: three sets of vectors (Q, K, V). With 12 heads, the 768 dimensions split into 12 × 64; each head operates on a 64-dimensional slice (\(d_k = 64\)).

Step 5: Attention Scores

For each token, the model measures how relevant every other token is by computing the dot product between that token's Query and all tokens' Keys, scaled by \(\sqrt{d_k}\) to keep the values stable.

Output: a 4×4 matrix (similarity of each token to all tokens). Example:

Step 6: Causal Masking + Softmax

In a GPT-style generative model, a token may only “see” itself and the tokens before it, not future tokens. So before softmax, future positions in the score matrix are set to \(-\infty\) (the causal mask). Softmax then turns the scores into a probability distribution that sums to 1 across each row.

Output: a 4×4 probability matrix. Example:

Step 7: Attention Output

The probability weights are used to average the Value vectors: tokens with larger weights contribute more. The outputs of all heads are then concatenated and projected back through the output matrix \(W^{O}\).

• Output: 4 vectors, each back to 768 dimensions.
• Example: \(\mathbf{z}_1 = [1.8,\ 2.9,\ 0.2,\ \dots,\ 1.5]\).

Step 8: Residual + Layer Normalization (Attention Sub-layer)

The attention output is added back to its input (a residual / skip connection) and then normalized. The residual keeps gradients flowing through deep networks, while normalization stabilizes the distribution of values across layers.

Part 3: The Feed-Forward Network

Step 9: Feed-Forward Layer 1 (Expansion)

Each token position is processed independently by a two-layer network. The first layer expands the dimension from 768 to 3072 (a 4× expansion), giving the model a richer representation space.

• Output: 4 vectors, each 3072-dimensional.

Step 10: Non-linear Activation

A non-linear activation is applied so the network can model non-linear relationships. GPT- and BERT-style Transformers typically use GELU, while the original Transformer used ReLU.

where \(\Phi(x)\) is the standard Gaussian cumulative distribution function. Unlike ReLU, which clips all negative values to zero, GELU weights inputs smoothly.

Step 11: Feed-Forward Layer 2 (Projection Back)

The second layer projects the representation back to the original model dimension, from 3072 to 768, so it can be added to the residual path.

where \(g\) is the activation output from Step 10.

Step 12: Residual + Layer Normalization (Feed-Forward Sub-layer)

Once again, the feed-forward output is added to its input and normalized, producing this block's final hidden state.

Repeat Steps 4–12 for every Transformer block. A small GPT-2-style model has 12 blocks; larger models can have dozens to hundreds, each containing Attention + Feed-Forward + Normalization. One block's output becomes the next block's input.

Part 4: From Numbers Back to Words

Step 13: Output Projection

After the final block, the hidden state is projected into vocabulary space through an output matrix (which in many models is weight-tied with the embedding matrix). The result is a raw score (logits) for every token in the vocabulary.

• Output: a \(\approx 50{,}000\)-dimensional vector for the last token position.

Step 14: Final Softmax

The logits are turned into a probability distribution over the entire vocabulary. Every value now lies between 0 and 1, and together they sum to 1.

• Example: \([0.001,\ 0.002,\ 0.85,\ 0.05,\ \dots]\), the value 0.85 is the probability of the candidate token “I”.

Step 15: Token Selection, Decode, and Repeat

The next token is chosen from the probability distribution. The simplest strategy is argmax (take the highest probability); in practice, sampling strategies such as temperature, top-k, or top-p are often used to control how creative the output is.

The selected token ID (say, 5021) is then decoded back into text (“I”). This token is appended to the input sequence, and the whole prediction cycle repeats to generate the next token, the autoregressive process, until an end token [EOS] appears or the maximum length is reached.

Final answer: “I am learning machine learning.”

Summary of the Flow

In short, LLM inference is a cycle of text → numbers → numbers → text. Text is turned into vectors (Steps 1–3), processed repeatedly by Transformer blocks that combine self-attention and feed-forward layers (Steps 4–12), and then projected back into the vocabulary to predict one token (Steps 13–15). That token is fed back in as input, and the cycle continues until the full answer is formed. It is the simplicity of the underlying operations, matrix multiplication, softmax, and normalization, repeated billions of times that, surprisingly, produces coherent language.

References

1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30. arXiv:1706.03762. https://arxiv.org/abs/1706.03762
2. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019, 4171–4186. DOI: 10.18653/v1/N19-1423
3. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016, 1715–1725. DOI: 10.18653/v1/P16-1162
4. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer Normalization. arXiv:1607.06450. https://arxiv.org/abs/1607.06450
5. Hendrycks, D., & Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv:1606.08415. https://arxiv.org/abs/1606.08415

Note: the Attention Is All You Need and Layer Normalization papers were published via NeurIPS/arXiv and do not carry a formal DOI; both are identified by their stable arXiv numbers. The DOIs for BERT and Subword Units were verified to resolve via doi.org.