Back to all writing in Foundations
Almost every modern AI system you have heard of — ChatGPT, Claude, Gemini, Llama, Mistral — is a transformer trained on text. The architecture behind them was described in a single 2017 paper, Attention Is All You Need, and the moving parts have not changed nearly as much as the marketing implies.
The Foundations pillar is the part of Defragging.ai you should read first. It explains what these systems are actually doing, in language that respects you without dumbing anything down.
Read these in order
- What is a token, really? — the unit a model sees instead of a word.
- Attention, in plain English — the one trick that makes transformers work.
Where this pillar is going
Future essays under Foundations will cover embeddings (how meaning gets turned into numbers), training vs. inference (why the bill comes twice), and context windows (why “just give it more context” is not free). When they are published they will be linked above.
Citations
- Attention Is All You Need (Vaswani et al., 2017) · accessed May 17, 2026
- Transformer (deep learning architecture) — Wikipedia · accessed May 17, 2026
- The Illustrated Transformer — Jay Alammar · accessed May 17, 2026
Essays in this pillar
- What is a token, really?
Models don't see words. They see tokens — chunks of bytes carved up by a learned vocabulary. Here is what that means in practice.
~2 min read
- Attention, in plain English
The one trick that makes transformers work, explained without matrices — and what changes once you understand it.
~2 min read