AI Transformation · Glossary

Retrieval-Augmented Generation (RAG)

Updated October 25, 2026

Retrieval-augmented generation (RAG) is a design in which an artificial intelligence (AI) system first finds the passages in a company's own documents that look relevant to a question, then hands them to the model with the question, so the model answers from text it can read rather than from what it remembers. The documents are split into chunks and encoded into a vector store, a searchable index of document chunks, and the chunks closest in meaning to the question go to the model. It exists because what a model learned in training is fixed when it ships, expensive to update, and never contained the company's own data.

The catch is that relevant-looking is not relevant. Andrej Karpathy, a founding member of OpenAI, explained in a public talk in early 2025, as a summary of that talk reports it, that knowledge inside the model is a vague recollection, while text placed in front of it is working memory, and retrieval converts one into the other. The discipline is not "paste everything" but "paste the right things".

Where does RAG go wrong?

In what it retrieves. A June 2026 KDnuggets article describes three failures, from experience rather than measurement:

1. Retrieval irrelevance, which that article ranks the most common: passages that share the question's vocabulary score high, answer nothing, and get blended into a confident, wrong answer.
2. Context poisoning: where several versions of a policy exist, retrieval returns both, and the model does not flag the contradiction.
3. The chunk trade-off: small chunks retrieve precisely, large chunks keep meaning intact, and every design picks one.

The retrieved source can itself be wrong, out of date, or the wrong document, so RAG lowers the risk of AI hallucination without removing it. A cited answer is one you can check; the checking still has to happen.

Retrieval is also one option among several. The same article, and separately Andrew Ng, say to match the method to the question. If the documents fit in what the model can read at once, load them and skip retrieval; summarize before retrieving when they do not; send simple lookups and broad questions down different routes; and use a knowledge graph, a map of the links between documents, when the answer lives in those links.

For company facts, retrieval and organized memory come before fine-tuning, which changes how the model behaves more than what it knows. An AI agent, a model using tools in a loop, can call retrieval as one of its tools; the What is an agentic workflow? article describes the pattern of work such an agent runs. In AI transformation planning, ask which method fits the question before anyone builds an index.