The model was confident. The PDF disagreed.

I pasted a question about a PDF into a chat model. It answered like it had read the thing cover to cover. It hadn’t. The answer was smooth, polite, and wrong in a way that would get someone fired if they shipped it.

That’s the whole RAG pitch, stripped of the hype: stop asking the model to remember your documents. Go fetch the relevant bits, then make it answer from those.

Why “just put it in the prompt” falls apart

Context windows got bigger. People got lazy. I did too.

Stuffing an entire handbook into the prompt works until:

• the doc is longer than the window
• you have many docs and don’t know which one matters
• you care about which page the claim came from
• cost/latency starts to hurt

Retrieval is just: find the few paragraphs that might matter, then generate. The model becomes a writer with a stack of sticky notes, not an oracle.

The pipeline I actually built

Four steps. Each one can quietly ruin everything.

1. Extract. Get text out of the PDF. Normal text PDFs are fine. Scans are not text — you need OCR first, and OCR will miss table structure, and you will hate tables.

2. Chunk. Split into pieces. Too big → retrieval is blunt. Too small → you lose the sentence that made the paragraph make sense. Overlap helps so a definition doesn’t die at the boundary.

3. Embed + store. Each chunk → vector. I used Chroma because it was the path of least resistance for a side project. Similar meaning → similar direction. In theory.

4. Retrieve + answer. Embed the question, pull nearest chunks, stuff them into the prompt, generate. Tell the model hard: answer only from the context. If it’s not there, say you don’t know.

Step 4 is the part everyone demos. Step 2 is the part that decides if the demo survives contact with real files.

What I got wrong

Fixed-size character chunks. Worked on the sample PDF. Broke on the real one — a definition split across two chunks, retriever returned the useless half, model confidently invented the rest.

Semantic-ish splits (paragraphs / sections) with overlap fixed more accuracy than any system prompt rewrite. I was annoyed by how unglamorous that was.

Second mistake: trusting top-1 similarity. Closest chunk can still be irrelevant. Sometimes the answer simply isn’t in the corpus. A score floor + an allowed “I don’t know” made the system more useful, not less. Users prefer a shrug over a confident lie. At least the ones I asked.

The boring conclusion

Hybrid setup: docs as source of truth, model as the thing that phrases the answer. Precision comes from grounding, not from the model being clever.

If your RAG is bad, don’t start by swapping models. Look at your chunks. Look at what actually got retrieved for a failing query. Nine times out of ten the bug is upstream of the LLM call.