Step by Step Practical RAG Optimization & Fine-Tuning Guide

I’ve worked on production-grade RAG systems across several domains internal knowledge agents, document Q&A, technical support, and code-aware retrieval. Each one taught me the same lesson the hard way:

The “use-everything” RAG stack is the most expensive mistake you can make.

Every technique costs you something latency, dollars, complexity, or a brand-new failure mode. Several of them actively hu**rt quality on the wrong workload. The teams that ship the best systems aren’t the ones with the longest tool list. They’re the ones who can tell you, for every component, exactly why it’s there and what they’d remove first.

This is the field guide I wish I had on day one. For each technique: what it does, when it earns its place, when to leave it out, and the cheaper or faster alternative.

First, the five principles everything rests on

1. Ground or refuse. Generate only from retrieved context. Never invent. Refuse politely when grounding is missing.

2. Bounded by design. One rewrite, one expansion, one judge. No recursion, ever.

3. Cite the anchor. Every claim links back to its canonical source.

4. Hybrid > single. BM25 for terms, vectors for meaning. Combine them don’t choose.

5. Low-ops or it dies. Keep knowledge fresh automatically (CDC / PR-driven ingest). Cache by content hash.

Hold those in mind every decision below is downstream of them.

The map: a RAG pipeline is a portfolio, not an architecture

Before we walk it, here’s the whole system. The rest of this guide follows a single query from raw document to cited answer.

Three layers, one database:

• Ingestion parse, chunk, embed (with a hash cache so unchanged content never re-embeds).

• Retrieval + Reasoning hybrid search → fusion → rerank → expand → synthesize → judge.

• Storage one Postgres (pgvector + FTS), same transaction, same backup, same pager.

Now let’s walk it, stage by stage.

Stage 1 Chunking: the unglamorous decision that determines whether any of this works

Nobody is excited about chunking. It still decides everything downstream. Bad chunks beat any model upgrade the wrong way. Mid-list and mid-code-fence splits poison retrieval long before the LLM ever sees a result. Get this right before you tune anything else.

Heading-aware, token-aware chunking (my default)

• Split at headings first, paragraphs second, word boundaries last.

• Lists are atomic never split a list mid-item.

• Code fences ≥ ½ target tokens become their own chunk.

• Target: 600 tokens (tiktoken cl100k_base), ~80-token overlap (tail of chunk N → head of chunk N+1).

• Stable chunk_content_hash = sha256(text + heading_path + section_anchor).

Fixed-size chunking (the lazy default)

• Wins when: flat-text corpora transcripts, OCR’d PDFs, ebooks. Nothing to split intelligently on.

• Loses when: structured Markdown, code, runbooks. You’ll cut citations in half and break procedures.

Late chunking

• What it does: embed the fu**ll docume**nt with a long-context model, then mean-pool over token spans to produce per-chunk embeddings.

• Why it’s interesting: chunk embeddings retain document-wide context coreferences like “it” and “this service” resolve correctly.

• Cost: long-context embedding model, more memory at ingest.

• Use when: high coreference density and your embedding model supports it.

Decision rule: structured Markdown/docs → heading-aware. Flat text → fixed-size with overlap. High-coreference prose → consider late chunking (or Contextual Retrieval, covered in Stage 5).

Stage 2 Retrieval: pick your lane(s)

BM25 (lexical Postgres FTS or OpenSearch)

• Does: ranks by TF-IDF with length normalization. Classical, deterministic.

• Wins when: proper nouns, acronyms, error codes (E_TIMEOUT_503), command names, file paths, IDs, version strings anywhere the exact token is the signal.

• Loses when: the user paraphrases. “Event replay” misses the page that says “reprocessing.”

• Alone when: internal code/log search users speak the same vocabulary as the corpus.

Dense vector search (pgvector, Pinecone, Qdrant)

• Does: encodes text into a 512–3072-dim vector; ANN index returns nearest neighbors.

• Wins when: conversational queries, paraphrases, “how do I…” exploratory asks.

• Loses when: exact tokens carry the meaning similarity can be confidently wrong on close-but-not-quite phrasing.

• Alone when: customer-facing assistants over conversational FAQs, recommendation-style retrieval.

Hybrid (BM25 + dense, fused)

• Wins when: mixed query distribution i.e. almost every real system.

• Latency cost is not 2×: both run in parallel. Your ceiling is max(bm25, vector), not the sum.

• Skip when: strict <50 ms SLA on a uniformly conversational corpus, or when one retriever is consistently adding noise.

Decision rule I use: any technic**al surface code, err**or codes, produ**ct names, runbooks → go hybrid. Pure prose (reviews, support transcripts) → den**se-on**ly is often enoug**h.

Stage 3 Fusion: combining two ranked lists

You now have two ranked lists with incompatible scoring models. How do you merge them?

Reciprocal Rank Fusion (RRF)

• Score-free: works on ranks, not raw scores. Critical BM25 scores and cosine distances live on different planets.

• Deterministic and embarrassingly cheap: pure Python, microseconds for top-K.

• Robust: if one retriever has a bad day, the other carries it.

• Use when: almost always. RRF was the single cheapest, biggest fix I made to any RAG I’ve built.

LLM-as-fuser

• Don’t. You’re adding 500–2000 ms and dollars to a problem RRF solves in microseconds.

Stage 4 Reranking: spend compute where it actually moves the needle

Retrieval is a bi-encoder problem query and passage are encoded independently: fast, but blind to token-level interaction. Reranking is where you pay to look at the pair together.

Cross-encoder reranker (bge-reranker-large, Cohere Rerank)

• Does: runs full cross-attention over [query, passage] pairs. Sees relationships a dot product physically cannot.

• Quality lift: in my experience, +15 to +30 NDCG on hard queries. More than HyDE. More than MQE. More than upgrading your embedding model.

• Cost: O(N) per query but only on top-K candidates after RRF, not the corpus.

• Latency: 50–200 ms for ~25 candidates on a single GPU. Plan for it.

• Skip when: strict <100 ms end-to-end SLA, or top-1 is already accurate (highly structured, exact-match-dominant corpora).

LLM rerankers (RankGPT, RankLLM)

• Reality check: slower (1–3 s), more expensive, and not consistently better than a good cross-encoder which was trained for exactly this task.

• Use when: zero-shot domains with no labeled data to fine-tune a cross-encoder, and latency is irrelevant.

No reranking

• Fine when: small corpus (<1000 docs) where RRF top-5 is already correct >90% of the time. Measure first don’t assume.

Stage 5 When retrieval falls short: recall tools (use surgically)

The temptation: “recall is bad, generate more queries wi**th an LL**M.” The reality: every LLM call in your hot path is 500–2000 ms and real money. Use these surgically.

Contextual Retrieval (ingest-time my first reach for unknown vocabulary gaps)

• Does: before embedding each chunk, prepend a short LLM-generated blurb situating it in its parent document (“This chunk is from the auth-service runbook, describing token-refresh…”). You embed context + chu**nk, so the vector carries disambiguating signal a bare chunk lacks.

• Wins when: chunks are individually ambiguous coreferences, generic headings, “it/this” references that only resolve at document scope.

• Cost: one-time, at ingest. No hot-path latency. Pairs well with BM25 over the same enriched text.

• Skip when: ingest cost is unacceptable, or your chunks already stand alone.

Multi-Query Expansion (MQE)

• Does: LLM generates 2–3 paraphrases; each runs through retrieval; results fused.

• Wins when: user vocabulary differs from doc vocabulary (“event replay” vs “reprocessing”).

• Cost: +1 LLM call, +N× retrieval calls.

• Cheaper alternative: a maintained glossary / synonym dictionary. Substitute terms client-side before retrieval no LLM, no latency. Covers ~80% of the value for known gaps; doesn’t generalize to unknowns.

HyDE (Hypothetical Document Embedding)

• Does: LLM writes a ≤80-word hypothetical answer; you embed th**at as the dense query.

• Wins when: the question is vague but the answer shape is predictable. “How does our auth work?” yields a hypothetical that retrieves better than the question.

• Loses when: the LLM confidently hallucinates a plausible-but-wrong direction now you’re retrieving near a wrong answer.

• Cost: +1 LLM call, ~500–1500 ms.

When to skip query rewrites entirely

• Coverage heuristic says you’re already there. Don’t rewrite a query that’s already answered. I skip MQE+HyDE on ~60% of queries because coverage ≥ 0.7 across ≥1–2 docs. That’s pure latency saved. (Covera**ge defined in Stage 7.)

Stage 6 Diversification and context expansion

Top-K retrieval has a dirty secret: the top 5 results are often five paraphrases of the same paragraph.

Maximal Marginal Relevance (MMR)

• Does: greedy selection trading relevance against similarity to already-selected results.

• Cost: pure local cosine math. No LLM. Microseconds.

• Tuning: λ = 0.7 relevance-leaning, good for focused factual queries. λ = 0.3–0.5 exploratory, “show me everything about X.”

• Earns its place when: top-K > 5 and your hits are likely near-duplicates (always true on chunked Markdown).

• Skip when: top-1 patterns (FAQ lookup). MMR with k=1 is a no-op.

Neighbor and link expansion

• Neighbor fetch (ord ± 1): after chunking splits “Step 4” from “Step 5,” neighbor expansion stitches procedural continuity back. Cheap SQL.

• Follow links (max=6): walk the relative links the author wrote. Author-written links are authoritative; LLM-invented retrieval paths are not.

• Use when: heading-aware chunked Markdown / structured docs. Skip on flat-text corpora.

Parent-document retrieval (the alternative pattern)

• Does: embed small chunks for retrieval precision, but return the parent section for synthesis context.

• Wins when: small chunks lose surrounding context the LLM needs to reason.

• Tradeoff: more tokens per query watch your context window and cost.

RAPTOR (hierarchical summarization)

• Does: recursively cluster + summarize chunks into a tree; retrieve at any level.

• Wins when: multi-document synthesis across a very large corpus (books, full codebases).

• Cost: massive ingest (cluster + LLM-summarize, repeatedly).

• Skip when: <1M-token corpus. The complexity isn’t worth it.

Stage 7 Reasoning bounds: the most important diagram in any RAG design

Unbounded agents are fun in demos and disastrous in production. Predictable latency, predictable cost, predictable behavior that’s the whole pitch. Here’s the bounded hot path I default to.

The bounded contract

• 1 rewrite (MQE+HyDE pass, only if coverage is low)

• 1 expansion (neighbors + links, then MMR)

• 1 judge (groundedness + citation check)

• 800 ms hard timeout on every external tool call

Coverage heuristic for early stopping

• Normalized [0, 1]. Deterministic. No LLM call.

• coverage ≥ 0.7 and ≥ 2 docs → skip rewrite, proceed to expand.

• Saves 1–2 seconds on ~60% of queries.

• Alternative: an LLM “can you answer from this context?” guard more accurate, but adds a round-trip. Use only if the heuristic is hurting quality.

The judge step

• One LLM call. Is every claim grounded in the provided context? Are the citations real?

• Three outcomes: pass (return), clarify (ephemeral prompt to the asker), decline (“I don’t know yet”).

• Why it matters: trust compounds. One hallucination in front of leadership can set adoption back a quarter.

• Skip when: demos, internal exploration. Never in user-facing or regulated environments.

What I deliberately do NOT do

• ReAct-style free-form loops: powerful in papers, uninvestigable in production. When something breaks at 2 a.m., you want to point at a node not unwind a tape.

• Self-RAG / CRAG (self-correction with retries): theoretically clean; in practice doubled latency for ~5% quality lift. Not worth it for most SLAs.

• Multi-hop reasoning agents: essential for so**me domains (literature review, investigations). For internal knowledge Q&A, one well-bounded pass with good retrieval beats a multi-hop loop.

The operations decisions that decide whether you stay up at 3 a.m.

One Postgres is enough resist the separate vector DB

• pgvector + FTS, joined to neighbor and link queries via plain SQL. Same transaction, backup, observability.

• Two systems = two pagers, two failure modes, two consistency stories.

• Split when: you actually hit pgvector’s limit (~10M+ vectors, high QPS). Most teams don’t. Measure first.

HNSW vs IVFFLAT

• HNSW: higher recall, faster queries, more RAM, slower build. Read-heavy, <100M vectors.

• IVFFLAT: less RAM, faster build, slower queries (recall depends on probes). Memory-constrained or frequent full rebuilds.

Cache by content hash. Always.

• Key: (model_id, content_hash, kind).

• Unchanged paragraphs never re-embed across PRs. Blue/green model swaps don’t require re-ingest.

• Embedding spend drops 90–95% in steady state.

Anything that needs a human to “remember to refresh” decays into stale results within two weeks. Leverage CDC!

The road not taken: techniques I considered and chose not to ship

GraphRAG / Knowledge-Graph RAG

• Does: extract entities + relationships into a KG; retrieve via graph traversal.

• Wins when: relational reasoning, multi-hop, comparing entities across docs.

• Cost: KG construction is LLM-heavy and continuous. Maintenance is hard.

• Skip when: queries are 90% factual lookup. Don’t pay graph cost for FAQ traffic.

• Revisit when: you have a clear class of multi-hop relational queries hybrid retrieval can’t answer.

Long-context instead of RAG (200K–2M token models)

• Does: stuff the entire corpus into context; skip retrieval.

• Wins when: small corpus (<100K tokens), latency- and cost-tolerant.

• Loses when: real corpora at scale cost scales linearly per query. Retrieval is still cheaper.

• The hybrid play: use RAG to retrieve a genero**us 50K-token context, then let long-context reason over it. Best of both for some workloads.

SPLADE / learned sparse retrieval

• Does: learns sparse representations that beat BM25 on some benchmarks.

• Skip when: you already have hybrid RAG. BM25 + dense + RRF already covers what SPLADE solves.

• Consider when: you can’t afford dense retrieval and need BM25-grade infra with better recall.

Matryoshka embeddings (multi-resolution)

• Does: train embeddings so the first n dims are themselves usable index high-dim, query low-dim, re-rank high-dim.

• Consider when: massive scale, latency-critical. For most systems, overkill.

What I’d tell myself before starting - Lessons Learned

1. Bound the agent on day one. Open-ended loops feel powerful in demos and become uninvestigable outages by week three.

2. Chunking beats any model upgrade. Get chunking right before you tune anything else.

3. RRF was the cheapest, biggest fix. Score-free fusion of two flawed retrievers beats one perfect one.

4. “I don’t know” beats a confident wrong answer. Trust compounds. So does its absence.

5. PR-driven ingest is the only ingest that survives. Humans don’t remember to refresh.

6. One Postgres is enough. Two systems is two pagers.

7. Latency you don’t spend is quality you don’t have to apologize for. Every hot-path LLM call is a tax make it earn its place.

RAG isn’t a single architecture. It’s a portfolio of techniques, each with a cost and a use case. The best systems aren’t the ones with the longest tool list they’re the ones whose builder can tell you, for every component, exactly why it’s there and what they’d remove first.

If you’re building one of these and want to compare notes, drop a comment or DM me. Always interested in what other practitioners chose differently and why.

---

Also published on LinkedIn if you’re building one of these and want to compare notes, that’s a great place to drop a comment.