RAG Architecture on Azure OpenAI

Why retrieval rather than fine-tuning

The instinct when a model does not know your domain is to train it on your domain. For most enterprise problems that is the wrong reach.

	RAG	Fine-tuning
Adds	Knowledge the model can cite	Behaviour, format, tone
Update cost	Re-index a document, minutes	Retrain, hours to days
Attribution	Natural — you know which source was used	None — knowledge is diffuse in weights
Access control	Enforceable at retrieval time	Impossible — weights do not have ACLs
Stale data	Fix the index	Retrain

Access control is the argument that ends the debate in enterprise settings. If a document is confidential to one department, a fine-tuned model has already absorbed it and will surface it to anyone who asks the right question. A retrieval system filters by the caller's permissions before the model ever sees the text. Any system touching HR, finance or customer data has to be RAG for this reason alone.

The two are not exclusive. Fine-tune for how the model should behave; retrieve for what it should know.

The pipeline

INGEST                          QUERY TIME
  │                                │
  ├─ parse & clean                 ├─ rewrite query
  ├─ chunk                         ├─ embed query
  ├─ embed chunks         ┌───────►├─ vector search  ─┐
  ├─ enrich with metadata │        ├─ keyword search ─┤ hybrid
  └─ index ───────────────┘        ├─ merge & rerank ─┘
                                   ├─ assemble context (with citations)
                                   ├─ generate
                                   └─ verify & attribute

Six of those eight query-time steps are retrieval. That ratio is the point: the generation call is the cheap, easy part, and it is where most teams spend their effort.

Chunking is the decision that matters most

Chunking determines what can ever be retrieved. A fact split across two chunks is a fact your system cannot answer with, no matter how good the model is.

Strategy	How	Suits
Fixed size	N tokens with overlap	Uniform prose; the naive default
Structural	Split on headings, sections, list items	Technical docs, specs, manuals — usually the best starting point
Semantic	Split where embedding similarity drops	Unstructured narrative text; more expensive to build
Parent-child	Embed small chunks, return the larger parent	When precision and context are both needed

Parent-child is under-used and solves a real tension. Small chunks retrieve precisely because they are focused; large chunks answer well because they carry context. Embedding the small chunk and returning its parent section gives you both, at the cost of some index complexity.

What to get right

Never split a table. Half a table is worse than no table — the model will confidently read the wrong column.
Keep headings with content. A chunk that says “must not exceed 30 seconds” without its heading is unusable.
Overlap by 10–20% so a fact near a boundary appears whole in at least one chunk.
Carry metadata on every chunk — source, section, version, effective date, ACL. This is what makes filtering and citation possible later.

Hybrid retrieval, and why pure vector is not enough

Vector search finds semantic similarity. It is poor at exact tokens — part numbers, error codes, protocol field names, version strings. Ask a pure vector system about error code 4002 and it will happily return passages about error handling generally.

Keyword search (BM25) is the opposite: exact on tokens, blind to paraphrase.

Run both and fuse the results. Reciprocal Rank Fusion is the standard approach and needs no tuning:

score(d) = Σ  1 / (k + rank_i(d))          k ≈ 60

vector rank 1, keyword rank 12  →  1/61  + 1/72  = 0.0303
vector rank 8, keyword rank 2   →  1/68  + 1/62  = 0.0308  ← ranks higher

A document that both methods consider reasonable beats one that a single method loved. On a technical corpus — specifications, API docs, protocol definitions — hybrid retrieval is typically the single largest quality improvement available, and it costs one extra query.

Reranking

Retrieval returns 50 candidates cheaply. A cross-encoder reranker scores each against the query properly and returns the best 5.

The difference is architectural. A bi-encoder embeds query and document separately, so similarity is a dot product between vectors computed in isolation. A cross-encoder reads query and document together and scores the pair, which is far more accurate and far too slow to run over a whole corpus.

The pattern is therefore: retrieve wide and cheap, rerank narrow and accurate. It adds latency — typically 100–300 ms — and it is usually worth it, because what lands in the context window is what the answer is made of.

Fewer, better chunks beat more chunks. Models attend unevenly across a long context — material in the middle of a large context window is measurably less likely to be used than material at the edges. Passing 20 mediocre chunks does not hedge your bets; it dilutes the good ones and raises cost. Five well-ranked chunks generally outperform twenty unranked.

Grounding and citation

An enterprise answer that cannot be traced to a source is not usable. Structure the context so attribution survives into the output:

[1] OCPI 2.2.1 §4.3 — Credentials (updated 2026-03-11)
The registration handshake replaces Token A with a permanent pair...

[2] Internal runbook — Partner onboarding (updated 2026-07-02)
Before returning Token C, confirm the partner's versions endpoint...

Answer using only the sources above. Cite as [1], [2].
If the sources do not contain the answer, say so.

Three things make this work in practice:

Number the sources and instruct citation explicitly.
Include the version or effective date in the header so the model can prefer current material and a reader can judge staleness.
Give explicit permission to fail. “If the sources do not contain the answer, say so” is the single highest-value sentence in a RAG prompt. Without it, a model asked a question its context cannot answer will produce something plausible.

Evaluation: two systems, measured separately

The most common evaluation mistake is scoring end-to-end answers and trying to reason backwards about what went wrong. Retrieval and generation fail differently and must be measured apart.

Layer	Metric	Answers
Retrieval	Recall@k	Was the right chunk retrieved at all?
Retrieval	MRR / nDCG	Was it ranked highly?
Generation	Faithfulness	Is the answer supported by the retrieved text?
Generation	Answer relevance	Does it address the question asked?
End to end	Correctness	Is it actually right?

The diagnostic that matters: if recall@k is low, no amount of prompt engineering will help. The material was never in the context. Teams routinely burn weeks tuning prompts on what is a chunking or indexing problem.

Build a golden set of 100–200 real questions with known correct sources before you build anything else. It is the regression suite — without it you cannot tell whether a change helped, and every subsequent decision is guesswork. This is also the foundation of AI governance.

Failure modes

Symptom	Usually is	Fix
Confidently wrong answer	Retrieved the wrong chunk	Check recall first, not the prompt
“I don't know” on answerable questions	Chunking split the fact	Structural chunking, larger overlap, parent-child
Right doc, wrong detail	Table or list was split	Never split structured content
Cites an outdated policy	No recency signal or stale index	Version metadata; filter or boost on effective date
Good on short questions, poor on complex ones	Single retrieval pass	Query decomposition — retrieve per sub-question
Answers leak across tenants	Filtering applied after retrieval	Filter during search, never post-hoc

The last row is a security defect, not a quality one. Retrieving broadly and then discarding unauthorised results means the data was already in your process, and a change in code order or a caching layer turns it into a live leak. Push the ACL into the search query itself.

Production lessons

Fix retrieval before prompts. Most “model” problems are retrieval problems.
Build the golden set first. Without a regression suite, you are guessing.
Use hybrid retrieval on technical corpora. Usually the biggest single quality win.
Enforce access control inside the search query, never as a post-filter.
Carry version and effective date as metadata so answers can prefer current policy.
Log the retrieved chunk IDs with every answer. When someone disputes an answer, this is what lets you reconstruct it — the same discipline as keeping raw payloads in a protocol integration.
Re-index on source change, not on a schedule. A nightly rebuild means a day of answering from withdrawn policy.
Cap context deliberately. More chunks costs money and dilutes attention.

Frequently asked questions

What is RAG?

Retrieval-augmented generation: instead of training knowledge into a model's weights, relevant documents are retrieved at query time and placed in the model's context so it can answer from them and cite them.

Should you use RAG or fine-tuning?

Use RAG for knowledge and fine-tuning for behaviour. RAG updates in minutes by re-indexing, supports citation, and — decisively for enterprise — allows access control to be enforced at retrieval time. A fine-tuned model has already absorbed confidential documents and cannot filter by who is asking.

Why does RAG give confidently wrong answers?

Almost always because retrieval returned the wrong material and the model answered faithfully from it. Measure recall@k before touching the prompt — if the correct chunk was never retrieved, no prompt engineering can fix it.

What is hybrid retrieval?

Running vector search and keyword search together and fusing the rankings, typically with Reciprocal Rank Fusion. Vector search handles paraphrase but is poor at exact tokens like error codes and part numbers; keyword search is the reverse. On technical corpora hybrid retrieval is usually the largest single quality improvement available.

How many chunks should you put in the context?

Fewer, better ones. Models attend unevenly across long contexts, so material in the middle of a large window is less likely to be used. Five well-reranked chunks generally outperform twenty unranked ones, at lower cost.