The model is not the hard part
Choosing a model takes an afternoon and the choice is rarely decisive; the frontier models are close enough that architecture dominates outcome. What actually determines whether a system succeeds:
- Can it find the right information? A retrieval problem.
- Is it allowed to show this user that information? An access control problem.
- How do you know a change made it better? An evaluation problem.
- What happens when it is wrong? A process and escalation problem.
- Does the unit economics work? A cost engineering problem.
None of those are machine learning problems. They are the problems of building reliable distributed systems that carry real transactions across trust boundaries — which is a discipline with decades of accumulated practice. The novelty is one non-deterministic component; the engineering is largely familiar.
The reference architecture
┌──────────────────────────────────────────────────────────┐
│ 5 GOVERNANCE eval suite · versioning · audit · ACL │ ← spans all
├──────────────────────────────────────────────────────────┤
│ 4 EXPERIENCE chat · search · workflow · API │
├──────────────────────────────────────────────────────────┤
│ 3 ORCHESTRATION prompts · tools · agents · routing │
├──────────────────────────────────────────────────────────┤
│ 2 RETRIEVAL chunking · embeddings · hybrid · rank │
├──────────────────────────────────────────────────────────┤
│ 1 DATA sources · ingestion · classification │
└──────────────────────────────────────────────────────────┘| Layer | Owns | Guide |
|---|---|---|
| 1 — Data | Ingestion, cleaning, classification, change detection | — |
| 2 — Retrieval | Chunking, embeddings, indexes, hybrid search, reranking | RAG · Vector search |
| 3 — Orchestration | Prompts, tool contracts, agent loops, routing | Agentic AI |
| 4 — Experience | The surface users touch and its failure behaviour | Conversational AI |
| 5 — Governance | Evaluation, versioning, audit, access control | AI governance |
Layer 5 is drawn spanning the rest deliberately. Governance is not a stage you reach after building; it is a property every layer must have, and retrofitting it is substantially harder than designing it in.
Start here
A sequence that produces something useful early and does not require rework:
| Phase | Build | You get |
|---|---|---|
| 1 | Golden set of 100–200 real questions with expected sources | The ability to measure anything at all |
| 2 | Ingestion + chunking + index, with ACL metadata from day one | A corpus you can search and scope |
| 3 | Hybrid retrieval with reranking, measured against phase 1 | Retrieval you can prove works |
| 4 | Grounded generation with citation and permission to refuse | A working assistant |
| 5 | Governance — versioning, audit trail, sampled review | Something you can operate and defend |
| 6 | Tools and agents, only where a workflow will not do | Action, not just answers |
Note that access control metadata belongs in phase 2, not a later hardening pass. Adding per-document ACLs after a corpus is embedded means a full re-index, and in the meantime you are running a system that cannot enforce permissions.
The .NET and Azure path
For an estate already on .NET and Azure, the components line up without introducing a parallel stack:
| Concern | Azure service | Note |
|---|---|---|
| Inference | Azure OpenAI Service | Regional deployment; data stays in your tenancy boundary |
| Retrieval | Azure AI Search | Hybrid search and semantic ranking built in rather than assembled |
| Orchestration | Semantic Kernel | Native .NET; tool calling and planning without a Python bridge |
| Identity | Entra ID | The same identity that scopes retrieval scopes the app |
| Secrets | Key Vault | Model keys, never in configuration |
| Observability | Application Insights | One trace across API, retrieval and inference |
The argument for staying inside one boundary is not vendor preference. A vector store outside your compliance perimeter is a data-residency conversation before it is an engineering one, and an AI system on separate identity infrastructure will drift out of alignment with the permissions it is supposed to honour. Keeping identity, network and audit consistent with the rest of the platform removes a whole category of problem.
Where a Python ecosystem component is genuinely better, expose it as a service behind the same boundary rather than splitting the platform.
Cost
AI systems fail budget review more often than they fail technically. The drivers, in rough order of impact:
| Driver | Lever |
|---|---|
| Context size per call | Rerank and pass fewer, better chunks — usually the largest saving |
| Model tier | Route simple requests to a smaller model; reserve the frontier model for hard ones |
| Agent step count | Bound the loop; a workflow instead of an agent where possible |
| Re-embedding | Re-index on change, not on a schedule |
| Retries | Cap them; unbounded retries multiply cost during an incident |
Track cost per resolved request, not cost per call. A cheaper model that fails and escalates is more expensive than a costlier one that resolves, once you count the human minutes. That single metric reframes most model-selection arguments.
Where this fits a platform business
Enterprise AI is most valuable where an organisation already has proprietary operational data and a support or knowledge burden around it. On a platform of any complexity — EV charging, IoT fleets, ERP — the recurring patterns are:
- Knowledge management — protocol specifications, partner integration guides, runbooks and incident history are exactly the corpus RAG suits: large, technical, frequently queried, and expensive to search manually.
- Intelligent support — grounding a support assistant in real operational data lets it answer about a specific session or site rather than in generalities.
- Workflow automation — triage, classification and summarisation over operational events, with bounded tool access for the routine actions that follow.
The common factor is that the value comes from the proprietary data, not from the model. Any competitor can call the same model; nobody else has your integration history, your partner runbooks, or your incident record. Architecture is what turns that asset into an answer.
Production lessons
- Measure before you tune. The golden set is the first artefact, not the last.
- Fix retrieval before prompts. Most model complaints are retrieval failures.
- Enforce access control inside the retrieval query. Never post-filter.
- Version prompts, models and indexes together, and pin model versions.
- Log the retrieved chunk IDs on every response. Without them nothing is diagnosable.
- Prefer a workflow to an agent wherever the flowchart can be drawn.
- Give the system permission to say it does not know.
- Track cost per resolved request.
- Design governance in from the start. Retrofitting it is far more expensive.
Frequently asked questions
Five layers: data ingestion and classification, retrieval, orchestration of prompts and tools, the user-facing experience, and governance spanning all of them. The model itself is the part you do not build — outcomes are decided by retrieval quality, access control, evaluation and cost engineering.
With a golden set of 100 to 200 real questions and their expected sources. Teams that build the pipeline first spend weeks tuning prompts on what turns out to be a chunking problem, because they had no way to see retrieval was failing.
Azure OpenAI for inference, Azure AI Search for hybrid retrieval, Semantic Kernel for orchestration in native .NET, Entra ID for the identity that scopes retrieval, Key Vault for model keys, and Application Insights for one trace across API, retrieval and inference. Keeping identity, network and audit consistent with the rest of the platform removes a category of compliance problems.
Context size per call is usually the largest, followed by model tier, agent step count, re-embedding frequency and retry behaviour. Track cost per resolved request rather than per call — a cheap model that fails and escalates costs more than an expensive one that resolves.
Usually not. Fine-tuning changes behaviour, format and tone; retrieval supplies knowledge. Since enterprise systems need citation and per-user access control, and a fine-tuned model can offer neither, retrieval is the default and fine-tuning is an addition where behaviour specifically needs changing.