Call the big model last: a RAG design, layer by layer
Your AI assistant just answered "Where is Hall B?" for the 1,500th time. On your most expensive model. A cost-aware design for an event platform: the problem, the fix, the math, and what breaks in production.
The headline
$419
$162
per event, 87% fewer big model calls
RAGModel routingMulti-tenantEvent platforms
Your AI assistant just answered "Where is Hall B?" for the 1,500th time. On your most expensive model.
Event platforms are close to my heart: I spent years building one that carried 10,000 people across 15 live events on a single day. Peaks are brutal, the same questions arrive in waves, and every vendor’s data must stay that vendor’s. So I ran the numbers for an AI assistant on a 2 day event.
1,500
attendees
25
vendors
35
admins
~17,000
questions in 2 days
01 · The problemThree kinds of users. One expensive answer path
Attendees · 88%
15,000 questions
Lookup
Vendors · 6%
1,000
Private search
Admins · 6%
1,050
Reasoning
- Attendees "What time is lunch?" Same answer for everyone. Asked hundreds of times. A lookup.
- Vendors "Which visitors asked about pricing?" Must only ever see their own booth data. A private search.
- Admins "Why did afternoon check-ins drop?" Reasoning across sessions, scans and feedback. Real analysis.
88% of the questions are simple. 100% of them pay big model prices.
02 · The mathSame questions. Same quality bar. 61% less
Everything on the big model
$419
per event · 17,050 big model calls
Routed design
$162
per event · 2,250 big model calls
87%
fewer big model calls
$25.7K
saved per 100 events a year
100%
of admin questions still get the big model
| Assumption | Value |
|---|---|
| Questions | Attendees 10 each, vendors 40 each, admins 30 each |
| Tokens | 4,000 in / 300 out per answer; admin answers 12,000 in / 800 out |
| Routing | 30% attendee cache hits, 10% attendee and 15% vendor escalation |
| Prices | Claude Opus 5.5 $4 / $20, Haiku 4.5 $1 / $5 per million tokens, checked 2 October 2026 |
03 · The solutionRoute every question to the cheapest step that answers it well
1 · Gateway
Authenticate. Role and vendor ID come from the token
2 · Cache
Exact match, then semantic. A hit answers now
3 · Router (small model)
Returns a lane and a confidence score
Lane A · FAQ
Agenda, venue, session FAQ · small model
Lane B · Vendor
Search filtered to one vendor ID · small model
Lane C · Analysis
Hybrid search and rerank across all event data · big model
4 · Grounding check
No source or low confidence goes to the big model
5 · Answer and log
Route, tokens, cost and escalation recorded
- 1A cache answers repeated questions before any model runs. Exact match first, then semantic.
- 2A small model reads the question and picks a lane, with a confidence score.
- 3Vendor questions search only that vendor’s data. The vendor ID comes from the auth token, never from the prompt.
- 4A small model answers, and it must cite its sources.
- 5Only hard or uncertain questions reach the big model.
04 · The evidenceCheap steps are good enough far more often than you think
68.8%
fewer API calls with a semantic cache
67%
fewer failed retrievals with hybrid search and reranking
95%
of GPT-4 quality kept by routing, at up to 85% lower cost
Research headlines promise up to 98% savings (FrugalGPT, Stanford). My event model says 61%. I would plan with 61%, then let real traffic earn the rest.
05 · What breaksThe diagram is easy. These are the hard parts
| Failure | Fix |
|---|---|
| Cache gives the wrong answer: "Lunch on day 1?" and "Lunch on day 2?" look almost the same to an embedding | Exact match first. Strict similarity threshold. Cache key includes event, day and role |
| One vendor sees another’s leads: the filter came from the prompt, a clever question got around it | Vendor ID comes only from the auth token. The filter is applied in the database query |
| Stale answers after a change: a session moves to Hall C, the cache still says Hall B | Clear related cache entries on every agenda or content update |
| A small model is confidently wrong: self-reported confidence is not proof | Grounding check: no source, no answer. Review a sample of answers every week |
| Escalation doubles latency: hard questions pay for the small model and then the big one | Send known hard question types straight to the big model. Cascade only the uncertain middle |
06 · How to build itWhat goes in each layer
| Layer | What to add | Tools |
|---|---|---|
| L0 Data | Chunk agenda, FAQ, scans, check-ins. Tag each chunk with event, vendor and role | pgvector, Qdrant, Pinecone, embeddings |
| L1 Gateway | Authenticate. Take role and vendor ID from the token claims only | ASP.NET Core, JWT, rate limiting |
| L2 Cache | Exact match, then semantic. Keys scoped by event, day, role. Clear on content change | Similarity threshold, TTL, invalidation events |
| L3 Router | Small model returns lane and confidence. Rules first for known question types | Small classifier model, structured JSON output |
| L4 Retrieval | Vector plus keyword search, then rerank. Tenant filter enforced in the store | Qdrant is_tenant index, Pinecone namespace per vendor, Postgres RLS |
| L5 Generation | Small model first. Grounding check. Escalate on missing sources or low confidence | Model cascade, citations required |
| L6 Cost | Cache the shared event context in the prompt. Run overnight admin reports as batch | Cache reads at 0.1x input price, batch 50% off |
| L7 Observability | Cost, route, cache hit and escalation rate per role. Review weekly and tune | Application Insights, KQL dashboards |
07 · The resultLower cost is only one of the wins
- Cost: 61% lower in the model. 87% fewer big model calls for the same questions.
- Speed at peak time: cache hits and small models reply faster when 1,500 people ask at once.
- Vendor data stays private: isolation is enforced by the database, not by the prompt.
- A bill you can predict: cost per role is measured every week, so you can tune it.
The routing around the model matters more than the model. Call the big model last, not first.
Sources
Written by
Anuj Gupta
Hands-on software engineer and architect. 23+ years writing enterprise software on .NET, Azure and Oracle Cloud, faster with AI. 21,669 hours on Upwork at a 5.0 rating.