Your AI assistant costs $13,728 a month. It should cost $517.
Most AI assistants are not expensive because of the model. They are expensive because of what we feed the model. Here is the story, the math and the exact design.
The headline
$13,728
$517
per month, same questions, better answers
RAGCostAzure OpenAIVector search
Your AI assistant just read 50,000 tokens to answer "Where is the VPN guide?"
Then someone asks again. And again. 3,000 questions a day, every day.
That is how a simple HR helpdesk ends up costing $13,728 a month. The fix brought it to $517. Same questions. Same quality bar. Better answers.
01 · The storyMonday, week four: three bad surprises
The company in this story is made up. Every problem in it is common in real projects.
A company with 5,000 employees launches an internal AI assistant. It answers HR, IT and finance questions from 12,000 internal documents. Policies, guides, forms, old PDFs. Everything.
Week one is great. People love it. "How many leave days do I have left?" "How do I connect to the VPN from home?" Answers in seconds. Week four is different. Three things land on the CTO’s desk on the same Monday.
- 1Finance asks why a helpdesk now costs about $13,700 a month in AI fees.
- 2An employee follows the assistant’s answer on parental leave. The answer came from a 2022 policy that was replaced last year.
- 3A junior analyst asks about salary bands. The assistant quotes a document only HR managers should see.
The model was one of the best on the market. So what went wrong?
02 · The problemThe model was never the problem
The team built the simplest thing that works in a demo. A keyword search found the 10 most likely documents. All 10 went, whole, into the biggest model with the question. That is about 50,000 tokens for every single question.
- Cost. You pay for every token the model reads, every time. "Where is the VPN guide?" cost the same as a hard question about tax rules. At 3,000 questions a day, that adds up fast.
- Wrong answers. More context does not mean better answers. Stanford researchers found that models do best when the useful fact sits at the start or end of the input, and get much worse when it sits in the middle. The 50,000 token pile also held the 2022 leave policy next to the 2025 one. The model had no way to know which one was current.
- Access leaks. The search never asked who was asking. The prompt said "do not share confidential information". Nothing enforced it.
A prompt is a request, not a lock.
Killing the assistant is the wrong answer. McKinsey found that office workers spend nearly 20% of their week looking for internal information or the person who has it. A good assistant gives that time back. The fix is to feed the model less, and feed it the right things. That is what RAG does when it is built well.
03 · The ideaRAG in plain English: find first, then answer
Think of the model as a very smart librarian. The first design handed the librarian 10 whole books for every question and said "the answer is in there somewhere". RAG works the way a good library works. Find the right page first. Then answer from that page, and show where it came from.
- Chunks. Each document is split into small pieces of a few paragraphs. Every piece keeps a label: document title, version, date, department, and who is allowed to see it.
- Vectors. Each chunk is turned into a list of numbers that captures its meaning, an embedding. "Internet access when working from home" lands right next to "VPN setup guide", even with no word in common.
- A vector database. It stores those numbers and finds the closest ones in milliseconds. I have built this on pgvector, Qdrant and Pinecone. pgvector runs inside the Postgres you may already have.
- The answer step. The model gets the question plus the best few chunks. About 4,000 tokens instead of 50,000. It answers from those chunks and cites them.
Same model. Twelve times less to read. Better answers.
Search is where answers are won or lost
| Search | Great at | Fails at |
|---|---|---|
| Keyword (BM25) | Exact words: "Form HR-17", error codes, product names | Same meaning, different words. "Remote access" misses "VPN" |
| Vector | Meaning and synonyms | Exact codes and numbers. "HR-17" and "HR-71" look almost the same |
| Hybrid + rerank | Both: run keyword and vector together, then a reranker keeps the best 5 of the top 20 to 50 | Costs a little more per question, still far less than whole documents |
Anthropic tested this on real document sets. Adding context to each chunk plus keyword search cut failed retrievals by 49%. Adding a reranker on top cut them by 67%.
Two filters matter as much as ranking. Search only the current version of each policy, and only what this user is allowed to see. Both are checked in the database query itself, before the model sees a single word. That fixes the wrong answer and the leak from the story.
04 · The math$13,728 a month down to $517
Same company, same 3,000 questions a day, same quality bar. Only the design changes.
Whole documents to the big model50,000 tokens a question
$13,728
RAG, best chunks only4,000 tokens, big model every time
$1,584
RAG + cache + small model firstbig model for the hardest 20%
$517
Tuned: FAQ table and a cache that learns60% never reach a model
~$100
| Design | Tokens per question | Cost per question | Per month |
|---|---|---|---|
| 1. Whole documents to the big model | 50,000 | $0.208 | $13,728 |
| 2. RAG, best chunks only, big model every time | 4,000 | $0.024 | $1,584 |
| 3. RAG + cache + small model first | 4,000, fewer calls | $0.008 | $517 |
Step 2 alone cuts the bill by 88%. That is RAG doing its job: read less, read the right things. Step 3 cuts another 67%, with three habits.
- A semantic cache. "How do I reset my password?" gets asked hundreds of times a week. Answer it once, then reuse the answer for questions that mean the same thing.
- A small model first. A small, fast model answers most helpdesk questions well when it gets the right chunks. A study on model routing kept 95% of GPT-4 quality at up to 85% lower cost.
- Escalate only when needed. If the small model cannot find a source or is unsure, the big model takes over. Hard questions still get the best answer.
05 · The solutionThe blueprint
The top row runs only when a document changes. The bottom rows run for every question, and most questions stop at the cache or the small model.
On every document change
Documents
Policies, guides, forms, PDFs
Chunk and label
Title, version, date, who may see it
Embed
Meaning as numbers
Vector database
pgvector, Qdrant or Pinecone
On every question, cheapest step first
Question
User identity from login
FAQ table
Approved answers, no model
Semantic cache
Same meaning, reuse the answer
Small model
Hybrid search, rerank, filters, cites sources
Big model
Only when unsure or no source
Filters in the queryCurrent version only · Only what this user may see · Never in the prompt
Five mistakes that kill trust
| Mistake | What happens | Fix |
|---|---|---|
| Bad chunking | A leave table is split in half and the numbers lose their headings | Split by headings, keep tables whole, put the document title on every chunk |
| Stale documents | The 2025 policy is uploaded, the 2022 one is still searchable | Re-index on every change. Mark old versions as replaced and filter them out |
| The cache gives the wrong answer | "Leave policy for India" and "for the UK" look almost the same to an embedding | A strict match threshold, a cache key with country, role and document version, cleared when a document changes |
| Access rules in the prompt | A clever question gets around "do not share confidential data" | Take the user’s identity from login and filter by it inside the search query |
| No way to measure | Every change is a guess | Keep 100 real questions with known answers. Run them after every change |
The last one matters most. Without a test set, you cannot tell if a cheaper design is also a worse one.
06 · The planFour focused weeks
- 1Measure. Collect 100 real questions and their correct answers from HR, IT and finance. Measure today’s cost per question and how often answers are wrong.
- 2Search. Chunk documents by heading with labels for version, date and access. Embed them. Set up hybrid search in pgvector or Qdrant with version and access filters.
- 3Answer. Add a reranker. Make the model cite its sources. Put a small model first and escalate when it is unsure. Run the 100 questions.
- 4Save. Add the FAQ table, the semantic cache and prompt caching. Build a simple dashboard for cost, cache hits and escalations. Roll out to one department and compare.
07 · The resultCan it go to $100? Yes
$517 is the careful number. A tuned system goes lower: an approved FAQ table for every question asked more than about 10 times a month, a cache that learns, the best 3 chunks instead of 5, and only 10% of questions escalated, to a mid-size model. Now the only thing that matters is how many questions never reach a model.
$517
Start
$176
30%
$125
50%
$100
60%
$80
68%
- 60% is earned, not promised. Start with 30% in your plan. The FAQ table and cache grow from real traffic.
- Every FAQ answer has an owner. And a source document. When the policy changes, the answer is flagged for review. Personal questions like "my payslip" never go in it.
- Cheaper only counts if it is still right. Every change must pass the 100 question test set first.
The big model is the most expensive step. Call it last, not first.
Sources
Written by
Anuj Gupta
Hands-on software engineer and architect. 23+ years writing enterprise software on .NET, Azure and Oracle Cloud, faster with AI. 21,669 hours on Upwork at a 5.0 rating.