RAG Implementation Cost in Production

Where do RAG projects actually fail?

RAG projects fail when the retrieved passages lack the answer and nobody measured it. Chunking, evals and index refresh drive RAG implementation cost, and budgets built around the demo miss all three.

Retrieval-augmented generation, introduced in a 2020 paper from Lewis and colleagues, pairs a retriever with a generator. The generator can only be as good as what the retriever hands it. That gap is why costs surprise teams six months after launch.

Why does the demo work and production does not?

A demo runs on a few dozen clean documents and a dozen questions the builder wrote. Production brings scanned PDFs, tables, duplicated versions of the same policy, and users who ask things in words the documents never use.

Retrieval quality degrades with every one of those. The failure is silent: the model still produces fluent text, just grounded in the wrong passage.

Where do the failures cluster?

Four places account for most of them:

• Chunks split a fact from its context.
• The right chunk is retrieved but ranked below the cut-off.
• Long contexts bury the answer, a position effect documented in the Lost in the Middle paper.
• The index goes stale.

Each maps to a section below, and each has a cost that rarely appears in a vendor quote.

How do chunking decisions change accuracy more than model choice?

Chunking determines what the retriever can find at all. A chunk that separates a rule from its exception, or a table from its header, cannot be retrieved correctly by any embedding model. Fix chunking before you compare models.

How should you size and split chunks?

Start from document structure, not token counts. Split on headings, list boundaries and table rows, then enforce a maximum size as a fallback. Fixed 500-token windows are a reasonable baseline to beat, not a design.

Overlap hides boundary errors but inflates the index and returns near-duplicate chunks that crowd out diverse results. Keep it small and deduplicate at query time.

Does adding context to each chunk help?

Anthropic's contextual retrieval write-up describes prepending a short, model-generated description of where the chunk sits in its document before embedding it. Anthropic reported in 2024 that contextual embeddings cut the top-20 retrieval failure rate by 35%, and by 49% when combined with contextual BM25.

That is Anthropic's result on its own test sets, not a promise for your corpus. It also has a price: Anthropic estimated about $1.02 per million document tokens for the one-off context generation, using prompt caching and its stated chunk sizes.

Re-ranking: is it worth it?

Re-ranking is worth it when the right chunk is already in your candidate set but sits too low to reach the prompt. It is wasted when first-stage recall is the problem. Measure recall at k before buying anything.

What does a re-ranker actually do?

First-stage retrieval is fast and approximate: it compares one query vector against millions of chunk vectors. A cross-encoder re-ranker then scores each query and candidate pair together, which is slower and more precise. You retrieve, say, 50 candidates and keep the best 5.

The BEIR benchmark paper reported that re-ranking and late-interaction models achieved the best zero-shot averages, at high computational cost. Both halves of that sentence matter.

When does it not pay?

Skip it when top-5 recall is already high, when your latency budget is tight, or when your volume makes per-query compute material. Anthropic's contextual retrieval post reports a further gain from adding a re-ranking step on top of contextual retrieval, so the effect is real, but it is additive to chunking, not a substitute.

A practical order of operations: fix chunking, add hybrid keyword and vector search, then test a re-ranker against your own eval set.

Do long contexts bury the answer?

They can. The Lost in the Middle paper found that models tend to use information at the start and end of a long prompt better than information in the middle. Passing twenty chunks to the generator is therefore not the same as passing the best five.

This is the second reason to re-rank and trim. Keep the candidate set short, place the strongest passages first, and test whether answer quality changes when you reorder the same chunks. If it does, position is costing you accuracy and a tighter cut-off is cheaper than a bigger model.

Why do evals get cut, and why shouldn't they?

An eval set is a list of real questions paired with the passages that answer them, scored automatically on every change. Without one, every chunking or model tweak is a guess, and regressions reach users before they reach you.

What should a RAG evaluation measure?

Measure retrieval and generation separately. For retrieval, track recall at k and mean reciprocal rank against labelled passages. For generation, track faithfulness to the retrieved context and answer relevance.

Open-source tooling such as the Ragas documentation defines metrics for context precision, context recall and faithfulness. Use them as a starting vocabulary, and calibrate any LLM-as-judge score against human labels on a sample.

How big does the eval set need to be?

Start with 100 to 200 real questions drawn from support tickets, search logs or subject experts. Include the awkward ones: ambiguous phrasing, questions with no answer in the corpus, and questions where two documents disagree.

This is the line that gets cut when a schedule tightens. It is also the line that lets you change a chunker on a Friday and know on Monday whether it helped. Plan labelling time from domain experts, because that is the slow part.

What does index refresh cost in a RAG system?

Index refresh is the hidden run-rate of a RAG system. It covers detecting changes, re-chunking, re-embedding, deleting stale vectors and propagating permissions. The infrastructure bill is small. The ownership cost is not.

What has to be refreshed?

Four things drift:

• Content. Content changes, so affected chunks must be re-embedded.
• Deletions. Documents are deleted, and their vectors must go too, or the system quotes withdrawn policy.
• Access rules. Permissions change, and retrieval must respect them at query time.
• Embedding models. When you change models, the entire index must be rebuilt, because vectors from different models are not comparable.

Why is this rarely quoted?

Because it is operational work, not a feature. It needs change detection on source systems, idempotent ingestion jobs, monitoring for failed documents, and a regression run of the eval set after each material change.

This is data engineering and MLOps work more than model work. If your team lacks it, data engineering and MLOps capacity is the real hiring question, not which vector database to pick.

What is the RAG implementation cost model for 1M documents?

For 1M documents, the one-off embedding bill is tens to low hundreds of dollars at published API rates. The recurring costs are query-time generation, re-ranking compute, storage and the engineering hours behind refresh and evals.

The numbers below use stated assumptions: 1,000 tokens per document and 500-token chunks, so 1 billion tokens and 2 million chunks. For query volume, assume 100,000 queries a month, each sending about 3,000 input tokens of question plus retrieved context. Embedding prices are OpenAI's list rates on its pricing page: $0.02 per million tokens for text-embedding-3-small and $0.13 for text-embedding-3-large. Check them before you plan.

What does each line item come to?

• One-off embedding, small model: 1B tokens at $0.02 per 1M comes to about $20.
• One-off embedding, large model: 1B tokens at $0.13 per 1M comes to about $130.
• Contextual chunk generation: Anthropic's $1.02 per 1M document tokens comes to about $1,020.
• Vector storage, 1,536 dimensions: 2M chunks at 4 bytes per float comes to about 12 GB.
• Vector storage, 3,072 dimensions: 2M chunks at 4 bytes per float comes to about 25 GB.
• Query-time input tokens: 100,000 queries a month at 3,000 tokens each comes to 300M tokens a month.
• Re-ranking: 100,000 queries with 50 candidates each comes to 5M pair scorings a month.
• Monthly refresh embedding: if 5% of documents change, about $1 on the small model.

What do the numbers tell you?

The refresh embedding and storage lines are trivial. The line that scales is the query-time token volume: multiply the 300M monthly input tokens (100,000 queries times 3,000 tokens) by your chosen model's current rate, and add output tokens. Double the query volume or the retrieved context and that line doubles.

Everything that makes the system trustworthy, evals, ingestion code, monitoring, permissions, is engineering time. That is scoped per project, which is why Viithiisys quotes it after a discovery call rather than from a rate card.

When should you skip RAG entirely?

Skip RAG when the corpus is small enough to fit in the context window, when questions need aggregation rather than passage lookup, or when the data is structured. A simpler architecture will cost less and fail less.

Which cases are better served without it?

A corpus of a few hundred pages can often go straight into a long context, ideally with prompt caching, with no retrieval layer to maintain. Questions like "how many contracts renewed last quarter" are database queries, not passage lookups, so text-to-SQL or a plain report is the right tool.

Pure keyword search also still wins where users know the exact identifier they want. Hybrid search is a reasonable middle path.

How do you decide before committing?

Run the cheap test first: take your 100 eval questions, paste the relevant documents into a long-context prompt, and score the answers. If that clears your accuracy target at an acceptable per-query cost, you have saved yourself an index to own.

If it does not, you now have a baseline score and a per-query cost to beat. Any RAG design has to improve on both, and the same eval set will tell you whether it does.

What should you do next?

Settle three things before choosing a vendor: the corpus, the eval set and who owns index refresh. Those decide most of the cost, and none of them depends on which vector database you pick.

If you want a second opinion on that scoping, our broken workflow assessment is a structured review of where a process is failing and whether AI is the right fix. Bring a sample of your documents and the twenty questions you most need answered correctly. Related work is covered under AI development services.