From RAG POC to Production: Making Stronger Retrieval Practical

In the first post, I covered the initial architecture for building a production Django RAG chatbot without turning it into an expensive science project. The first version used a straightforward retrieval path: embed the user’s question, search PostgreSQL with pgvector, pass the retrieved context to the LLM, and log enough metadata to understand what happened.

That was the right starting point. It kept the system understandable, avoided external AI APIs for retrieval, and made the feature cheap enough to run.

After living with that design, the next set of problems became clearer. The chatbot needed to handle more than English semantic similarity. It needed better exact-term matching for technical strings, better multilingual support, better refresh behavior, better latency visibility, and a more flexible retrieval flow for questions that needed deeper searching.

This post covers the next iteration of that retrieval system: corpus cleanup, multilingual embeddings, hybrid retrieval, reciprocal-rank fusion, CPU-native reranking, layered retrieval, tool-aware streaming, status events, incremental refreshes, and better observability.

The important framing is that the new retrieval system is not simply a faster version of the original one. The original path was faster because it did much less. The updated retrieval engine is more capable, but some of that capability is more expensive. The final design keeps the cheap parts in the default request path and moves the expensive parts behind tools so the system pays for deeper retrieval only when it is useful. Additional optimization work reduces the impact of that extra processing through faster CPU-native models, parallelization, streaming, and status updates, improving perceived latency even when the backend is doing more work.

The original path was fast because it was simple

The first version used a small English embedding model and dense vector search. For a proof of concept, that was a good tradeoff. Dense retrieval is useful because it can match text by meaning rather than exact wording, and it gives you a working RAG system quickly.

It was also very fast. Query embedding plus dense vector search was typically in the low hundreds of milliseconds. There was no separate lexical retrieval path, no reciprocal-rank fusion, and no reranker.

The limitations showed up in real support-style questions.

Dense semantic search can blur details that need to match exactly: error codes, product names, version numbers, command names, configuration keys, repository names, object types, and other precise strings. A semantically similar chunk is not always the correct chunk if the exact technical term is wrong or missing.

The original path was also English-first. That was acceptable for the initial version, but it limited the system once the goal became broader production use.

The next iteration needed to keep the operational simplicity of the original stack while making retrieval more capable.

Cleaning the corpus before changing the model

Before changing retrieval models, we cleaned up what was being indexed.

One issue was base64-encoded content, such as embedded images, appearing in collected documents. That content can dramatically inflate document size without improving text retrieval. It increases storage, slows processing, and can add noise to chunking and embedding.

The updated collection path strips base64-encoded payloads before indexing. The goal is to avoid embedding data that cannot help answer a text question.

That is not a model improvement, but it matters. RAG quality is not only a model-selection problem. If the corpus contains large blocks of irrelevant encoded data, both the embedding model and the database waste work on content that should never have entered the searchable text path.

It also matters for indexing performance. Less junk content means fewer useless tokens to process, fewer bad chunks to store, and less avoidable work during refreshes.

Moving to multilingual embeddings

The first major retrieval change was replacing the English-only embedding model with a multilingual one.

The updated dense embedder is gte-multilingual-base exported as an INT8 ONNX model and served through onnxruntime. It produces a ready-made 768-dimensional sentence_embedding, so the application does not need to implement manual pooling. The ONNX graph takes input_ids and attention_mask inputs, and the tokenizer path can use the XLM-RoBERTa tokenizer without requiring trust_remote_code.

That combination matters operationally.

A multilingual model that is too slow to crawl the corpus is not useful. In testing, the heavier multilingual path was much too expensive for CPU-based indexing. The INT8 ONNX gte-multilingual-base path is a much better fit for this environment: it keeps multilingual semantic retrieval local, CPU-native, license-clean, and fast enough for both query-time embedding and background indexing.

The practical result is that the system can support multilingual dense retrieval without sending customer data to an external embedding API and without requiring GPU infrastructure.

Hybrid retrieval without relying on one model to do everything

The retrieval path still uses hybrid search, but the final design separates the semantic and lexical responsibilities.

The dense side uses multilingual embeddings for semantic similarity and cross-language matching.

The lexical side handles exact-term matching for technical strings: error codes, product names, version numbers, command names, configuration keys, and other terms that need literal matching.

The result sets are then combined with reciprocal-rank fusion.

This design is less model-dependent than requiring one embedder to produce every retrieval signal. Dense embeddings do what dense embeddings are good at: meaning and multilingual similarity. Lexical search does what lexical search is good at: exact terms and precise strings.

That split is a good fit for technical support content. A user may ask a question in natural language, but the best matching document may depend on an exact error code or product term. The retrieval system needs both signals.

Why reciprocal-rank fusion

Dense and lexical search scores are not directly comparable. Dense similarity and lexical relevance represent different signals, so naively merging raw scores can produce unstable behavior.

Reciprocal-rank fusion avoids that by using rank position instead of raw score normalization. A document gets credit for ranking well in either result set, and extra credit if it ranks well in both.

That makes it a good fit for this kind of hybrid retrieval.

Dense search can find conceptually relevant content even when the wording differs. Lexical search can pull up content containing exact error codes, versions, product names, or configuration values. RRF gives the chatbot a better candidate set before the LLM ever sees context.

Adding a reranker without making every turn pay for it

The updated retrieval system also adds a cross-encoder reranker. The original shipped path did not have one.

A cross-encoder evaluates the query and candidate text together, instead of embedding them separately and comparing vectors. This usually improves final candidate ordering, but it is more expensive than vector search.

That cost matters. The first attempted reranker path was not viable for an interactive CPU-only system. It added roughly 17–25 seconds per turn in testing, which was not acceptable.

The working implementation uses gte-multilingual-reranker-base as an INT8 ONNX model through onnxruntime. That model is smaller than the heavier reranker we tested, is Apache-2.0 licensed, runs locally on CPU, and avoids the torch runtime issue we hit in the PyTorch path.

With the ONNX path, reranking a small pool takes around a couple of seconds in dev, depending on candidate pool size and background load. That is practical for deeper search, but it is still too expensive to force into every chat turn when many questions do not need it.

The final design keeps reranking available but moves it out of the automatic upfront path.

Layered retrieval: cheap upfront, deeper on demand

The most important design change was splitting retrieval by purpose.

Every turn still gets a cheap upfront retrieval pass. That pass is hybrid retrieval: dense semantic search plus lexical search, fused with RRF, without paying the reranker cost. On dev, this brought the automatic retrieval path down from roughly reranked low-single-digit seconds to roughly tens of milliseconds as cheap hybrid-only retrieval.

That upfront pass gives the model baseline grounding for normal questions without making every turn pay for deep search.

The deeper reranked path moved behind a search_knowledge_base tool. When the model needs stronger documentation evidence, it can call the tool and run the more expensive reranked search on demand.

That gives the system two different retrieval paths:

This is better than either extreme.

If reranking runs upfront on every turn, simple questions pay for work they do not need. If retrieval is tool-only, every answer depends on the model deciding to call the search tool at the right time. Keeping cheap upfront retrieval plus a deeper search tool gives the system a useful baseline and a stronger fallback when the question requires it.

Better diagnostic searches

The layered design is especially useful for diagnostic questions.

A user might ask a vague question like “what errored, why, and how do I fix it?” The original question may not contain the actual error code or job detail needed for a good knowledge-base search.

With the layered tool design, the model can first call an account or backup tool, discover the specific job error, status, or message, and then call search_knowledge_base with that more precise information.

That produces better documentation matches than searching the KB with only the user’s original vague question.

The retrieval flow becomes:

1. Use cheap upfront context for baseline grounding.
2. Call account or backup tools when customer-specific data is needed.
3. Use the actual error or object details returned by those tools to run a reranked KB search.
4. Stream the final answer with better evidence.

This is a better use of reranking because the expensive search happens when the model has a more specific query.

Reranking as a configurable quality dial

The reranked search path is configurable rather than hard-coded.

Some workloads benefit from reranking. Others may be fine with hybrid retrieval alone. Some deployments may prefer lower latency; others may prefer stronger final ordering.

The updated configuration supports that tradeoff. Reranking can be disabled for a fast hybrid-only mode. The candidate pool can be tuned, with a smaller default pool for normal use and a larger pool available when quality matters more than latency. A timeout fails open to the hybrid ranking so a slow reranker cannot stall a response.

In practice, the useful modes look like this:

This is better than forcing every deployment to use the same retrieval depth. It gives operators a way to tune behavior without redeploying.

Storage impact

The updated embedding model uses 768-dimensional vectors instead of the original 384-dimensional vectors. That increases dense vector size, though not as much as the heavier 1024-dimensional path we tested.

The storage design still matters.

Dense vectors can be stored with halfvec / fp16 instead of naive fp32. A 768-dimensional fp32 vector would be roughly 3 KB. Stored as fp16, the dense vector is roughly 1.5 KB. That keeps the multilingual vector size close to the old 384-dimensional fp32 footprint while providing a stronger multilingual embedding model.

The lexical side also has storage cost, but it is separate from the dense vector. The point is not that the new retrieval system stores less than the original one in every dimension. It does more. The point is that the extra capability is stored efficiently enough to be practical.

Corpus cleanup helps here, too. Removing base64 payloads and other non-useful content reduces the amount of text that needs to be chunked, embedded, stored, and searched.

Indexing throughput matters

Query latency gets most of the attention, but indexing throughput matters just as much for a production RAG system.

A model can look acceptable for one user query and still be a bad fit for crawling or refreshing a knowledge base. Background indexing has to process many pages and chunks. If each document takes too long on the CPU, refreshes become operationally painful.

That is why the embedder change matters. The heavier multilingual path made indexing too slow for this environment. The INT8 ONNX multilingual embedder is much more practical for CPU-based crawling and refreshes.

This is also why the model runtime details are important. A clean ONNX graph with built-in pooling and simple inputs is easier to operate than a path that depends on custom model code, runtime prompts, or heavyweight PyTorch execution for every embedding call.

The goal is not only good retrieval quality. The goal is good retrieval quality with an indexing pipeline that can keep up.

Incremental refreshes

The original refresh model was closer to full reprocessing. That is acceptable early on, but it becomes wasteful as the corpus grows.

The updated pipeline uses content hashes so unchanged documents can be skipped. If a document has not changed, there is no reason to re-extract, re-chunk, and re-embed it during a normal refresh.

Crawls can also resume from a cursor instead of starting over each time. A per-source lock prevents manual refreshes and scheduled refreshes from processing the same source concurrently.

Routine refreshes now process only changed content. That reduces CPU time, embedding work, and database churn.

A model change is the exception. If the embedding model changes, unchanged content still needs to be re-embedded because the vector space has changed. In that case, a one-time full re-embed is expected.

Query embedding cache

The updated system caches embeddings for identical question text for a limited period.

This is useful for repeated support questions, especially around common error codes or common restore workflows. If the same question is asked many times, only the first request needs to pay the query embedding cost. Later requests can reuse the cached embedding.

This does not change retrieval behavior, but it removes avoidable repeated work. Common repeated questions only need to be embedded once during the cache window, reducing CPU time in the embedding service and shaving latency from repeat requests.

Tool-aware streaming

After reranking moved out of the automatic upfront path, retrieval was no longer the main reason some real turns felt slow. The next bottleneck was how tool turns streamed.

Almost every real turn exposes account or backup tools. Previously, when a turn involved tools, the response path could fall back to non-streaming behavior. The model would complete the tool call flow, generate the final answer, and only then return text to the user.

That made the user wait for the whole tool-enabled completion before seeing anything.

The fix was making the streaming tool-aware end-to-end. The model now streams text as it is generated. If it calls a tool, the loop executes the tool and then continues streaming the follow-up model call.

That does not make the model computation disappear, but it changes the user experience. Instead of waiting for the entire tool-enabled completion before seeing anything, the user gets the answer text as soon as it is generated.

Moderation was also adjusted to run once and overlap with retrieval. Previously, tool turns could effectively pay moderation cost twice. Running it once, concurrently with retrieval, removes unnecessary serial latency.

Tool-status events

Streaming helps once the model starts producing answer text, but tool turns can still have a pre-answer gap. The model may need to decide to call a tool, dispatch it, wait for the result, and then produce the final response.

If the UI shows nothing specific during that period, the user reads it as a stall.

The updated chat flow emits tool-status events. When a tool is dispatched, the UI can immediately show a muted status such as “Searching your backups…” or “Working on that restore…”. That status is replaced by the first answer token once the model starts streaming the response.

This is a small UI change, but it matters for perceived latency. A few seconds of visible progress feels different from a few seconds of a spinner with no explanation.

The backend work still has to happen. The status event simply makes that work visible.

Latency observability

The updated path has more moving parts, so a single total response time is not enough to debug performance.

A slow answer could be caused by embedding, dense search, lexical search, RRF, reranking, tool execution, moderation, model generation, streaming behavior, or some external dependency.

The updated logging records per-stage timings. Retrieval is split into embedding, search, and reranking. Tool execution and model timings are tracked separately. Slow turns can raise Sentry alerts, so latency issues show up as production signals instead of only as user complaints.

This matters because the retrieval system is now tunable. If reranking dominates latency, the pool size or timeout can be adjusted. If search dominates, indexes and query plans can be examined. If the model dominates, streaming behavior and tool flow become the place to look.

Without stage-level timings, those decisions would be guesses.

Model runtime isolation

Another operational issue is where the models run.

Loading embedding and reranking models into every web worker is inefficient. It increases memory usage and creates more cold-load points after deploys or restarts.

The updated architecture runs embedding and reranking through a per-host model service over a Unix socket. Web workers call the local service instead of loading their own model copies.

This keeps web worker memory lower and centralizes CPU-bound model execution. Models are pre-warmed at startup to avoid a cold-load stall on the first chat request. CPU thread usage is tunable per host, so model inference does not saturate the machine.

The reranker runs through ONNX, and the embedder can also use an ONNX path. That separation is useful because the critical retrieval models stay on a stable CPU-native runtime path instead of depending on heavier model execution inside web workers.

Before and after

The retrieval path changed from a simple dense-only POC design to a layered production design.

The updated path is not a cheaper or lighter version of the original path in every respect. It is a more capable retrieval system with a higher ceiling and more tuning controls.

The final design keeps normal turns fast by using cheap upfront hybrid retrieval, then reserves reranked search for questions that need deeper evidence. The embedder change keeps multilingual crawling practical on the CPU. Tool-aware streaming and status events make longer tool turns feel responsive while the backend does the extra work.

Lessons learned

The main lesson from this iteration is that production RAG improvements are usually systems work, not just model selection.

Changing the embedding model helped, but only as part of a larger set of changes:

• Remove junk from the corpus before indexing;
• Use multilingual dense retrieval for semantic similarity;
• Use lexical retrieval for exact technical terms;
• Fuse rankings instead of relying on one score type;
• Add reranking where the quality gain is worth the latency;
• Move expensive reranking out of the automatic path;
• Keep cheap upfront retrieval for baseline grounding;
• Expose deeper reranked search as a tool;
• Make reranking configurable and fail-open;
• Choose an embedding model that is fast enough for CPU indexing;
• Store larger vectors efficiently;
• Refresh incrementally;
• Cache repeated query embeddings;
• Stream tool-enabled responses;
• Show tool-status events during pre-answer work;
• Log timings by stage;
• Isolate model runtime from web workers.

The first version proved that a Django-based RAG chatbot could answer questions from internal knowledge without breaking the cost model. This iteration made the retrieval system more suitable for production use: multilingual, more precise, more configurable, more observable, and still practical to run on CPU-only infrastructure.

That is the difference between a RAG proof of concept and a retrieval system you can operate.