Building a Production Django RAG Chatbot Without Breaking the Bank

Everyone wants to add AI to their products right now. Very few teams stop to ask whether they can actually support it once the proof of concept is over.

• Can we afford it?
• Can we observe it?
• Can we control costs?
• Can we prevent abuse?
• Can we improve it over time?

Those questions matter a lot more than whether you can get a chatbot working in a weekend.

At CyberFortress, we’re currently designing a retrieval-augmented generation (RAG) chatbot for a customer-facing Django application, which is the heart of our Veeam customer portal. The goal wasn’t to build the most advanced AI assistant possible; it was to build something useful, affordable, and maintainable.

Here’s how we’re approaching it.

Start With the Problem, Not the Model

The easiest mistake to make is beginning with the question:

Which LLM should we use?

Instead, we started with:

What problems are customers actually trying to solve?

In our case, most users aren’t uploading massive log files or asking complex architectural questions. They need quick answers:

• “How do I configure this?”
• “Why is this failing?”
• “Where can I find this setting?”
• “What does this error mean?”
• “How do I perform this task?”

Most questions are repetitive and grounded in documentation. That immediately pushed us toward a RAG architecture instead of relying entirely on the model’s training data. It also provided a practical path to launch quickly, validate assumptions with real users, and improve the experience before investing in more advanced capabilities.

The Architecture

At a high level, the flow looks like this:

1. User submits a question.
2. Relevant documentation is retrieved from a vector database.
3. Context is assembled into a prompt.
4. A lower-cost model generates an answer.
5. Complex requests can be escalated to a more capable model.
6. Feedback, telemetry, and user ratings create the signals that drive future improvements.

Simple in theory. The details are where production systems either succeed or become expensive science projects.

Reference Architecture

The important thing to notice is what isn’t there. No additional vector database cluster. No separate orchestration platform. No half-dozen new services to maintain. Every new component introduces operational overhead. We intentionally kept the architecture as simple as possible.

Why We Chose PostgreSQL and pgvector

One of the biggest temptations when building RAG systems is introducing an entirely new database stack.

• Pinecone
• Weaviate
• Qdrant
• Chroma

Those are all excellent products. But we already run PostgreSQL. Adding pgvector allowed us to extend the infrastructure we already understand rather than introducing another operational dependency.

The advantages were immediate:

• Existing backup processes continue to work.
• Existing monitoring continues to work.
• Existing security practices continue to work.
• Existing operational expertise still applies.
• Existing disaster recovery plans remain valid.

For many organizations, the best technology decision is the one that reduces complexity. Not the one with the most hype.

Building the Knowledge Base

A RAG system is only as useful as the information you give it. One of the first challenges we faced wasn’t choosing a model. It was deciding how to populate and maintain the knowledge base behind it.

• Documentation changes.
• Knowledge base articles evolve.
• New product releases introduce new workflows and error messages.

A one-time import quickly becomes outdated. Instead, we approached ingestion as an ongoing process rather than a migration project. Our pipeline crawls approved documentation sources, extracts meaningful content, converts it into a format suitable for retrieval, and generates embeddings that are stored alongside metadata in PostgreSQL. That metadata turned out to be just as important as the embeddings themselves.

By tracking information such as document source, product, version, article title, and last updated date, we gain more control over retrieval quality and can make smarter decisions about what context to present to the model. Just as importantly, we can refresh and grow the knowledge base over time. When documentation changes, the chatbot can change with it. The most effective chat agents aren’t built on static snapshots of knowledge. They’re designed to evolve alongside the products they support.

Memory Beyond a Single Conversation

Once we had RAG working well, we ran into another limitation that had nothing to do with the quality of the underlying model.

The chatbot knew our documentation, but it didn’t know our users.

Each conversation maintained its own context. Recent messages were replayed directly, and older exchanges were compressed into a rolling summary to keep prompts manageable. That worked well within a single thread, but every new conversation effectively reset the relationship. Users who had already explained their environment, described a recurring issue, or provided important context had to start over.

From a user’s perspective, that feels unnatural.

If someone mentioned last week that they’re running Veeam Backup & Replication on Windows Server 2019, or that they’re troubleshooting a specific recurring error, they don’t expect to repeat that information every time they open a new chat. Humans naturally carry relevant context forward. A production chatbot should do the same, provided it can do so responsibly.

Rather than introducing another database or external service, we reused the same PostgreSQL and pgvector infrastructure that already powered retrieval. Each conversation already maintains a rolling summary designed to capture the important facts while filtering out conversational noise. That summary turned out to be an ideal source for memory.

When a new question arrives, the chatbot can semantically search a user’s previous conversations and retrieve the most relevant context from earlier threads. We intentionally avoided embedding every individual message because, in practice, that tends to teach the system that users frequently say things like “Thanks, that fixed it,” while burying the details that actually matter. A single fact-focused representation of each conversation kept the index compact and improved precision.

Of course, memory introduces challenges beyond simply adding another vector search query. Recall has to respect tenant boundaries. Deleted conversations shouldn’t leave artifacts behind. Failure modes need to be graceful. If the memory system can’t retrieve anything useful, the chatbot should continue operating normally rather than producing a degraded experience. We also chose to make the capability explicitly controllable through feature flags so that it could be enabled gradually and disabled instantly if needed.

From a user’s perspective, this entire feature boils down to a simple expectation. Not having to say:

“I already told you this.”

Meeting that expectation turned out to require considerably more thought than adding another embedding table, but it also moved the experience noticeably closer to interacting with a system that understands your environment over time rather than a stateless API that starts from zero every time you click “New Conversation.”

Why the Cheapest Model Isn’t Always the Wrong Choice

There’s a tendency to assume customer-facing AI requires the largest model available. Our experience suggests otherwise.

Most questions don’t need advanced reasoning; they’re retrieval problems. If the correct documentation has already been retrieved, even smaller models can produce excellent responses. Our current approach uses a lower-cost model as the default path. More expensive models are reserved for situations that justify the additional cost. That strategy dramatically reduces operating expenses while maintaining a high-quality experience.

The goal isn’t to build the smartest chatbot possible. It’s to build one that reliably solves customer problems in a way that’s sustainable for both the customer and the business.

Cost Controls Matter More Than Demos

A prototype handling ten questions per day looks impressive. Production traffic changes everything. Without controls, costs can scale rapidly.

We designed several safeguards from the beginning:

• Per-user rate limits.
• Per-customer quotas.
• Platform-wide budget controls.
• Different usage weights depending on model cost.
• The ability to disable or reroute expensive functionality.

These controls aren’t exciting. They’re essential. Every production AI system eventually becomes a financial system. Design accordingly.

What Does This Actually Cost?

One of the first questions leadership asks is:

“How much is this going to cost us?”

Unfortunately, many AI demos skip that discussion entirely. Let’s look at a hypothetical example using publicly available pricing.

Assume:

• 1,000 users
• Each user asks 10 questions per month
• Average request:
  • 2,000 input tokens
  • 500 output tokens
• Total monthly requests: 10,000

That works out to approximately:

• 20 million input tokens per month
• 5 million output tokens per month

Scenario 1: One Model for Everything

If every request goes to the same model, the monthly costs can vary significantly.

Even at only 10,000 requests per month, the spread is substantial. Sending every request to the most capable model may produce excellent answers, but it also means that simple questions consume premium resources.

Scenario 2: Lower-Cost Model + Escalation

A more realistic approach is to reserve premium models for situations that genuinely require them.

Suppose:

• 90% of requests use GPT-4o-mini
• 10% escalate to GPT-5-mini

The monthly costs would look like this:

GPT-4o-mini (9,000 requests)

• 18 million input tokens × $0.15 per million = $2.70
• 4.5 million output tokens × $0.60 per million = $2.70

Subtotal: $5.40

GPT-5-mini (1,000 requests)

• 2 million input tokens × $2.50 per million = $5.00
• 0.5 million output tokens × $2.00 per million = $1.00

Subtotal: $6.00

Total Monthly Cost: $11.40

That’s less than the cost of lunch for an entire month of AI interactions across 1,000 users.

Scenario 3: Frontier Escalations

What if you wanted access to a frontier model for the most challenging questions?

Suppose:

• 90% of requests use GPT-4o-mini
• 10% escalate to Claude Opus 4.8

The costs would be:

GPT-4o-mini subtotal: $5.40

Claude Opus 4.8 (1,000 requests)

• 2 million input tokens × $15 per million = $30
• 0.5 million output tokens × $75 per million = $37.50

Subtotal: $67.50

Total Monthly Cost: $72.90

Even this hybrid approach remains nearly 90% less expensive than routing every request to Opus.

The Bigger Lesson

The most interesting takeaway isn’t that GPT-4o-mini is cheap; it’s that architectural decisions fundamentally change the economics of AI. A chatbot handling 10,000 requests per month can cost anywhere from roughly $6 to nearly $700 per month, depending on how requests are routed.

Cost optimization isn’t about choosing the cheapest model; it’s about matching the right capability to the right problem. Most customer questions don’t require frontier reasoning. They require finding the correct information and presenting it clearly. Reserve expensive intelligence for expensive problems.

A Note on Pricing: Model pricing changes rapidly. The specific numbers will evolve over time, but the architectural principle remains the same: default to the least expensive option that delivers acceptable results and selectively escalate when additional capability is justified.

Observability Is Not Optional

One of the most difficult questions to answer after deployment is:

Why did the chatbot give that answer?

Without instrumentation, you may never know.

We wanted visibility into:

• Which documents were retrieved?
• Which prompts were generated?
• Which model produced the response?
• How many tokens were consumed?
• Response latency
• User feedback
• Escalation frequency

This transforms troubleshooting from guesswork into investigation. It also creates opportunities to improve the system over time. If a particular document consistently leads to poor outcomes, we can identify it. If a model starts producing unexpected responses, we can investigate. If costs spike unexpectedly, we can trace the source.

You can’t improve what you can’t measure.

Retrieval Quality Beats Model Quality

Many chatbot failures aren’t actually model failures; they’re retrieval failures. If irrelevant documents are selected, even the smartest model will generate poor answers. Improving chunking strategies, metadata quality, document freshness, and search relevance often produces larger gains than switching to a more expensive model.

Before upgrading the model, evaluate whether you’re retrieving the right information in the first place. You might already have the answer. You just aren’t finding it.

Closing the Feedback Loop

One of the more interesting questions we had to answer wasn’t technical. It was deciding how an enterprise AI system should improve over time.

From a purely engineering perspective, the simplest approach would have been to collect as much data as possible, measure every interaction, and optimize aggressively using whatever signals happened to be available. Increasingly, however, organizations expect transparency and control over how AI systems operate and how their data contributes to future improvements.

We decided that participation should be intentional.

At the organizational level, administrators can choose whether to enable generative AI features at all. If they disable those capabilities, the chatbot experience simply isn’t available. Organizations that do enable AI are then presented with a separate decision: whether they want to allow anonymized, aggregated interaction data to be used to improve the platform.

Choosing not to participate in improvement activities doesn’t prevent users from benefiting from the chatbot. They still receive the same retrieval capabilities, the same responses, and the same functionality. The difference is that ratings, feedback signals, and observability data remain isolated from the broader improvement process.

We also wanted feedback to occur naturally within the experience itself. Originally, users had to leave the conversation, navigate through menus, and complete a separate form. Unsurprisingly, very few people are motivated to do that after they’ve already received an answer.

Instead, users can now rate individual responses directly within the chat interface and provide feedback or feature requests without leaving the conversation. Those ratings become substantially more valuable when paired with context. Highly rated interactions can reveal effective retrieval patterns and prompt strategies. Poorly rated responses often expose missing documentation, confusing workflows, situations where the chatbot should have escalated rather than attempted an answer, or emerging product pain points.

The ratings themselves don’t simply disappear into a dashboard that nobody checks.

Because we already relied on PostgreSQL and pgvector elsewhere in the system, we chose to reuse that infrastructure here as well. Feedback and interaction quality signals can be embedded and analyzed alongside the conversational context that produced them. Rather than asking only, “How many users liked this response?” we can ask more meaningful questions. Which topics consistently generate poor outcomes? Are certain categories of questions associated with lower confidence? Are there gaps in the knowledge base that repeatedly frustrate users? Which interactions represent examples of the system performing particularly well?

Those insights extend well beyond the chatbot itself. Documentation teams gain visibility into areas that require clarification or expansion. Support teams can identify recurring pain points before they become larger issues and adjust workflows accordingly. Product managers gain a better understanding of where customers struggle most frequently and which feature requests continue surfacing through conversations and support interactions. Engineering teams receive concrete examples of where the system falls short instead of relying on assumptions.

Perhaps more interestingly, those insights don’t stop at reporting. Improvements to documentation flow back into the same knowledge base powering retrieval, allowing future responses to benefit from what previous conversations uncovered. Questions reveal gaps, documentation closes those gaps, the RAG corpus improves, and the next customer receives a better answer. In some cases, those same patterns may drive updates to support processes or inspire entirely new features. What initially looks like a chatbot gradually becomes a feedback loop connecting support interactions, documentation quality, product decisions, and the customer experience itself.

That balance between continuous improvement and explicit consent turned out to be just as important as selecting the right model, choosing the right vector database, or tuning the retrieval pipeline. In many ways, it’s the difference between building an AI demo and building an AI product that organizations are genuinely comfortable adopting.

Security and Governance Matter

It’s easy to focus exclusively on the technology.

By the time you start asking questions about retention policies, access controls, and deletion requests, you’ve already moved beyond building a chatbot and into building a product.

Consider:

• What information should the chatbot never access?
• How long should prompts and responses be retained?
• Who can view telemetry data?
• How will customers request the deletion of stored interactions?
• What safeguards prevent abuse?

Security, compliance, and governance aren’t afterthoughts. They should be part of the architecture from day one.

Final Thoughts

Building a chatbot is easy. Building one that people trust is much harder. The difference isn’t usually a better model. It’s a thoughtful architecture.

• Use infrastructure your team already understands.
• Control costs before they become problems.
• Measure everything.
• Focus on retrieval quality.
• Learn from your users.

The companies that succeed with AI won’t necessarily be the ones using the biggest models. They’ll be the ones building systems they can actually operate long after the demo ends. A successful production chatbot isn’t defined by how impressive it looks during a demo; it’s defined by whether it’s still delivering value six months later—without surprising your users, overwhelming your support teams, or blowing up your cloud bill.

Bringing AI Into Production

If there’s one thing we’ve learned through this process, it’s that successful AI initiatives aren’t defined by flashy demos or the latest model release. They’re defined by thoughtful architecture, operational discipline, and a relentless focus on solving real customer problems.

Organizations everywhere are trying to figure out how to move beyond experimentation and deploy AI in ways that are secure, cost-effective, and genuinely useful. It’s not always easy, but it is achievable with the right approach.

If your organization is exploring how AI, automation, or intelligent support experiences fit into your products and services, I’d love to hear what you’re building and the challenges you’re encountering along the way.

And if you’re exploring how AI, automation, and intelligent support experiences fit into your products and services, or looking to strengthen your broader resilience strategy, the team at CyberFortress is always happy to start a conversation.

After all, the same principles apply whether you’re protecting critical data or deploying AI into production: keep it simple, design for the long term, and build systems your team can confidently operate.