RAG Chatbot: Architecture, Cost and Build Guide

What a RAG chatbot is, in plain terms

A large language model is very good at language and reasoning but knows nothing about your products, prices, policies or customers. Ask it anyway and it will often produce a confident, plausible and wrong answer.

Retrieval-augmented generation fixes that with an open-book exam. Before the model answers, the system looks up the few passages in your content most likely to contain the answer and puts them in front of the model with an instruction: answer from these, quote them, and say so if they don't cover the question.

Everything that makes a RAG chatbot good or bad follows from that idea: how well the content is prepared, how well the right passages are found, how strictly the model sticks to them, and how carefully the whole thing is tested.

RAG vs a plain LLM vs fine-tuning

These are often confused. They solve different problems, and for a chatbot that must answer from company content, RAG is almost always the core.

RAG chatbot architecture

A RAG chatbot has two paths. Ingestion runs whenever content changes and prepares it for search. The question path runs for every message and should finish in a second or two.

Chunking: the decision that matters most

If the passage that answers a question is split in half, or buried in a chunk about something else, no model can recover. Most accuracy problems we fix in existing RAG systems start here, not in the model.

• Start with structure-aware chunks of a few hundred tokens and a small overlap, then tune using your evaluation set.
• Keep each chunk's heading path (for example “Billing › Refunds › Annual plans”) in its text so it makes sense on its own.
• Convert tables to a consistent text or Markdown form, and never split a table across chunks.
• Store the source URL and section anchor so every citation links to the exact place.

Retrieval: hybrid search and re-ranking

Vector search finds passages with similar meaning even when the words differ. Keyword search finds exact terms such as product codes, error messages and names, which vector search often misses. Production systems use both.

A minimal RAG pipeline with pgvector and OpenAI

This is the core of a RAG chatbot in a few dozen lines: a table for chunks with embeddings, a search query, and a guarded prompt. A production system adds keyword search, permissions, re-ranking, caching and evaluation on top.

-- schema.sql (PostgreSQL with the pgvector extension)
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id          bigserial PRIMARY KEY,
  source_url  text NOT NULL,
  heading     text,
  content     text NOT NULL,
  allowed     text[] NOT NULL DEFAULT '{public}',  -- groups that may see it
  embedding   vector(1536) NOT NULL                -- text-embedding-3-small
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

# answer.py — pip install openai "psycopg[binary]" pgvector
import os
import psycopg
from pgvector.psycopg import register_vector
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY
conn = psycopg.connect(os.environ["DATABASE_URL"])
register_vector(conn)

SYSTEM = (
    "You answer questions for ACME support using ONLY the numbered sources. "
    "Cite sources like [1]. If the sources do not contain the answer, say you "
    "don't know and offer to connect the user with a person."
)

def embed(text: str) -> list[float]:
    return client.embeddings.create(
        model="text-embedding-3-small", input=text
    ).data[0].embedding

def retrieve(question: str, groups: list[str], k: int = 5):
    q = embed(question)
    return conn.execute(
        """SELECT source_url, heading, content
             FROM chunks
            WHERE allowed && %s          -- permission filter
            ORDER BY embedding <=> %s::vector
            LIMIT %s""",
        (groups, q, k),
    ).fetchall()

def answer(question: str, groups: list[str]) -> str:
    rows = retrieve(question, groups)
    if not rows:
        return "I couldn't find that in our documentation. Shall I connect you with the team?"
    sources = "\n\n".join(
        f"[{i}] {h or ''} ({url})\n{text}" for i, (url, h, text) in enumerate(rows, 1)
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Sources:\n{sources}\n\nQuestion: {question}"},
        ],
    )
    return resp.choices[0].message.content

if __name__ == "__main__":
    print(answer("How do I get a refund on an annual plan?", ["public", "customers"]))

The same structure works with Claude, Gemini or a self-hosted open-weight model; only the client calls change. Keeping the permission filter inside the SQL query, not in the prompt, is what makes the access control reliable.

Choosing a vector database

For most business chatbots the index holds thousands to a few million chunks, which every option below handles. Pick on hosting, hybrid search and what your team already runs.

Accuracy and safety guardrails

A chatbot that is confidently wrong does more damage than no chatbot. These controls belong in every production build.

How to evaluate a RAG chatbot

Trying a few questions in a demo tells you very little. Build a test set of 100–300 real questions from tickets, chats and search logs, with the expected answer and source for each, and score every change against it.

Failures usually point to one of three fixes: missing or unclear content (update the source), poor retrieval (chunking, metadata, hybrid search, re-ranking) or a loose prompt (stricter grounding instructions). Changing the model is rarely the first fix.

Keeping the knowledge base current

A RAG chatbot is only as current as its index. The most common complaint a few months after launch is not that the chatbot is wrong in general, but that it quotes last quarter's price or a retired policy. Plan updates from the start rather than re-importing everything by hand.

• Sync each source on a schedule or on change (webhooks from the help centre, CMS or file store), and re-embed only the documents that changed.
• Delete chunks when their source is removed or unpublished, so retired content cannot be retrieved.
• Store a version or updated date on every chunk and prefer the newest when two passages disagree.
• Give one person or team ownership of the content the chatbot uses, with a simple way to flag wrong answers back to them.
• Re-run the evaluation set after every large content change, not only after code changes.

Common RAG mistakes we fix in existing chatbots

Where RAG chatbots are used

Any place where people repeatedly ask questions whose answers already exist somewhere in your content.

How much does a RAG chatbot cost, and how long does it take?

Build cost depends mostly on the number and messiness of content sources, the channels and integrations, and how rigorous the evaluation needs to be. Running costs (model tokens, embeddings, hosting) scale with usage.

Ranges match our AI chatbot development page. You get a fixed quote after a discovery call. For a real example of a larger build, read our case study of a RAG chatbot for a healthcare provider across four channels and three languages.

Frequently asked questions