
A chatbot that is confidently wrong costs more than no chatbot at all. The answer sounds finished, the customer acts on it, and somebody on your team spends an afternoon undoing it.
That is the problem retrieval solves. Rather than trusting whatever a model absorbed during training, a retrieval-augmented chatbot looks up your own documents first and answers from what it found. The jargon is RAG. The practical meaning is that every answer can be traced back to a document somebody in your business owns.
What Retrieval-Augmented Generation Actually Does
Four steps, and only the last two involve the model at all.
- The question arrives. Somebody asks what the return policy is on electronics, or whether a drawing meets the fire code.
- Retrieval runs first. The question is turned into an embedding and matched against an index of your material: policies, manuals, past jobs, standards. The closest passages come back.
- The prompt is assembled. Those passages go to the model with the question and an instruction to answer from them, and to say so when they do not contain the answer.
- The answer carries its sources. The reply comes back with the passages it used attached, so the reader can check it instead of trusting it.

The difference from a plain chatbot is that third step. A model on its own can only work from training data, which is months old and has never seen your documents. Retrieval closes that gap at the moment the question is asked rather than at training time.
What Changes Against a Plain Chatbot
Four things, in the order businesses usually notice them.
- Answers can be checked. A grounded answer points at a passage. An ungrounded one asks for trust. In regulated work that difference decides whether the system can be used at all.
- Updates stop being projects. Change a price, a policy or a product and you update the documents. Nothing is retrained. The next question picks up the change.
- Your data stays yours. The model is never trained on your material. It receives a few passages per question and keeps nothing, which is a much easier conversation with a compliance team.
- It is cheaper than fine-tuning. You pay for an index and for embeddings rather than for GPU training runs, and fine-tuning teaches a model how to behave rather than what is true this week.
One claim to be careful with. The line doing the rounds is that RAG cuts hallucinations by half. We went looking for the study behind it and could not find one: the academic work shows retrieval helps and that results vary with the retriever, and the specific percentages trace back to vendor benchmarks of their own products. What you can measure in your own system is simpler and more useful. How often the answer carried a source, and how often that source actually supported it.
Where It Is Working
Five places RAG has earned its keep, with ours attached where we have built them.
- Support desks. The questions repeat, the answers live in a knowledge base, and the cost of being wrong is a refund rather than a lawsuit. This is the easiest place to start.
- Internal knowledge. Engineering wikis, HR policy, old projects. Butler AI inside our field-service platform searches previous quotes and jobs with pgvector before it drafts a new quote, so a price reflects what the business actually charged last time.
- Regulated checking. Our building-code compliance platform indexes FBC, IBC and NFPA standards as embeddings in ChromaDB and retrieves the relevant passages before every check it runs. A finding reads as a citation, not an opinion.
- Healthcare content. Medd AI drafts with Gemini under instructions to carry in-text citations in AMA format, then GPT-4o audits the draft for factual accuracy, citation format and readability. Anything that fails comes back with an issue log instead of quietly shipping.
- Search inside a product. Our travel-operations platform runs pgvector semantic search over its own records, which is the same machinery pointed at a product rather than a policy library.
How One Is Actually Built
Six parts. Most of the work is in the first three, which is the opposite of what people expect.
- Ingestion and chunking. Documents are split into passages. Split by structure, section and paragraph, rather than by a fixed character count, or you will retrieve half a sentence with the important half missing.
- Embedding and storage. Each passage becomes a vector in an index. We use pgvector when the data already lives in PostgreSQL, which keeps the scoping and the backups in one place, and ChromaDB on the compliance platform, where the standards are a separate corpus from customer data.
- Retrieval. The question is embedded and the closest passages come back. Similar is not the same as relevant, so the field-service build adds Cohere reranking on top of the vector search to push the genuinely useful passages above the merely similar ones.
- Prompt construction. The passages and the question go to the model together with the rule that matters most: answer from these, and say when they are not enough.
- The answer and its evidence. Worth being specific about what a good one carries. Every finding on the compliance platform arrives with the code citation, the retrieved passage excerpt, the extracted value from the drawing, a severity and the source page. A reviewer can confirm it in seconds.
- Keeping it current. An index that drifts is worse than no index. In the field-service platform, Celery recomputes embeddings asynchronously every time a quote or job is saved, so retrieval reflects this week rather than launch week.

Four Ways RAG Goes Wrong
- Chunking that ignores the document. Fixed-size splitting cuts tables in half and separates a rule from its exception. The retrieval looks fine and the answers are subtly wrong.
- Retrieval with no second opinion. Vector similarity alone returns passages that look related. A reranking pass is the cheapest quality improvement available in most builds.
- No way to say I do not know. If the documents do not answer the question, the system has to say so. An agent that fills the gap with a plausible paragraph has recreated the problem you bought it to solve.
- A stale index. Retrieval is only current if ingestion keeps running. Make the refresh automatic and part of the save, not a job somebody remembers to run.
What We Have Learned Shipping These
Four patterns show up in the builds that work.
Start with one body of knowledge. One product's documentation, one department's policies, one standard. A system that answers narrowly and correctly earns the right to be widened; one that answers everything badly does not.
Fix the documents before blaming the model. Out of date, contradictory or badly written source material produces out of date, contradictory answers, and no amount of prompt work will paper over it.
Measure retrieval and generation separately. When an answer is wrong, it is either because the wrong passages came back or because the model misused the right ones, and those are different repairs. Until you can tell them apart you are guessing.
Keep a person in the loop where the answer carries risk. On the compliance platform a reviewer can edit any extracted field and re-run the checks, corrections are stored separately from the raw model output, and every edit and approval is logged with who did it. That is what makes an AI result usable as evidence. More on that in our guide to putting an agent into a business.
The Business Case, Briefly
- Repeat questions get answered without a person, at any hour.
- New staff ask the system instead of interrupting a colleague.
- Answers stay current because the documents are the source, not the model.
- One index serves everybody at once, which no support team can do.
- No training runs to pay for, so the cost sits in storage and embeddings rather than GPUs.
For context on the market rather than the technology: Fortune Business Insights puts conversational AI at $17.97 billion in 2026, reaching $82.46 billion by 2034, a 21% compound rate. Read it as one firm's model, because other houses measure the same category differently and land a long way apart.
What Comes Up in Scoping
RAG or fine-tuning?
Fine-tuning changes how a model behaves. Retrieval changes what it knows right now. If the problem is that the answers are wrong about your business, retrieval is the fix, and it is cheaper to run and simpler to keep current. Tuning is for tone and format, not for facts.
What drives the cost?
The size and state of the knowledge base first, because messy source material is the expensive part. Then the number of integrations, whether answers must be cited, and how much evaluation the use case needs before anyone trusts it. Storage and embeddings are rarely the big line.
Does it work in more than one language?
Yes. Embedding models cover many languages, so the index does not need a copy per language, and the answer can come back in the language of the question. Our field-service platform runs English, German, French and Italian in one build, which is the same principle at the interface level.
What about confidential material?
This is where RAG is stronger than the alternative, because the model is never trained on your data. Keep the vectors beside the rows they belong to, scope retrieval by tenant before similarity rather than after, and the sensitive material never leaves your database. We wrote that up in the piece on building an AI SaaS product.
Which businesses get the most from it?
Anywhere a wrong answer is expensive and the right answer is already written down somewhere: healthcare, finance, legal, construction compliance, and any software product with real documentation. If your answers change weekly, retrieval is the only approach that keeps up.
Start With One Shelf, Not the Library
Pick the body of documents your team answers questions from most often. Clean it up. Index it. Make the system cite what it used and admit when the shelf does not hold the answer. That is a working RAG chatbot, and it is a smaller project than the category makes it sound.
If you want a second opinion on which shelf to start with, book a free 30-minute call, and our AI automation and AI agents page lists what a build includes. We have been shipping production AI since 2020, and our average job feedback on Upwork is 4.99.
One for the comments: what is the first set of documents you would point a chatbot at, and who owns keeping it current?

Comments
No comments yet. Be the first to share your thoughts.
Leave a comment