Almost every AI chatbot looks good in a demo. The demo uses ten questions the team already knows, clean documents, one user, and no deadline. Production has thousands of users asking things nobody planned for, data that changes every hour, people trying to break it, and a finance team watching the bill.
We are building a customer-facing chatbot for a large retailer right now, and most of the work is not the model. It is everything this article covers. Here is what goes wrong between demo and production, and what actually fixes it.
Demo vs production
The demo tests ten questions. Production asks everything else.
The demo
- 10 questions the team wrote
- Clean, current documents
- One user, no deadline
Production
- Thousands of users, vague and misspelled questions
- Prices, stock and policies that change every day
- People trying to break it, and a bill someone watches
The short answer
AI chatbots break in production because the demo tested the model, not the system. The usual causes are poor retrieval, stale or conflicting data, no evaluation set, no guardrails against misuse, tools with too much permission, and no budgets for latency or cost. The fix is to treat the chatbot like any production system: measure it on real questions before every change, ground every answer in a source or say "I don't know", limit what it can do, and watch it after launch.
Where chatbots break
The demo tests the model. Production tests the twelve things around it.
The model
What the demo tested
The system around it, which production tests
1. It retrieves the wrong thing
Most business chatbots answer from your documents (RAG: retrieval-augmented generation). The model is only as good as the passages it is given. In the demo, the right passage is easy to find. In production, questions are vague, misspelled, or use words your documents don't.
Fix: measure retrieval on its own, before the model sees anything: for a set of real questions, is the right passage in the top results? Improve chunking (by section, not by fixed length), add keyword search alongside vector search (hybrid search), re-rank results, and add metadata filters (country, product line, date) so the search doesn't mix up similar documents.
2. The data is stale, duplicated or contradictory
The returns policy was updated last month, but the old PDF is still indexed. Two pages give two prices. In retail this is constant: prices, stock and promotions change daily.
Fix: never answer live facts from embeddings. Price, stock, order status and delivery dates come from live APIs through tool calls; documents are only for stable knowledge (policies, guides, product descriptions). Give every document an owner and a version, re-index on change, and remove superseded versions rather than keeping both.
The fix for stale answers
Live facts come from live systems. Documents are for what doesn't change daily.
Changes by the hour
Live API, through a tool call
Read at the moment of the question, so it is never out of date.
Changes by the quarter
Indexed documents
Each one has an owner and a version, and is re-indexed when it changes.
Today's price, read from last month's PDFThis is how a demo-grade bot gets it wrong in front of a customer.
3. Nobody can say whether it got better or worse
The team changes a prompt to fix one complaint, and three other answers quietly break. Without an evaluation set, every change is a guess.
Fix: build an evaluation set early: a few hundred real questions with expected answers or expected behaviour (including "should refuse" and "should hand off to a human"). Run it on every change to prompts, models, retrieval or data, and block releases that score worse. Track separate scores for retrieval, correctness, groundedness (is every claim supported by a source?) and tone.
4. The "AI judge" is trusted blindly
Using a model to grade answers (LLM-as-judge) is necessary at scale, but judges have their own biases: they prefer long answers, miss subtle factual errors, and drift when the judge model is updated.
Fix: calibrate the judge against human labels on a sample, score narrow questions ("does the answer contain a fact not in the sources?") rather than "is this good?", keep humans reviewing a slice of production conversations every week, and pin the judge's model version.
5. It makes things up instead of saying "I don't know"
The most damaging failure in front of customers is a confident wrong answer: a policy that doesn't exist, a discount nobody offered.
Fix: require every factual answer to cite the passage it came from, and return an explicit "I couldn't find that" when retrieval finds nothing good enough. Test the "no answer" path as seriously as the happy path. For anything with money or commitments attached, answer from a tool call or hand off.
Same question, two bots
A confident wrong answer costs more than an honest “I don't know”.
Demo-grade
Can I return these shoes? I bought them 45 days ago.
Yes! You can return any item within 60 days for a full refund.
No source. That policy doesn't exist.
Production-grade
Can I return these shoes? I bought them 45 days ago.
Returns are accepted within 30 days of delivery, so this order is outside the window. If the shoes are faulty, I can connect you with someone who can help.
Source: Returns policy, current version
6. People try to break it
Users will ask it to ignore its instructions, reveal its prompt, write something offensive, or give a discount. Documents and web pages it reads can contain hidden instructions too (prompt injection).
Fix: treat all user input and retrieved text as untrusted. Separate instructions from data, filter inputs and outputs, keep a red-team test set of attacks in your evals, and make sure that even a fully compromised prompt can't do harm, which is the next point.
7. Its tools have too much power
An agent that can look up orders can often, with the wrong design, look up anyone's orders, or issue refunds.
Fix: least privilege. The agent acts with the signed-in customer's permissions, not an admin key. Read-only by default; actions that change things (cancel, refund, change address) need confirmation, limits, or a human. Log every tool call.
Least privilege
The more an action can cost, the more it takes to approve it.
- Automatic
Look things up · My order, stock in my size
Runs with the signed-in customer's permissions
- Confirm
Change things · Cancel an order, change an address
Customer confirms first, within set limits
- Human
Money and exceptions · Refunds, discounts, anything off-policy
A person approves
- Blocked
Anyone else's data · Another customer's order
Impossible by design: there is no admin key to misuse
8. It's too slow
A demo answer in 8 seconds is fine. In a live chat, customers leave. Multi-step agents (search, then call a tool, then call another) add up fast.
Fix: set a latency budget per step (for example, a p95 under two seconds) and measure it. Stream responses, run independent steps in parallel, cache frequent answers and retrieval results, and use smaller, faster models for simple steps such as classifying the question.
Latency budget
Same answer: eight seconds, or words on screen in under two.
Demo build
8.0s
One large model for every step, one after another
Budget: 2 seconds.
Production build
1.4s
Small model to classify, search and tools in parallel, answer streamed
9. The bill surprises everyone
Costs scale with traffic, conversation length, retrieved context and retries. One loop bug can burn a month's budget in a night.
Fix: estimate cost per conversation before launch, cap tokens per request and spend per day, route simple questions to cheaper models, trim retrieved context, cache, and alert on unusual spend.
10. It doesn't know when to hand over
Some conversations should never be handled by AI: complaints, legal threats, distressed customers, high-value orders.
Fix: define handoff rules up front, detect them (topic, sentiment, repeated failure, explicit request for a human), pass the full conversation to the agent who picks it up, and measure how often handoff happens and why.
11. Nobody can see what happened
A customer says the bot told them something wrong. Without logs, nobody can check.
Fix: trace every conversation end to end: the question, what was retrieved, which tools were called, the model's answer and its cost, with personal data masked. Review failures weekly and turn them into new evaluation cases. That loop is what makes a chatbot better over time.
The improvement loop
Every failure in production becomes a test it can't fail again.
01
Conversation
A real customer, a real question
02
Trace
Question, sources, tools, answer, cost
03
Weekly review
Find the answers that went wrong
04
New eval case
Each failure becomes a test
05
Release gate
Every change must pass them all
Back into production, with one less way to fail.
12. Model updates change behaviour
Model providers update and retire models. A new version can be better on average and worse on your questions.
Fix: pin model versions, rerun the full evaluation set before switching, roll out gradually (a small share of traffic first), and keep the ability to roll back.
What this looks like in retail
A retail assistant is a good example because it mixes every one of these problems. The questions customers ask fall into a few kinds, and each needs a different design:
| Customer asks | Where the answer must come from |
|---|---|
| "Do you have this in size 9?" | Live stock API, never documents |
| "Where is my order?" | Order system, with the customer's own permissions |
| "Can I return this after 30 days?" | Current returns policy, cited |
| "Which of these two is better for running?" | Product data plus careful, grounded comparison |
| "I want to complain" | Handoff to a person |
On our retail build, the hardest decisions have not been about the model. They have been about which source answers which kind of question, how to test answers before any customer sees them, and exactly when the assistant should step aside for a person.
A checklist before you launch
- Evaluation set of real questions, run on every change, with a release threshold
- Retrieval measured separately from answer quality
- Live facts (price, stock, orders) come from APIs, not documents
- Every answer cites a source or says "I don't know"
- Prompt-injection and misuse tests in the eval set
- Tools run with the user's permissions; risky actions need confirmation
- Latency budget per step and cost cap per day
- Clear handoff to a human
- Full tracing with personal data masked
- Model versions pinned, gradual rollout, rollback ready
When to get help
If your chatbot works in the demo and not in real use, the fix is rarely a better prompt. It is usually retrieval, evaluation and the system around the model. That is the work we do at Kalpstack: AI agents, chatbots and RAG systems built for production, along with the web, mobile and data work around them. Book a call and we'll tell you honestly what is wrong and what it would take to fix.
Also read: AI Agency vs Freelancer: Who Should Build Your AI Agent?