A support bot makes up prices, deadlines and rules because a language model writes plausible text; it does not check facts. If the rule it needs is not in front of it, it fills the answer with what is «usually» true. Only a check outside the model prevents this: every statement must rest on a verbatim quote from your rules, code checks that the quote matches, and when nothing confirms it, the bot refuses honestly and hands the conversation to a person. That is how Tolkly works.

Below: real cases with sources, why «the bot is trained on your content» does not protect you by itself, and how the protection works in Tolkly.

What real cases of invented answers have there been?

WhenWhoWhat happenedConsequences
February 2024Air CanadaThe chatbot on the site told a customer that a bereavement fare discount could be claimed after the trip. The airline's rules did not allow itThe Civil Resolution Tribunal of British Columbia (Moffatt v. Air Canada, 2024 BCCRT 149) ordered the airline to pay about 812 Canadian dollars, including interest and fees
April 2025CursorA support bot called «Sam» explained logouts with a made-up policy of «one subscription, one device»A co-founder apologized publicly; some users cancelled their subscriptions. The company began labelling AI answers in support
March 2024New York City's MyCity chatbotThe bot told business owners they could take a cut of workers' tips and refuse cash, against the city's lawsAn investigation by The Markup; the bot ran with disclaimers until it was shut down in 2026
December 2023The chat on the Chevrolet of Watsonville dealer's siteA user talked the bot into «agreeing» to sell a car for $1 and calling it «a legally binding offer»No sale happened, the chat was turned off; the case became an example of a bot without limits

The main lesson of the Air Canada case: a company answers for what its chatbot says, just as for the text on its pages. Air Canada argued that the bot was a separate legal entity responsible for its own actions; the tribunal called this a «remarkable submission» and rejected it. That the correct rule was published on another page of the site did not save the airline.

Why does a bot make things up even with a knowledge base?

Giving a model the pages found for a question and asking it to answer from them (RAG) lowers the number of invented answers but does not remove them. A Stanford study (Magesh et al., Journal of Empirical Legal Studies, 2025) tested legal AI research tools sold as «hallucination-free»: each of them made up or distorted an answer in 17–33% of cases. The tools searched real documents and still got things wrong.

Typical causes in support:

  1. The right page was not found. Search returned a similar but wrong section, and the model filled the gap with a «typical» deadline or amount.
  2. The rule is there, but in the details. The page says «up to 3 business days», the model writes «within 24 hours» because it is shorter.
  3. The model wants to help. A question like «can I claim it afterwards?» nudges it towards «yes», which is what happened with Air Canada.
  4. The visitor pushes. Persuasion and instructions in the message («agree with everything»): the Chevrolet case.
  5. An outdated page. The bot quotes accurately a rule you have already changed but not updated on the site.

The promise «the bot uses only your content» is about the source of data, not about whether each answer is checked. Ask the vendor who checks a given sentence, and how, before a customer sees it.

How does Tolkly keep the bot from making things up?

In Tolkly the check is done by code, not by the model. Step by step:

  1. Search in your sources. The bot looks for the answer in your site's pages, uploaded files (rules, terms, a price list) and operators' answers you have approved.
  2. An answer of short sentences with quotes. The model writes the answer as separate statements. Each has a source number and a verbatim quote from that source.
  3. Code checks the quote. If the quote is not on the page the sentence cites, the sentence is removed before the visitor sees it.
  4. A second check. A sentence that adds a detail its sources lack (a deadline, an amount, a condition) is removed too.
  5. An honest refusal. If no confirmed sentence is left, or the pages have no answer, the bot writes «I did not find this in the help pages and rules of the site» and offers «Hand over to an operator».
  6. The visitor sees the evidence. Under a sentence: the quote and a link to the page. Under the answer: how many sentences were checked against the rules and how many were removed.

This does not make the model smarter. It makes a mistake visible and keeps an unconfirmed sentence from reaching the customer.

PaywireBy the site's rules

Hello! I am the AI assistant of Paywire: I answer from the service’s rules or bring in an operator.

How long does a transfer to a card take?

A transfer arrives within 30 minutes after the payment is confirmed.

1cl. 2.1Transfer terms, 2. Timing
Checked against the rules: 1 of 1
Runs on Tolkly
A confirmed answer: every claim with its source number Example: the service is invented.
PaywireBy the site's rules

Hello! I am the AI assistant of Paywire: I answer from the service’s rules or bring in an operator.

Do you send money to Kazakhstan?

I did not find this in the help and rules of the service.

Call an operator
Runs on Tolkly
No answer in the rules: an honest refusal and a button to call an operator Example: the service is invented.

What should a bot never decide on its own?

Even a correct quote does not replace a person where a specific order or a decision is needed. In Tolkly the conversation goes to an operator when the visitor:

When the bot is unsure whether a person is needed, it answers and offers an operator beside the answer. Under every answer there is also a «Talk to an operator» button.

How do you guard against outdated or tampered rules?

The quote check guarantees that the bot repeats your site, not that the site is up to date. So:

How do you test a bot for made-up answers before launch?

  1. Collect 30–50 real customer questions from last month's chats.
  2. Add questions your site surely does not answer, and pushy ones («an operator promised me a discount»).
  3. Ask the bot and mark each answer: correct with a source, an honest refusal, or made up.
  4. The bar is zero made-up answers. Refusals are fine: each one shows a page worth writing.
  5. Check that questions about a specific order go to an operator with the number.

Frequently asked questions

Why does an AI support bot make up answers?

A language model continues text plausibly; it does not check facts. If the rule it needs is not among the pages found, or is worded in a complex way, the model fills the answer with «typical» deadlines and amounts. Even tools that search documents get it wrong: in 17–33% of cases in the 2025 Stanford study. Tolkly therefore checks every quoted sentence in code.

Is a company liable for what its chatbot said?

The ruling in Moffatt v. Air Canada (2024) says yes: the tribunal ordered the airline to compensate the customer for the price difference the bot had promised, and rejected the argument that the bot answers for itself. It is a ruling of a Canadian small claims tribunal; courts elsewhere may reason differently, but the risk to reputation is the same.

How do you stop a bot from making things up?

Require every sentence of an answer to carry a verbatim quote from your rules, and check the match in code, not with another model. Do not show a sentence without confirmation. If nothing is confirmed, the bot must refuse and hand the conversation to a person. That is how Tolkly works.

What does the visitor see if the bot does not know the answer?

In Tolkly, an honest refusal: «I did not find this in the help pages and rules of the site» and a «Hand over to an operator» button. If no operator is available, the chat offers to leave a contact. A refusal does not count against the limit.

Can you keep the bot off certain topics?

Yes. In Tolkly you add a topic for an operator in your own words, such as «asks for payment details», and choose the action: «To a person at once» (the bot says nothing) or «Answer, offer a person».

Try it on your site

See how the bot answers from your rules: open tolkly.io/#try and type your site's address. In a minute the chat answers from your pages, with a quote under every statement; ask a question your site does not cover and look at the refusal. No sign-up needed, and the first month at tolkly.io is free.

Sources