How AI support chats read your site: Intercom, Zendesk, Chatwoot, Crisp and Tolkly

In short. Almost every major AI chat reads a site by itself: Intercom Fin, Zendesk, Chatwoot, Crisp and Tolkly. You give an address, and the bot finds the pages through the sitemap or by following links. They differ elsewhere: which pages the bot sees, how often it reads the site again, and what it does with a page once read. Intercom reads the most: it runs JavaScript and takes up to 4,000 pages. Chatwoot turns pages into question-and-answer pairs of its own. Tolkly keeps pages word for word, reads the site again every day, and checks every quote in an answer against the page's text in code.

How do they compare?

	How it finds pages	sitemap.xml	JavaScript	robots.txt	How often it reads again	Limit
Intercom Fin	A domain and extra start URLs, then links	Yes, a setting or a start URL	Yes: Fast, Auto and Full modes	Obeyed if turned on; off by default	Usually weekly; paused after 90 days if no page of the source was used in an answer	4,000 pages per source
Zendesk (AI agents Advanced add-on)	A start URL, then the sitemap or links	Yes, nested ones too	Yes, plus a fast mode without it	Not stated	On a schedule (weekly or monthly advised) and by hand	Up to 500 start URLs per crawler
Chatwoot Captain	A page and the links under the same path	Not stated	No: an empty page should be replaced with a PDF	Obeyed; if robots.txt blocks the site, no pages are added and no error is shown	Monthly on Startups, weekly on Business, daily on Enterprise	500 pages per crawl, 2,000 documents per account
Crisp (Hugo AI Agent)	A domain, filters by path and language	Not stated	No	Not stated	By hand only	1–10 domains by plan
Tolkly	The sitemap first; without one, links from the home page	Yes: from robots.txt, the usual address or your link; gzip too	No: reads the HTML the server sends	Obeyed; lifted for sections of your own site	Daily at 04:00 Moscow time; weekly or by hand if you choose	2,000 pages per site, 20 sites, 100 files

Sources and dates: the help centres of Intercom, Zendesk, Chatwoot and Crisp, read 2026-11-09 (links at the end). «Not stated» means the help pages are silent; the feature may exist without being described.

Why does it matter how a bot reads a site?

A bot answers only from what it has read. Most mistakes of an AI chat on a site start here:

The page is not in the index. It is missing from the sitemap, no link leads to it, robots.txt closes it, or it needs a sign-in. The bot then refuses or, worse, answers from a similar page.
A script draws the text. If the rules appear only after JavaScript runs, a bot without a browser sees an empty page. Intercom and Zendesk handle this; Chatwoot, Crisp and Tolkly do not.
A menu and footer on every page. The same blocks crowd search, and the bot finds the menu instead of the rule. Intercom removes navigation, headers and footers by default; Zendesk removes what you name with CSS selectors. Tolkly keeps repeated blocks once.
An outdated page. You changed a term on the site, and the bot answers by the old one for a week. The less often a bot reads the site, the longer it is wrong.
The site broke. If the site showed a placeholder or a bot wall during a run, a bot without a check may replace every page with that placeholder.

How do Intercom and Zendesk read a site?

Intercom Fin takes a domain and follows links to the pages under it; a sitemap can be turned on in settings or added as a start URL. Unwanted pages are excluded with URL globs, unwanted parts of a page with CSS selectors. Full mode opens each page in a browser and runs JavaScript. Fin takes up to 4,000 pages from one source. It reads again usually once a week; if no page of a source was used in a Fin answer for 90 days, the schedule pauses. Whether robots.txt is obeyed is up to you, and the toggle is off by default.

Zendesk has found pages by itself since May 2026, through the sitemap or links: a start URL is enough. You can keep only the elements you need on a page, or remove the rest, with CSS selectors. The crawler has a fast mode without JavaScript and a slower one with it. Zendesk advises a weekly or monthly reimport.

For reading a site, these two are the strongest tools in this comparison. If a script draws your rules and the site cannot change, that is a real reason to pick them.

How do Chatwoot and Crisp read a site?

Chatwoot Captain reads a page and the links under the same path, up to 500 pages per crawl. From each document it writes question-and-answer pairs, which go live at once with no approval. The bot answers from those pairs, not from the original page, so it cannot show a verbatim quote from your rules under an answer. How often Captain reads the site again depends on the plan: monthly on Startups, weekly on Business, daily on Enterprise.

Crisp reads a domain and does not run JavaScript. It does not read the site again by itself: you refresh it with a button.

How does Tolkly read a site?

The sitemap first. Tolkly takes the sitemap named in robots.txt, the one at the usual address, or the one you link. Without a sitemap it follows links from the home page. A page missing from the sitemap can be added by hand.
Only the sections you need. Pages under the site's address are read. Sections can be included or excluded with star masks: /blog/*/2024/.
robots.txt is obeyed. You can lift it for a section of your own site, but the app asks you to confirm the site is yours and keeps your name and the date.
Menu and footer once. A block repeated on half the pages or more is kept on one page and does not crowd out the rules.
Every day. The site is read again at 04:00 Moscow time; weekly or only on request if you choose. Changed pages are updated, pages gone from the sitemap leave the knowledge base.
A suspicious run waits for you. If one run would change or remove more than 45% of a site's pages, Tolkly writes nothing and keeps answering from the pages it had until you decide. That is what a bot wall, a parked domain or a broken site looks like.
Copies of your site only with your consent. By default, a new domain the chat appears on gets into answers only once you allow it. Otherwise a copy of your site with one changed detail, such as payment details, could get into answers.
Pages are kept word for word. Every claim in an answer carries a source number and a quote, and code checks the quote against the page's text. A sentence without that check passing is not shown.

What Tolkly cannot do: it does not run JavaScript and does not read pages behind a sign-in. If a site was read badly, the app shows a ready instruction for the site's developer: list the pages in the sitemap, render them on the server, open them in robots.txt. Rules that are not on the site go in as files: PDF, DOCX, Markdown, TXT, HTML.

PaywireBy the site's rules

Hello! I am the AI assistant of Paywire: I answer from the service’s rules or bring in an operator.

How long does a transfer to a card take?

A transfer arrives within 30 minutes after the payment is confirmed.

1cl. 2.1Transfer terms, 2. Timing

Checked against the rules: 1 of 1

Runs on Tolkly

A Tolkly answer: a source number on the claim and the clause under the answer Example: the service is invented.

How do you check that a bot has read your site?

Fetch the rules page with curl https://your-site/page or open «View page source». If the rules are not there, a script draws them, and a bot without a browser will not see them.
Check that the page is in sitemap.xml and not closed in robots.txt.
After the first run, compare the number of pages read with the number of pages on the site.
Ask the bot 20–30 questions from last month's real chats and look at which page it answered from.
Change one term on the site and check when the bot starts answering by the new one.

Which to pick?

Intercom or Zendesk, if a script draws your rules and the site cannot change, if you have more than a few thousand pages, or if you need to cut parts of pages by CSS. Mind the price: Intercom charges per seat and per Fin outcome, and Zendesk's crawler for AI agents is in the AI agents Advanced add-on.

Chatwoot, if you want an open platform and are fine with the bot answering from question-and-answer pairs it wrote itself.

Tolkly, if your site sends its text from the server, your rules change often, and every answer must rest on a verbatim quote from your page, with an honest refusal and a handover to an operator when nothing confirms it.

Try Tolkly

Enter your site's address in «Try it on your site» at tolkly.io/#try: Tolkly reads a few pages and answers from them with quotes. The first month is free, no card needed.

Frequently asked questions

Which AI chat can read a site by its sitemap?

Intercom Fin (a setting, or a sitemap added as a start URL), Zendesk (finds the sitemap itself, nested ones too) and Tolkly (takes the sitemap from robots.txt, the usual address or your link). The help centres of Chatwoot and Crisp do not mention sitemaps: they follow links from the address you give.

Does an AI bot see text drawn by JavaScript?

Intercom and Zendesk do: both have a mode that opens the page in a browser. Crisp and Tolkly read only the HTML the server sends; Chatwoot advises uploading a PDF for such pages.

How often does an AI chat read a site again?

Tolkly: daily by default. Intercom: usually weekly, and the schedule pauses if no page of the source was used in an answer for 90 days. Chatwoot: monthly, weekly or daily by plan. Crisp: by hand only.

Does the bot read pages closed in robots.txt?

Chatwoot and Tolkly obey robots.txt. In Intercom it is a toggle, off by default. In Tolkly you can lift it for sections of your own site, and the app keeps who confirmed it and when.

What if the bot does not see the page with the rules?

Check that the text is in the HTML the server sends, that the page is in the sitemap, not closed by robots.txt and not behind a sign-in. Upload rules that are not on the site as a file.