# Knowledge base and reindexing

> What the bot reads, how often it reads your sites again, how it picks up mirrors and learns from operators' answers.

The bot answers only from your knowledge base. It holds the pages of your sites we have read and the questions and answers you have approved or written. This page covers what we read, when we read it again, and how to keep your rules where the bot can see them.

## What we read
- **The sitemap first.** We take the pages from your site's sitemap: the one named in `robots.txt`, or the one at the usual address. A gzipped sitemap works too.
- **No sitemap, then links.** A site with no sitemap is read from its home page, following links.
- **Only pages under the site's address.** If the site's address is `example.com/help/`, we read only pages under `/help/`.
- **`robots.txt` is respected.** We don't read pages it closes. For your own site you can lift it for some addresses: «Advanced settings» › «Read despite robots.txt». The app asks you to confirm the site is yours and keeps your name and the date.
- **HTML pages only.** We don't read PDF files or documents the site links to: upload them as files ([Files](#files)).
- **No JavaScript.** We read the HTML the server sends. We don't see text a script draws after the page loads.
- **Your menu and footer once.** What repeats on every page is kept once, so it doesn't crowd out the rules.

If a page isn't in your sitemap, add it to «Extra pages». We read up to 2,000 pages from each site.

## Sites
The project's «Knowledge» section opens with «The AI knows: 2 sites, 12 files…» and «Read all again». Each count links to its part: «Sites», «Files» or the «Q&A» section. Below are [what the AI knows about the project](#card), the sites, the files and one line on what can be connected soon. Whether the chat's new domains are read is set in «Install» › «Domains» ([The domain rule](#domain-rule)).

«Sites» lists your sites and the sites found on your domains, one row each. A row shows how many pages were read and when. The common schedule is said once in the part's note; a site on another one names its own. A dot opens the grey line only when a site needs you, with what is wrong in a few words, such as «no links seen». For all such sites, one «Hand it to an AI agent» row at the end of the part holds a prompt for the developer.

An owner or admin presses «Add a site». A site needs only its address. The rest is under «Advanced settings», a line per setting: what it is now and «Edit». Its editor opens under the line, one at a time; the whole form saves at once.

- **Sitemap:** we find it, you give its address, or there is none and we follow links.
- **Which pages:** all, only the sections you list, or all but them.
- **Extra pages:** addresses the sitemap does not list.
- **Read despite robots.txt:** sections robots.txt closes to reading.
- **Name:** the site's address by default.
- **Read again:** as every site (every day), every week, or only when asked.

A section is the start of a page's address after the domain: `/ru/faq/` is every page whose address starts so. A star stands for any characters, slashes too: `/blog/*/2024/` or `*/archive/`. Nothing needs escaping, and a whole address pasted in loses its domain by itself. «Own expression» stays for those who write regular expressions.

A project has up to 20 sites. The first site is the main one: the site card and the widget's colours come from it. If another source already holds a page's text, it is kept once.

«…» on a row edits or removes the site. A removed site's pages leave the knowledge base at once. A project always keeps at least one site of its own. Operators see the list without the buttons.

## Files
Rules, terms of service, an AML policy or a price list that are not on your site go in as files: drop them on «Files» or press «Upload files». Files show as a grid of cards: a mark with the extension, the name, the size and the number of fragments. Past 12 files a search and an order appear, and the grid folds after two rows. The box in a card's corner selects it, to remove several files at once.

- **Formats.** PDF, DOCX, Markdown, TXT and HTML, up to 10 MB each. We tell the format by the file itself. Open an old `.doc` in Word and save it as `.docx`.
- **Headings.** The bot splits a file into sections by its headings. In a PDF a heading is a line set larger than the text; in Word it is a heading style or a short paragraph that is bold all through. Page numbers and running headers are left out.
- **The link in an answer.** A file has no address on your site, so the bot links to a page with its text, exactly what it answers from, scrolled to the right section. The file's name in its source row opens that page.
- **Scans.** A scan holds pictures of pages, no text. Such a file is not read, and its row says why. Save the document with a text layer.
- **Long files.** We read up to a million characters of a file. If it is longer, the row says only its start was read.
- **A new version.** Drop the new file on the old one's card, or choose «…» › «Replace the file». A dialog shows both files, «Now» and «New», and asks to confirm. A file uploaded under a name that is a source already gets «Replace the file» or «Add as new». The bot reads the new version at once; links in earlier answers open the new text. A file is never read again by itself: it changes only when you upload a new one. The same file is not uploaded twice.

A project holds up to 100 files; they do not count toward the 20 sources.

## The domain rule
Mirrors change faster than anyone edits settings. In «Install» › «Domains», the «Read new domains» row turns the domain rule on: it reads your new domains by itself. Above it in the same block are the domains the chat loaded on and what the AI does with each; a domain that waits for you has «Allow».

- **The mask.** The rule looks only at domains that match its mask: «Advanced: the mask and a new domain» › «Domain mask». The syntax is the same as in the [domains list](https://tolkly.io/docs/en/domains.md#strict). By default it is the subdomains of your main site, for example `*.brand.com`. For numbered mirrors write `brand*.com`: a star stands for any letters and digits. The mask saves when you leave the field. There, too, «Add a domain» puts your domain on the allowed list ahead of time: the AI reads it once the chat loads there.
- **«Which domains»: «Only allowed» or «All but blocked».** Only allowed by default: a domain counts only once you allow it, in «Install» › «Domains», in Telegram or with your mirror script. Without this, a copy of your site with one changed detail, such as a wallet address, could get into answers by itself. A blocked domain is never read.
- **A copy is a mirror.** We compare a new domain with your main site: up to 20 pages, starting with the home page, text against text, and the home page's `rel=canonical`. We don't read a copy again: answers to a visitor there already link to that domain.
- **New text is a site.** A domain with text of its own becomes a site of the project and is read every day. The company's Telegram group gets a message about each one: «a mirror of …» or «a new site … is indexed».
- **When it runs.** When you allow a domain, and once a day. The list shows the domains the chat has been on and what the rule does with each. If you take a domain off your list or block it, its site and its pages go on the next check.

## Runs that wait for you
If one run would change or remove more than 45% of a site's pages, nothing is written. The bot keeps answering from the pages it had. That is what a bot wall, a parked domain or a broken site looks like.

The row's dot turns orange and the row says what changed: «11 changed, 1 gone of 12 pages». The company's Telegram group gets a message too, with a link to the sources.

- If the site really changed, press «Accept changes»: the site is read again and written.
- «Reject» keeps the old pages. The next run checks again.

Sites under 10 pages are never held.

## Make your rules readable
- Put rules, limits and terms on an ordinary HTML page, not on an image or in a pop-up drawn by a script. Rules that exist only as a PDF go in as a [file](#files).
- List that page in your sitemap.
- Don't close it in `robots.txt`.
- Keep it under the site's address.
- When you change a rule, change the page too: the bot quotes what the page says.

### Hand it to your site's AI agent
If a site's last reading went badly, owners and admins see a «Hand it to an AI agent» row in «Sites». It appears when:

- `robots.txt` closes the home page;
- fewer than 5 pages were read;
- pages came with no text: scripts draw them, or they are behind a sign-in.

The prompt asks the coding agent that edits your site to:

- list the help pages in the sitemap;
- render them on the server;
- let us in through `robots.txt`;
- keep the FAQ outside a sign-in;
- check everything with `curl`.

Below is the prompt with a sample site. «Open in…» → «Prompt for an AI agent» copies it.

```text
Make this website readable for the Tolkly support chat: it answers visitors only from the site’s own pages, and its crawler reads them badly now.

Site: https://your-site.example
Crawler: the user agent "TolklyBot/0.1 (+https://github.com/tolkly/tolkly)"; in robots.txt it reads the groups for TolklyBot and for *

1. Read https://tolkly.io/docs/en/knowledge.md. If it does not load, this prompt is enough.
2. Find the pages with the answers visitors need: help, FAQ, rules and terms, prices and limits, contacts.
3. Serve a sitemap.xml that lists every one of them, at https://your-site.example/sitemap.xml or named by a "Sitemap:" line in robots.txt. When a sitemap exists the crawler reads only the pages it lists and follows no links, so a page missing from it is never read.
4. Render those pages on the server: their text must be in the HTML the server sends, with no JavaScript run. In a single-page app use the framework’s server rendering or prerender these routes; an empty shell that scripts fill in reads as a page with no text. Leave no <meta name="robots" content="noindex"> on them.
5. In robots.txt, let the crawler in: no Disallow line in the group for TolklyBot or for * may cover the home page or these pages. An Allow line does not reopen what a Disallow closes for this crawler, so remove or narrow the Disallow instead. Do not turn the user agent away at a firewall, CDN or bot protection either.
6. Keep the FAQ and the rules outside a sign-in: a page that asks the crawler to log in reads as a page with no text.
7. Check it: curl -A "TolklyBot/0.1 (+https://github.com/tolkly/tolkly)" <a help page’s address> prints the answer’s text in the HTML, https://your-site.example/robots.txt closes none of these pages, and the sitemap lists them. Then list the files you changed, and tell me to press «Read again» on the site in Tolkly’s «Knowledge» section.
```

## The daily reindex
Each site is read again on its own schedule, at 04:00 Moscow time: every day (the default) or every week. A site set to «Only when asked» is never read again on its own. The site's form shows that time in your project's time zone, as in «Read again every day at 04:00 (Moscow Standard Time)».

- Changed pages are read again; the rest are kept as they are.
- Pages gone from your sitemap leave the knowledge base. A site read by following links loses no pages because one request failed.

## Reindex by hand
If a change must count today, an owner or admin presses «Read again» (the arrow on the site's row) or «Read all again» above the list. What a row may say:

| Words | What they mean |
|---|---|
| «Not read yet» | Added, not read yet (orange dot) |
| «Waiting for its turn…» | Another site or project is being read first |
| «Reading the site now…» | The run is going |
| «Read … · Every day · next …» | Done |
| «Not read …: the site did not answer» | The run failed (red dot). It may also say «the site turned us away (a bot wall or a login)», «the site answered with an error», «the server restarted mid-run» |
| «… changed, … gone of … pages» | The run waits for your decision (orange dot) |
| «robots.txt closes the home page to us» | Your `robots.txt` closes the home page itself to the bot (red dot) |

## When a run fails
A failed run is not retried until the next scheduled time. Once your site answers again, press «Read again». Until then the bot keeps answering from the pages it read before. If it keeps failing, tell us.

## What the AI knows about the project
With every question, the AI is told what your project is, whatever domain the chat is on. By default that is the title and description of your main site's home page. It is the block «What the AI knows about the project» at the top of «Knowledge». An owner or admin presses «Edit», writes it in their own words (up to 1,500 characters) and presses «Save». The main site is then read again to take it in. «Use the home page's» goes back to the default.

## Q&A
When a conversation an operator answered is closed, the AI saves the operator's answer as a draft: a question and an answer, in the operator's words only. Drafts wait in the project's «Q&A» section, under «New». The most asked come first («Asked 3 times»).

You can also write an answer yourself, without changing the site or uploading a file: «Write an answer» in «Q&A», «Answer» beside a question in «Reports» › «Missing from the knowledge base», or «Write the answer» beside a frequent question the bot would refuse. «Save and approve» puts it in the knowledge base at once; «Save to New» leaves it for another look.

- If a close question is there already, its answer is shown first: edit that one, or «Save anyway».
- The question becomes the title of the answer's page, so it cannot hold an order or transaction number, an email or a phone.
- For something passing (withdrawals delayed today), set «Answer until» under «Advanced»: after that day the bot stops answering from it and its page closes.

- **«Approve»**: from now on the bot answers that question itself. In the chat it cites «Support answer» («Ответ поддержки» in a project set up in Russian). The answer has its own page.
- **«Edit»**, then «Save and approve»: fix the words first.
- **«Dismiss»**: if the question is asked again, no new draft appears. «Approve after all» brings it back.
- **«Withdraw»** on an approved answer: the bot stops answering from it at once, the answer's page closes, and the answer moves to «Dismissed».

Only owners and admins can write, approve, edit and dismiss. Operators see what became of their answers.