An answer engine – ChatGPT, Claude, Perplexity, Google's AI overview – retrieves a chunk of your page, usually one heading's worth of text, and answers from that chunk alone. It does not read your page top to bottom, and it does not see the sidebar that gave the page its context.
So the practical question is not "is my docs site indexed". It is: would this paragraph, read entirely on its own, be a correct and complete answer? Everything below follows from that.
What Dewstack already does for you
You do not need to configure any of this. On a public workspace, Dewstack:
- Turns your H2s and H3s into anchored table-of-contents entries with stable id slugs, so every section has its own linkable URL. H1 and H2 form the top level of the ToC, H3 and H4 the indented level. Those same headings are the natural chunk boundaries a retriever splits on.
- Emits
schema.org/ArticleJSON-LD in every page's head, with the headline, description, URL, published and modified dates, the site it belongs to and the publisher. - Sets a canonical URL on every page, so a page reachable by more than one path consolidates to one.
- Publishes a sitemap at
/sitemap.xmlcovering every published page and child page. - Publishes
/llms.txt,/llms-full.txtand a.mdalternate for every page, so a model can read your content without wading through HTML.
The sitemap follows the same rule as everything else: it is served for public workspaces and
returns 404 for private and password-protected ones, and it lists only pages with status
Publish.
The No-Indexing toggle suppresses all of this
Under Customize → SEO there is a No-Indexing switch. Turn it on and every page is served
with robots: noindex, which asks search engines and most crawlers to leave your docs out
entirely. It is the right setting for a staging workspace and the wrong one for a live docs site –
check it first if your pages are not showing up anywhere.
What you have to do yourself
Answer in the first paragraph under each heading
Put the direct answer immediately under the heading, then elaborate. A chunk that opens with setup, history or a list of caveats gets retrieved and then discarded, because the first sentences did not match the question.
Bad: "There are a few things to know about custom domains before we begin. Historically, DNS…"
Good: "Add a CNAME record for your subdomain at your DNS provider, then click Verify & Save. Propagation can take up to 24 hours."
Write headings as the question a reader would type
"Point a custom domain at your docs" retrieves. "Domains" does not. The heading is often the strongest single signal a retriever has about what the chunk underneath is for, and it is what lands in your table of contents anyway.
Spell out the subject in every chunk
A chunk is read in isolation, so "it" and "this" have no referent. Write "Dewstack serves an
llms.txt at your docs root", not "it serves one at the root". Same rule for acronyms: expand
each one the first time it appears on each page, because the model may never see the page
where you defined it.
Give the number, not the noun
"60 requests per minute" beats "the rate limit". "Up to 500 published pages" beats "a generous cap". Concrete values survive summarisation; vague ones get dropped or, worse, invented.
Make the first sentence carry the page
Your llms.txt excerpt is generated from the start of the page body and truncated to about 160
characters. That sentence is what an agent sees when deciding whether to fetch the page at all. If
it reads "Welcome to this guide!", the agent learns nothing and moves on.
Keep one topic per page
If a page needs "and" in its title, it will produce chunks that answer neither half well. Split it. Two focused pages outrank one page that covers both.
A quick check before you publish
- Read each H2 and the paragraph under it, and nothing else. Does it answer the heading?
- Replace every "it", "this" and "the above" that has no subject in the same paragraph.
- Replace every vague quantity with the real number.
- Open the page's
.mdURL and skim it. That is what the assistant reads.
