Learn DNS · Found by search engines and AI
Search engines may read your site (robots.txt and AI crawlers)
In one lineReviewed 2026-11-13The whole idea in one short sentence. The rest of the page explains it step by step.
robots.txt is a sign on your website's door telling robots which rooms they may enter. A wrong line can lock out search engines or AI assistants by accident. You can also choose to block AI training while still letting AI search find you.
What it is
The file lives at /robots.txt. DNSFix reads it and checks whether the big search engines and the main AI assistants are allowed. We split AI robots into two kinds:
- AI search assistants that answer customers' questions and link to you. Usually worth allowing.
- AI training robots that collect text to train models. Blocking them is your choice and doesn't hurt search.
What happens if it's not fixed
Disallow: /for everyone hides your whole site from search.- Blocking AI search assistants means they can't recommend you.
How to fix it
- Open
yourbusiness.com/robots.txtand look forDisallow: /lines. - Remove any that block Googlebot, Bingbot or the AI assistants you want.
- Add a
Sitemap:line with your sitemap address.
A simple, open robots.txt:
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xml
Related
- Search engines can list your home page (noindex)
- No firewall blocking AI assistants
- Sitemap for search engines
- llms.txt
This page explains: Search engines may read your site, AI crawlers (part of Search engines may read your site).