Should your hotel website let AI crawlers in? robots.txt choices explained

robots.txt is a plain text file at the root of your domain that tells automated visitors, called crawlers, which pages they may read. It is a request, not a lock. Decide per purpose: keep search crawlers in, then choose separately whether training collectors and the fetchers assistants use may read your pages.

For example, an agency adds one line to a hotel's robots.txt to stop AI scraping, and the hotel may drop out of the answers travelers get when they ask an assistant for a room. Nothing breaks, no error appears, and nobody may notice for months. After reading this guide you will be able to name the three kinds of automated readers that visit your site, check what your site, firewall and booking engine actually allow today, and put the decision in writing so it survives the next redesign.

Who actually reads a hotel website today?#

Three kinds of automated readers arrive at a hotel site, and they want different things: search crawlers that build the index behind search results, training collectors that gather text to build future models, and fetchers that read a page at the moment a traveler asks a question.

Search crawlers fetch pages so those pages can be listed and ranked. Google's documentation, last updated 30 October 2026, says rules addressed to the Googlebot user agent affect Google Search, including Discover and all Search features, as well as products such as Images, Video mode and News. If you block this reader, your pages stop being read, and over time they leave the index.

Training collectors gather content in bulk to improve future models. Google handles this with a separate robots.txt token rather than a separate crawler, which is why the same visit can serve two purposes: one fetch feeds the search index, and the content it collected may or may not feed model training depending on how you set that token.

Assistant fetchers work at question time. A traveler asks for a hotel with a pool near a given street, and a page is read then and there to build the answer. Blocking this reader changes what an assistant can say about your property, which is a different question from whether a model was trained on your text. If you are new to this shift, start with what changes when travelers ask instead of browse.

Two practical cautions. First, the same company can appear under several names, and a single provider may publish one page listing all of its crawlers; that page, not a blog post or an agency's memory, is the source to trust. Google's own page warns that user agent strings can be spoofed, so a name in a log is evidence of a claim, not proof of identity. Second, a hotel's booking path often sits on a different host from the marketing site, with its own file and its own settings, so "we checked the website" is only half a check.

Other providers publish their own crawler pages; read the provider's page, not a copied list.

What is robots.txt, in plain words?#

robots.txt is a plain text file served at the root of a domain that a crawler reads before it requests pages; it groups rules by user agent and says which paths that agent may or may not fetch.

The structure is simple: a user-agent line names the crawler, and the allow and disallow lines under it list paths. A group with no matching name is ignored by that crawler. The file governs fetching, not listing: a rule stops a crawler from reading a page, and it does not remove a page that an index already holds, nor does it lock anything.

The file is public. Anyone, including a competitor, can read it, so it also advertises what the hotel is trying to keep out of sight. It is voluntary: a crawler that ignores the file will ignore it whether the rule is written politely or firmly; enforcement belongs to the firewall or the login screen, not to the text file. And it is per host: a subdomain, a staging site and a booking engine on its own domain each have their own file, and a rule on one does not travel to the others.

robots.txt is not the only file people ask about. llms.txt is a proposal for a different purpose, and it is not a switch that turns assistant access on or off; see llms.txt for hotel websites: what the file is, and what it will not do before adding anything to your site.

How do you check what your site allows right now?#

Read the file first, then check every layer in front of it, because a firewall or security setting can block a crawler before robots.txt is ever fetched.

Open the robots.txt address of the main domain in a browser. It is public, so this needs no login and no agency ticket; read every user-agent group and write down which ones disallow the whole site. Repeat the check on the booking engine's host and on any subdomain that serves real content, since each host answers separately.

Then ask what the firewall, CDN or security plugin does to automated visitors. A bot-fighting rule that blocks everything except a browser is stricter than any robots.txt line and overrides it silently. Check page-level signals too: a noindex instruction in the page itself keeps a page out of an index even when robots.txt allows the crawl, and the two settings are often changed by different people.

Ask the agency for the change history: who added which rule, on what date, for what reason. A rule with no owner is a rule nobody will revisit. If server logs are available, look at which user agents actually arrive and which are refused; that shows the real behaviour rather than the intended behaviour. For the wider security picture, our hotel cybersecurity checklist for owners and general managers covers the same layers in more depth.

Should you block AI training crawlers?#

It is a business decision, not a technical one, and it can be made separately from the decision to answer travelers' questions, because training use and answering use are different purposes served by different controls.

For Google, the training decision is made with a separate robots.txt token, Google-Extended. Google's crawler documentation (opens in a new tab), last updated 30 October 2026, says the token has no HTTP user agent string of its own and that crawling is done with existing Google user agents. The same page states that Google-Extended lets publishers manage whether crawled content may be used to train future generations of Gemini models, and that it also controls use for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

It also states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal, which means the training choice and the search choice can be argued on their own merits. The trade-off is real in both directions: blocking training costs nothing measurable in search per Google's statement, but a model that never read the property cannot describe it from memory. No verified figure exists on what either choice does to bookings, so decide on principle, not on a promise.

The vocabulary is still moving. Google says it intends to support the shared AIPREF training-preference signal ("user-agent: _usageai-training") in early 2027, which means today's answer is a dated decision, not a permanent one. Other providers publish their own crawler pages and their own controls; read each provider's page before writing a rule, and never copy a list of names from a forum post. What an assistant says about your hotel also depends on the facts it finds, which is a separate problem covered in how AI assistants describe your hotel: keeping facts consistent.

Which pages should stay out of every index?#

Staff areas, test and staging pages, internal documents and anything whose link carries a token should stay out of every index, and the ones holding personal data need a login rather than a crawler rule.

Administration and staff paths belong nowhere near an index; they are also the pages most likely to be found by someone probing a site. Test pages, old campaign landing pages and staging subdomains confuse both search engines and assistants, because two versions of the same offer compete with each other.

Guest documents, invoices and reservation screens must be protected by authentication and by links that cannot be guessed; a robots.txt line on a public URL protects nothing, and personal data duties apply regardless of what the file says. Be consistent across the layers: a page blocked in robots.txt but linked from the menu can still surface as a bare address in results, because the rule stops reading, not listing.

Give the agency one written list of paths that must never be indexed, with the reason next to each, so a future redesign does not quietly reopen them.

Beside each item, write what protects it: a robots.txt rule, a noindex tag, a login, or all three. Only the last two actually protect anything.

Put the decision in writing and review it twice a year#

Write one page that states the purpose, the token, the person who approved it and the date, then review it every six months and whenever the agency, the firewall or the booking engine changes.

The page should answer four questions: what may be crawled, what may be used for training, what may be read at question time, and who decided each. Set two fixed review dates a year, and add a trigger for change: a new website, a new agency, a new firewall, a new booking engine host, or a provider announcing a new crawler or token.

At each review, re-fetch every robots.txt, re-check the firewall behaviour and re-read the providers' own crawler pages, since names and controls change between reviews. Keep the record short enough that a general manager can read it in two minutes; a decision nobody can restate is a decision that will be reversed by accident. Treat any unexplained change to robots.txt or bot settings as an incident, and ask the agency to report such changes before they go live.

What to do next#

Open your robots.txt today, list every user-agent group you find, and send your agency one written question: what does the firewall block, and what does the booking engine's own host allow? That single check turns a vague worry about AI into a decision you own, can explain to your general manager, and can revisit on a fixed date.