Running an AI pilot in a hotel: scope, baseline and a clear stop rule

An AI pilot in a hotel is a time-boxed trial of one tool on one task, measured against a before-state recorded with a timer and a tally sheet. It needs a fixed test set, a named team, success and failure rules written in advance, and a stop rule that ends the trial early if safety, cost or staff acceptance fails.

For example, a hotel runs a six-week trial, everyone agrees the tool is impressive, and nobody can say whether the front desk is faster than in May. The demo had been convincing; the decision still rested on a feeling. This happens because the pilot was never designed to produce an answer, only an experience. After reading this article you will be able to set up an AI pilot project that can succeed or fail honestly: one task, one team, a measured before-state, written rules and a clean exit.

What separates an AI pilot from a demo?#

A demo answers whether a tool can do something; a pilot answers whether it changes one task in your hotel, and what that change costs in review time. The distinction matters because demos are run by people who know the tool's strengths, on cases chosen to flatter it. A pilot is run by your staff, on your work, against your own numbers.

The unit of a pilot is a task with a start and an end, such as drafting the reply to a rate enquiry or writing the shift handover note. Never a department, never a tool. "Improve the front desk" is not a pilotable unit; "draft the reply to a rate enquiry" is.

Three things must exist before day one: a measured before-state, a fixed set of real cases with known correct answers, and a decision rule agreed in writing. A pilot without a stop rule is a purchase with extra steps. The stop rule is what makes stopping a normal outcome rather than a failed relationship.

The output of a pilot is a short written answer that a manager who was not in the room can read, check and act on. Keep the pilot and the rollout separate: the pilot measures, the rollout trains people and changes routines. Mixing them hides the measurement. Say at the start who decides at the end: one named person, not a committee, and a date on which that decision is taken.

For guidance on choosing the task itself, see our article on where to start with AI in a hotel: back office before guests. This article assumes the task is already chosen.

Which task and which team should the pilot cover?#

Choose one task with high volume, a visible before-state, a small blast radius and one named owner, and keep the pilot inside the team that does that task today. Blast radius is the test: a mistake must be caught by a person before it reaches a guest, a payment or a fiscal document. Anything that moves money, notifies a guest identity or issues a document stays out of a first pilot.

Write the task as a sentence with a verb and an object. Draft the reply to a rate enquiry. Summarize the night's guest requests. Prepare the maintenance list from guest messages. Not "improve the front desk".

Good candidates queue up, get redone, or depend on one person who knows how. Bad candidates are rare, seasonal, or already handled well. A task done twice a month gives you no data; a task done twenty times a shift gives you data in a week.

One team means the people on the shifts that actually do the work, including the night shift if the task happens at night. A second department doubles the variables and halves the evidence. Name the operator, the reviewer and the owner before the first day; a pilot with no named reviewer produces output nobody is responsible for.

How do you measure the before-state?#

Measure the current process for a fixed window, on the same task and with the same people, using a timer and a tally sheet rather than recollection. Memory is not an instrument. Nobody remembers how long the rate enquiry took in March; everybody remembers the bad one.

The three numbers worth counting#

Record handling time from the moment the request arrives to the moment the reply is sent, including the review. Record the error count: cases a person had to correct. Record rework: cases sent back or done twice. These three numbers, and no others, carry the whole comparison.

Define the unit precisely and never change it mid-pilot. If the clock starts when the guest writes, it starts there in the after-state too. Measure the slow cases as well as the typical one: a hotel feels its worst hour, not its median, so keep the longest times in the record. Measure review time honestly. If a person must read every output before it is used, that reading is part of the task, not an overhead outside it.

How long the before-state window should be#

Long enough to cover a full weekly cycle of the task, including one busy period. Two weeks is often enough for a daily task; a weekly task needs its own cycle. Log the date, the shift and who measured. An unattributed sheet cannot be defended when someone questions the result six months later. Keep the raw sheet: it is the only artefact that makes the after-state comparable, and it costs nothing to keep.

For the step before the pilot, see our guide on how to test a hotel AI assistant before staff or guests use it.

What are success, failure and stop rules, and when should they be written?#

They are three sentences written and signed before the first day: what counts as success, what counts as failure, and what ends the trial immediately whatever the numbers say. Written after the results, they are not rules; they are a description of what happened, arranged to justify a decision already taken.

A success rule must be falsifiable. "Staff like it" is not a rule. "The typical handling time falls and the error count does not rise, on the same measure" is a rule. A failure rule is not the absence of success; write it separately: no measurable difference after the fixed period, or a difference that costs more review time than it saves.

Stop rules cover safety, cost and people. A data incident, an error that reached a guest, staff declining to use the tool, the vendor failing the agreed test cases, or cost per task above a ceiling agreed in advance. Set the margin before you see the data. Deciding after the results what counts as good is how pilots become purchases.

Write the rules with the vendor in the room. A vendor who resists a stop rule has told you something useful before you have spent anything. Put the rules on one page with the task definition and the before-state numbers, and give a copy to the operator and the reviewer, not only to management.

How do you run the pilot day to day?#

With named people, a parallel run of the old process, one issue log and a short weekly review that reads the same three numbers. The pilot does not need new machinery; it needs the same three numbers, counted the same way, every week.

Run the old process in parallel for a defined period, then set a cutover date. Without a date the parallel run becomes permanent and the comparison never closes. Change one thing at a time: if the task definition, the team or the tool changes mid-pilot, restart the measurement instead of blending two periods.

Keep one issue log, one format: date, task, what the tool produced, what the person did instead, and whether the cause was the tool or the process around it. That last column is the one managers skip and later need. Tell staff what is being measured and that the measurement is of the task, not of them. People who expect a performance review produce different numbers.

Hold a thirty-minute weekly review with the operator, the reviewer and the owner, and read the tally sheet rather than the vendor dashboard. Escalate on the stop rule the day it fires, not at the end of the period. A stop rule that waits for the closing meeting is decoration.

Once a tool is in daily use, the same discipline continues: see our weekly review checklist for a hotel AI assistant after go-live.

How do you read the result honestly?#

Compare the after-state with your own before-state, not with the demo, and report the slow cases as well as the typical one. The only valid comparison is the sheet you filled in before the tool arrived. A vendor benchmark, however well run, measures a different hotel.

Report the typical case and the worst case together. A good average with a bad tail is a bad result for a hotel, because the tail lands at check-in time. Discount the first weeks: early speed often comes from attention, not from the tool, so compare the later weeks with the before-state and say so in the report.

Ask the people who did the work. Their account explains why the numbers moved; it does not replace the numbers. A null or negative result is a result. Written down, it saves the next pilot from repeating the same task choice. The hotel writes its own report. The vendor may add one, but the sheet, the log and the conclusion belong to the hotel.

What exit and data-return terms should you agree before starting?#

Agree in writing what data leaves the hotel, what comes back at the end, in what format, and what the next step costs, before the pilot begins. Once the pilot is running, your negotiating position weakens by the week.

Data in: which fields are sent, where they are processed, how long they are kept, and who at the vendor can see them. Keep guest identity data out of a first pilot entirely. Data out: an export in a usable format, a written confirmation of deletion, and a short handover note. Ask for the format now, not at the end.

Settle ownership of the hotel's own material: the test cases and their correct answers, the configuration, the prompts and the issue log. Say who may take them and who may reuse them. Agree the notice period and the price of continuing; ask what the next step costs before the pilot starts, while you can still decline. For a fuller list, see our article on questions to ask an AI vendor before buying for your hotel, and for cost structures, how AI tools are priced: seats, usage and what drives the bill.

Describe the personal data duties in general terms and have the contract checked by a legal adviser and, where money or documents are involved, by your accountant; the scope depends on your own case. A vendor who will not put deletion and export in writing is a finding, and it is cheaper to find it now.

What to do next#

Before the next demo, write the one-page pilot sheet: the task as a sentence, the three numbers and how they will be counted, the success, failure and stop rules, the named operator, reviewer and decision maker, and the exit and data-return terms. Get it signed by the hotel and the vendor. Everything else in this article is the discipline of filling that page in honestly, and the same three numbers carry into the weekly review once a tool is in daily use.