After go-live: a weekly review checklist for a hotel AI assistant

AI assistant monitoring is the weekly habit of reading a sample of the assistant's guest conversations and checking each one against the hotel's own facts, its handover rules and what the guest did next. It exists because launch-day testing only proves the assistant on the questions you thought of; week six brings new menus, prices and staff.

The assistant passed every test you wrote before launch. Six weeks later a guest asks about the new autumn menu, and it answers with last season's prices, politely and with confidence. Nothing broke; the hotel simply moved. This article gives you the weekly half hour that catches that drift: how to sample transcripts, how to classify a wrong answer, how to read handovers, and when to rerun the original test set. If you have not built a test set yet, start with how to test a hotel AI assistant before staff or guests use it.

What is AI assistant monitoring, and why does week six differ from launch day?#

AI assistant monitoring is the weekly practice of reading a sample of the assistant's real conversations and checking each against the hotel's own facts, its handover rules and what the guest did next. Launch-day testing proves the assistant only on the questions you thought to ask. It compares the assistant against a fixed set of cases with pass rules you wrote yourself; it is a snapshot, valid on the day it runs and no longer.

After go-live, three things move. The hotel's facts move: menus, prices, spa hours, seasonal offers. The instructions move: tone, what to refuse, when to hand over to a person. And sometimes the model behind the assistant moves, when the supplier updates it. Monitoring is therefore mostly fact-watching, not model-watching. As a working rule, check the fact sheet first, before suspecting the assistant's reasoning: a fact sheet that no longer matches the hotel is the first thing to rule out.

The review produces three short lists: fix now (a guest could be misled today), fix this week, and watch (one occurrence, no action yet). It is not a satisfaction survey and not a dashboard. A dashboard shows how many conversations happened; it does not show whether the answers were right.

Write the review down. A review that leaves no trace cannot be compared with next week's, and the comparison is where the value sits.

Who owns the weekly review, and when should it happen?#

One named person owns it, with a named deputy, and it happens in the same thirty-minute slot every week. The owner is usually the front office manager, or whoever was named as the tool's owner at launch. The general manager reads the one-page note, not the transcripts.

The owner needs two powers: the power to change the fact sheet, or a same-week route to whoever can, and the power to switch a category of question off so it goes straight to a person without waiting for a meeting. The deputy matters because the review is the first thing dropped in a busy week, and a busy week is exactly when the assistant is answering the most messages.

Time-box it at thirty minutes, on the same day each week, before the rota is set, so fixes land in the week they were found. The output is a dated one-page note with the three lists and the fact-sheet version it refers to, so a later change can be traced back to the week it was decided. Whoever writes guest-facing wording should sit in for the first ten minutes, or at least read the note before the next week.

If the review sits inside a wider trial, fold it into the discipline described in running an AI pilot in a hotel: scope, baseline and a clear stop rule.

How do you sample transcripts in a small hotel?#

Sample by rule, not by mood: read every conversation that involved a handover, a complaint or a repeated question, plus a fixed handful of ordinary ones. Pure random sampling hides the failures that matter, because failures cluster in a few categories rather than spreading evenly.

If the week produced few conversations, read all of them. A standing rule of "all, up to a fixed number, then the buckets below" keeps the sample honest as volume grows. Read the whole thread, not the last message; the failure is often the assistant's second answer, after the guest said "no, I mean".

Note what the guest did next: booked, called reception, wrote again, or stopped replying. That next step is the closest thing to an outcome measure the hotel has. Read the guest's words, not the assistant's: the guest's phrasing is the input the fact sheet has to match, and it is what you will paste into the fix. Keep the sample in one place with its date; the same conversation read in two consecutive weeks is itself a signal that nothing was fixed.

For the handover bucket, the routing rules in AI replies to guest messages: when to hand over to staff are the reference you are checking against.

How do you classify a wrong answer, and what do you do with it?#

Classify by source, because the source decides the fix: a missing fact, a stale fact, a wrong instruction, or a fact that was present and misread. A missing fact means the guest asked something the fact sheet never covered; the fix is to add the fact, using the guest's own wording as the question it must answer. A stale fact means the sheet says one thing and the hotel does another; the fix is a change process, not a wording change, so ask why the price or the opening time moved without the sheet.

A wrong instruction means the assistant answered a question it was told to route to a person; the fix is the instruction, and probably a new test case so it does not come back. A misread fact means the fact was there and correct. This is the only category where the model or the prompt is the suspect, and it needs several examples before anyone changes anything.

Severity is a second axis. A wrong answer the guest could not detect is worse than one they corrected in the next message, because only the second one leaves a trace. Log the guest's exact wording next to the fix, so the next review can search for that phrase and confirm it is gone. What you keep, and how tamper-evident it is, matters too; see AI audit trails in hotels: what a good record should show.

How long did guests wait for a handover?#

Measure from the guest's message to a person's first reply, not the assistant's response time. The assistant answers in seconds; the risk sits in the gap after it steps aside. Record three timestamps per handover: guest sent, assistant handed over, person replied. The third minus the first is the number that matters.

The assistant's own speed is not a quality measure. A fast wrong answer is worse than a slow correct one, and a fast handover to an empty desk is worse than both. Set the hotel's own limit by time of day; for example, ten minutes during reception hours, and a first reply by 09:00 for messages that arrive overnight. The limit is the hotel's decision, not an industry figure.

Count handovers that were never picked up at all. Those are the ones guests remember, and they never appear in a satisfaction score because the guest simply leaves. Read the reason for each handover: if one category hands over every time, either the instruction is too cautious or the fact is missing, and both are fixable this week. Finally, check the handover message itself: did the guest know that a person was coming, and by when?

Which guest questions does the fact sheet not cover, and what do staff override?#

The list of new questions is the most valuable output of the review, and the pattern of staff overrides shows which instructions are wrong before a guest ever notices. Every new question gets one of three decisions: add a fact, add a refusal with a route to a person, or leave that topic to a person permanently.

Keep a do-not-answer list: rate negotiation, complaints, medical questions, legal questions, anything about another guest, and anything the hotel is not allowed to promise. Overrides are the draft replies staff edited before sending and the proposed actions they rejected. A cluster in one category is an instruction problem, not a staff problem, and do not punish overrides; a hotel where staff never override is a hotel where nobody is reading the drafts.

The fact sheet needs an owner, a version and a change log, so any fix can be traced to the week it was made and undone if it causes a new problem. Watch the sheet's size: one long document becomes slow to fix and hard to test, so keep it in sections by topic so a change touches one area.

When should you rerun the original test set?#

Monthly, and after any change to the hotel's data, the assistant's instructions or the model behind it, using the same cases so the results can be compared. The original test set is the only fixed ruler the hotel has; changing the cases between runs destroys the comparison and hides regressions.

A change that fixes one case often breaks another. Adding a fact about parking can make the assistant answer a question it used to route to a person. Record the date, the fact-sheet version and who ran the set next to the result, so a failure can be tied to a specific change. Add cases over time, but keep the original set intact and report it separately from the new ones. If a run fails a case that passed before, treat it as a regression and find the change that caused it before adding anything new.

The monthly run is also the moment to decide whether the assistant's scope should grow, shrink or stop. A stop rule agreed in advance, as described in agreeing a stop rule before an AI pilot starts, is easier to apply than one invented in a bad week.

What to do next#

Put a thirty-minute slot in next week's calendar, name one owner and one deputy, and write the first three lists from last week's conversations. The weekly loop turns transcripts into fact-sheet changes; the monthly rerun, built from the same cases you tested before launch, proves those changes did not break what already worked.