How to test a hotel AI assistant before staff or guests use it

A test is a fixed set of your own desk and guest questions, each with the expected answer and a pass rule written down before the assistant runs. Include traps where the right answer is to refuse or hand over. Score every answer, get a named person to sign off, and retest after any change.

The demo runs on a clean fictional property and answers every question smoothly. Then someone at your own desk asks about a refund on an agency booking, and the assistant produces a cancellation policy that does not exist in any document you own. Knowing how to test an AI assistant before it reaches your staff or guests is what stands between those two moments. After reading this, you will be able to build a test set from your own questions, run it, score it and decide on it.

Why is a vendor demo not a test?#

A demo shows what the assistant does with the vendor's chosen questions and the vendor's data. A test shows what it does with your questions, against answers you wrote down before you looked.

In a demo, the inputs are selected. The vendor picks the questions, the property is often fictional, and the records behind it are clean and complete, so the assistant is never asked about a field nobody filled in. A demo also has no pass rule. Nothing is written down that says what a wrong answer would look like, so a smooth, confident reply reads as a correct one.

Fluency is the specific danger. The US National Institute of Standards and Technology defines confabulation in its Generative Artificial Intelligence Profile (NIST AI 600-1, July 2024) (opens in a new tab) as the production of confidently stated but erroneous or false content, by which users may be misled, and lists over-reliance on AI among the risks of these systems. Regulators in Türkiye and elsewhere make the same point in practice: generated answers can look plausible and convincing while being wrong, and can read coherently even when they are incorrect. Checking accuracy matters, with regular monitoring and human oversight where needed. Confirm your own compliance obligations with a legal adviser.

A demo almost never shows the refusal: what the assistant does when the correct answer is no, I do not know, or ask a colleague. That is exactly the behavior a hotel needs most. A demo is still useful for one thing: deciding whether the assistant is worth an afternoon of your own testing, and collecting candidate questions for the test set. Before that afternoon, read up on the questions to ask an AI vendor before buying for your hotel.

How do you build a test set from your own desk questions and guest messages?#

Collect the questions staff and guests actually ask, strip everything that identifies a person or a booking, and keep them in one dated file with a named owner.

Draw from places where real questions already exist: front office shift notes, guest messages by email and messaging apps, the complaints log, reservation change requests, night audit queries, and requests for the spa, the restaurant or a transfer. Anonymize before anything else. Remove names, identity and passport numbers, phone, email, booking and room numbers, and any date combination that points to one stay. Replace them with placeholders that keep the shape of the question, for example a guest asking to move a booking of [N] nights to [a later date].

Treat the test set as personal data until it is not. Data protection authorities recommend, as good practice, that people define and test what data the system may access and what decisions it may take during development. Confirm your own handling with a legal adviser before you paste anything into a tool.

Keep the questions messy. Real messages are short, misspelled, mixed between languages and missing context. Tidying them up makes the test easier than the desk. Tag each item with its source and its language, so a failure can be traced back to the channel it came from. Size the set to the jobs the assistant will do, not to a round number: every task it will answer or act on needs at least one item, and the whole set should be small enough that two people can score it by hand in one sitting.

Which traps belong in the test set?#

Every test set needs items where the correct behavior is to refuse, to say it does not know, or to hand the guest to a named person, because those are the answers a demo never shows.

Money traps are first: a refund, a discount or a waived fee outside the written policy. The correct behavior is to state the policy and offer to pass the request to a person, never to agree to an amount. Rule-bending traps come next: an early check-in when the room is not ready, an extra guest beyond the room's capacity, a smoking room in a non-smoking property. The correct behavior is a clear no with the reason. Missing-data traps test the gap: the answer exists in principle but the field is empty, for example a rate that was never loaded. The correct behavior is to say the information is not available, not to fill the gap with a plausible figure. Boundary traps cover a question about another guest's booking, a request for someone else's bill, a question about a different hotel, or an instruction to ignore the rules it was given. The correct behavior is refusal and handover.

Plausibility matters. A trap nobody would ever ask teaches nothing; build traps from requests your desk has genuinely had to refuse. NIST's Generative Artificial Intelligence Profile (July 2024) lists human-AI arrangements that lead to automation bias or over-reliance among generative AI risks. Traps are how you find out whether your own team will catch a confident wrong answer before the guest does. Each trap needs its expected behavior written as an action, not as a script: hand over to reservations, do not state a figure, do not confirm the booking. Exact wording can vary; the behavior cannot.

For the wider picture on where these safeguards fit, see AI in hotels: uses, limits and safeguards.

How do you write the expected answer and the pass rule before you run the test?#

For every item, write what a good answer must contain and what makes it a fail, in the same file, before the assistant sees the question.

The expected answer has three parts: the facts that must appear (the policy, the correct date, the rule that applies), the action that must be taken (hand over to reservations, take no action), and what must not appear (a promised refund, an invented figure, another guest's detail). The pass rule must be a sentence a second person can apply without asking you. For example: pass if it states the cancellation window from the policy document and offers to transfer the request to reservations; fail if it states any refund amount.

Write from the source of truth, meaning the policy document, the rate sheet or the standard operating procedure, not from an earlier answer the assistant gave. Freeze the version. Date the file and record what the assistant will run against: the model version the vendor names, the instructions it was given, and the data snapshot it reads.

Write first because of how reading works. Once a fluent answer is in front of you, you start grading it against what it just said. Writing the rule beforehand is what protects you from that. Have a second person read the pass rules before the run. If they cannot apply a rule without asking you what you meant, the rule is not finished.

How do you score each answer?#

Put every answer into one of four outcomes, record the reason next to the label, and keep the raw answer so the score can be checked later.

Score the action as well as the words. If the assistant can change something in the system, the test must check what it did, not only what it said. Score the route where the assistant can show one. An answer that cites the policy document and an answer that produced a similar-sounding sentence are not the same result, even when the text matches.

Use two scorers on the trap items. Where they disagree, the finding is usually about your pass rule, not about the assistant, and the rule needs rewriting. Do not stop at one average figure. A high average hides a single wrong answer on the item that matters, such as a refund, a guest's personal data or a legal duty, and that is the item the launch decision should turn on. Keep the answers, the scores and the date together in one file. A score without the answer cannot be re-examined when the assistant changes.

Where the assistant suggests prices or rate changes, the same four labels apply, but the check is whether the reason given matches the rule that fired, not whether the number looks reasonable. The detail is in AI rate recommendations: what a revenue manager should check.

Who signs off, and what blocks going live?#

A named person inside the hotel signs off, not the vendor, and a fixed list of failures blocks launch whatever the average score says.

Name the roles before the run: front office manager for desk questions, guest relations for guest messages, an IT or data owner for access and logging, and one manager who owns the policy answers. One of them is accountable for the decision. The vendor's own results are an input to your test, never the sign-off.

The blockers, regardless of the average: any wrong answer involving money, a guest's personal data, or a legal duty such as the guest identity notification (KBS, the misafir kimlik bildirimi) or the invoice; any action taken without the confirmation it required; any answer that invents a policy; any trap that produced a confident wrong answer twice. The Personal Data Protection Authority's agentic AI document (February 2026) treats oversight as something to define in advance: under which conditions and at which stages people come in, extra human review for potentially high-risk decisions at deployment, and monitoring of the system's behavior and interactions once it is in use.

Write the launch scope down in one page: which tasks the assistant may carry out, which it may only answer, and which it must hand over. A narrow scope is easier to test and easier to withdraw. Record the decision, the date and the test file it rests on, so the next person can see why the assistant was allowed in front of a guest.

When do you retest?#

Retest the full set after any change to the data it reads, the instructions it follows or the model behind it, and run the trap items on a fixed schedule in between.

Three triggers. A data change: a new rate plan, a changed policy, a field renamed or emptied. An instruction change: a new rule in the prompt or the knowledge document. A model change: the vendor updates or swaps the model, and you may hear about it late or not at all; ask in writing how you will be told. If there is no clear answer, schedule a periodic trap run instead of waiting to be told.

Keep the old answers. A regression is an item that used to pass and now fails, and you can only see it if the earlier result is stored beside the new one. Run the traps first. They are the cheapest signal to produce and the ones that matter most to a guest. Version the test set: when you add an item, date it. Never quietly edit an old item until it passes; that turns the test into a demo.

Watch the system after launch, not only before it. The Personal Data Protection Authority's generative AI guide (24 November 2025) stresses regular monitoring of outputs supported by human oversight where needed, and the practical form is a weekly review of a sample of real conversations against the same pass rules. Choosing a model by measurement is the vendor's work, not the hotel's, but the hotel can ask for the evidence behind that choice and for the results of the last run. For one example of that approach, see how a model can be chosen by measured evaluation.

What to do next#

Start this week, small. Take one shift's worth of desk questions and guest messages, strip the identifiers, write the expected answer and the pass rule for each item, and run the set once before anyone else sees the assistant. The parts fit together in one order: questions from your own desk, traps for refusal and handover, pass rules written before the run, four-way scoring with the raw answers kept, a named sign-off with fixed blockers, and the same file rerun after every change to data, instructions or model.