The provider changed the model: how a hotel re-tests an AI assistant before trusting it again

Re-testing means running a fixed set of questions with known correct answers, on frozen data, before and after any model change, and scoring each answer as right, wrong or unsafe. You cannot test the model itself; you test the answers your hotel depends on, then roll back or restrict actions if the threshold you set in advance is crossed.

For example: nothing changed at the hotel, and the assistant that said late checkout was free until 14:00 last month now says 13:00 and a fee. The provider replaced the model behind it, no one at the desk was told, and the first person to notice is a guest at the counter. This article gives you the routine that catches this before a guest does: a small fixed question set, a scoring rule, a rollback threshold agreed in advance, and the questions to put to your provider in writing. You can build the first version in a week.

Why do answers change when the provider changes the model?#

Because the assistant is a wrapper around a model the provider runs, and the provider can replace that model on its own schedule. The hotel buys the assistant; it does not own the engine inside it.

Three things can change at once: the model itself, the instructions the vendor wraps around it, and the tools and data it can reach. Only the third is under the hotel's control. A change is not automatically worse. It may fix a problem you reported. The purpose of re-testing is not to prove the new model is bad; it is to find out whether the answers your hotel depends on still hold.

Drift shows up in ordinary desk situations: a fee quoted from a policy that was retired, a facility described that is closed for the season, a request routed to a person who no longer handles it, a refusal where the assistant used to help. The failure is quiet. Nothing errors, no screen turns red, and the assistant still writes confident sentences. The only reliable detector is a known question with a known answer.

What the hotel can test, and what only the provider knows#

What the hotel can see is the answer text, the action taken, and the data it used. What only the provider sees is which model version is running, what changed in the wrapper, and what the provider tested before shipping it. This is why the hotel's test is an output test: run your questions, compare against what you know to be true. The same logic applies when choosing a model in the first place, which is why we describe how a model is chosen by measured evaluation rather than by reputation.

Treat every model change as unpriced risk rather than as good or bad news, and price it with a run of your own questions.

What should be in a hotel's fixed question set?#

Twenty to forty questions taken from real desk situations, each with the answer that was correct on a stated date, run against frozen data the same way every time. That is the whole instrument.

Draw the questions from the desk, not from a product demo: late checkout and its fee, breakfast hours, cancellation terms for a named rate, pet and child policy, airport transfer, an invoice request, identification at check-in, pool season, a room move, a two-night quote for a named room type. Every question must have one right answer at the time of asking and be answerable from the hotel's own data. If two answers are defensible, it is a discussion topic, not a test question.

Include refusal questions: a medical question, a refund outside policy, a request for another guest's room number, a discount the hotel does not offer. The correct answer is a refusal or a handover to a person. Refusals are answers too, and they break as easily as facts do.

Freeze the inputs: same date range, same rate codes, same policy text, same language. Otherwise you are testing the calendar, not the assistant. Write the expected answer as the fact, not as wording to match: late checkout until 14:00 with a stated fee, not a sentence the assistant has to reproduce word for word. Keep the set small enough that one person can run it in about an hour. A set that takes a day gets postponed, and a postponed set protects nobody. If you are building a set from scratch, start from how to test a hotel AI assistant before staff or guests use it.

How do you score an answer: right, wrong or unsafe?#

Use three verdicts only: right, wrong and unsafe. More categories make two people score the same answer differently, and a score nobody agrees on cannot trigger anything.

Right means the answer matches the expected fact and, where an action was expected, the action taken was the one you expected. Wrong means the fact is off, the answer is vague, or it invents something harmless. This is a quality problem, fixed on a normal schedule.

Unsafe means money, law or safety: it promised a refund, quoted a rate that is not in the system, told a guest identification is not needed, sent a message to a guest that cannot be recalled, or took an action outside the tier the hotel set for it. Unsafe is not a quality problem. It is a stop.

Score blind where you can: two people, or one person who does not know which run an answer came from. Otherwise the scorer starts defending the version they expected to win. Record the exact answer text and the date, not a summary. A run you cannot show later is an opinion, not a record.

Note a fourth observation separately: changed but still right. Wording drift matters for guest-facing tone and for whether staff still trust what they read.

What is your rollback threshold, and who signs off?#

Write the threshold and the names before the run. Any unsafe answer stops that action until it is fixed; a defined number of wrong answers in a run triggers a rollback or a restriction to answers only.

Separate the two thresholds: unsafe is a stop, wrong is a rate. For example, one unsafe answer stops that action, and more than two wrong answers in twenty triggers a restriction or a rollback. These are illustrative numbers; the point is that you choose them in advance and write them down.

Rollback only exists if the provider keeps the previous version available and the hotel can switch back. Ask before you need it, and test the switch once so the first attempt is not during an incident. Define a degradation ladder and which threshold puts you on each rung: full actions, answers only, staff only, off.

Name two people: one operational, such as the front office manager or IT, and one accountable, such as the general manager or revenue manager, for anything touching money, legal duties or guest messages. Names and date go into the run record. Tell the desk what changed and what to do while an action is restricted. An assistant quietly switched off creates its own incidents at the counter.

Keep the decision with the run, so that the next change is judged against a written standard rather than against memory.

What should you ask the provider for in writing?#

Ask for notice before a change that affects your assistant, the version identifier in use before and after, and a rollback path with a stated window. Put the answers in the contract annex, not in an email thread.

Notice means advance warning of any change that affects your assistant, not a release note published afterwards. Identity means the version identifier in use today and the one that will replace it, so a run can be tied to a version rather than to a date. For the change log, you will not get the recipe; you can ask what changed in the model, the instructions, the tools and the data sources.

Ask for evidence: the provider's own evaluation summary, and whether your question categories were in it. Ask for control: a deprecation window, the option to pin a version, and how long the previous version stays available after a change. Ask for a contact: a named person and a response time for a failed run. For a fuller list of pre-purchase questions, see questions to ask an AI vendor before buying for your hotel.

If the provider cannot give notice or pinning, your control is your own fixed schedule plus the ability to switch actions off, and you should say so internally rather than assume protection you do not have.

When else should you re-run the set, and how do you keep it current?#

Re-run on any provider notice, on any hotel price or policy change, and on a fixed calendar date. Then keep the set current by adding questions from real guest conversations and retiring questions whose answers legitimately changed.

The triggers: a provider notice; a hotel price or policy change; a new room type, outlet or language; a new connection to rates or to the property management system; any guest complaint about an answer; a fixed monthly or quarterly date. Policy and price changes are the ones hotels forget. The assistant is answering from data the hotel just edited, while the test set still holds the old expected answer.

Retire a question only when its answer legitimately changed, and record the new expected answer with a date and a reason. Add a question every time a guest asks something the set did not cover. The set should grow from real conversations, not from imagination.

Never edit an expected answer to make a failing run pass. That converts a test into a formality and removes the only warning you had. Version the set and keep the last few runs. The pattern across runs tells you more than any single score. After go-live, fold a light version of this into your routine, alongside a weekly review checklist for a hotel AI assistant.

We apply the same discipline to ourselves: AI Otel built a 30-case benchmark to decide which models can serve each layer, and results will be published when measured. It is built and awaiting measurement, not a published score, and we have no trained model yet. The platform pages describe each component with its status word.

What to do next#

Write the first twenty questions this week, freeze the data they run against, and run the set once now, while nothing has changed. That run is your baseline: it gives the threshold and the two sign-off names something to attach to, and it turns the next provider change from a surprise at the front desk into a dated comparison you can read in an hour. To see how we handle this problem on our own platform, request a briefing.