Accuracy, false alarms and missed cases: reading an AI tool's test results

Accuracy is the share of all decisions a tool got right. Precision is the share of its alarms that were real; recall is the share of real cases it caught. A single accuracy figure hides both, so read the false alarms, the missed cases, the baseline and the sample size before you trust a score.

A vendor's slide says the assistant is 95 percent accurate. That figure is invented for this article; the problem is not. One percentage cannot tell a hotel whether the tool will miss the burst pipe or wake the technician for nothing. After reading this, you will be able to take any test score apart, ask for the four numbers that matter, and decide which error your hotel can afford.

What does a 95 percent accuracy score actually measure?#

Accuracy is the share of all decisions in a test set that the tool got right. It says nothing about which decisions it got wrong, and that is where the cost sits.

Run one example through the whole article. An assistant reads incoming guest messages and decides whether each one is an urgent maintenance issue to escalate to the duty technician, or a routine message for the normal queue. The test set is, for example, 200 past messages, each labelled urgent or routine by a person. Forty of them are genuinely urgent.

The tool flags 50 messages as urgent. Thirty-eight of those are right; 12 are false alarms. It misses 2 of the 40 real ones. The arithmetic, written out: 38 caught plus 148 correctly left alone is 186 right out of 200, so accuracy is 93 percent.

Why does the number flatter? Because 160 of the 200 messages are routine, a tool that flags nothing at all scores 80 percent without doing any work. Accuracy rewards the majority case. The invented 95 percent on the slide could be recall, or accuracy on an easier task, or accuracy on a different test set. The first question is always: which of these is it? The 95 percent is a sales line invented for this article; the arithmetic under it is the part that is real.

Accuracy 186/200 = 93 percent. Precision 38/50 = 76 percent. Recall 38/40 = 95 percent.

What is a false alarm and what is a missed case?#

A false alarm is the tool acting on something that did not need action; a missed case is the tool staying quiet when it should have acted. Both are errors, and they arrive with different bills.

A false positive, in plain words: the tool said yes and the truth was no. In the example, a request for an extra pillow is escalated as urgent and a technician walks to the room for nothing. A false negative, in plain words: the tool said no and the truth was yes. The leaking ceiling stays in the general queue until the guest calls the front desk.

The two errors are not symmetric. One spends staff minutes; the other spends the guest's stay and possibly the room. A tool can be tuned to trade one for the other: raise its sensitivity and false alarms rise; lower it and missed cases rise. No setting removes both, so the hotel has to choose.

In hotel work the truth is often a person's judgement, so the labels themselves carry error. Ask who labelled the test set and whether your own front desk would agree with those labels. Ask the vendor which of the two errors their headline number describes, and on which cases. If you are running the test yourself, the same questions apply; see how to test a hotel AI assistant before staff or guests use it.

Precision and recall in plain words#

Precision is the share of the tool's alarms that were real. Recall is the share of real cases the tool caught.

Precision is true positives divided by everything the tool flagged. In the example, 38 out of 50, so 76 percent: three alarms in four were real, and one in four wasted a trip. Recall is true positives divided by everything that was genuinely urgent. In the example, 38 out of 40, so 95 percent: it caught 95 of every 100 real problems.

The memory hook: precision is about the tool's output, recall is about the world. Two useless tools make the point. One that flags every message has perfect recall and terrible precision. One that flags nothing has zero recall and can be dressed up as cautious.

A combined score that merges the two hides the same choice one level down. For hotel work, keep precision and recall apart and read them next to each other. Always ask for the raw counts behind the percentages, because percentages on small numbers move a long way with one case. This is precision and recall explained with one example; the same reading applies to any score you are shown.

Why is a baseline more useful than a score?#

A baseline is what the hotel does today on the same cases, and a score only means something next to it.

Baselines a hotel can actually build: a person reading every message; a keyword rule that sends anything containing leak, broken or not working to maintenance; or the current queue with no assistant at all. If the keyword rule already catches 36 of the 40 urgent messages, the assistant's 38 is an improvement of two cases, not a leap from zero.

The baseline must be measured on the same test set, with the same labels and the same scoring rule, or the comparison is meaningless. A baseline also protects against the reverse case: a vendor score that is worse than the process the hotel already runs. Ask who ran the comparison, because a number produced by the vendor about the vendor is a starting point, not a result. Write the baseline down before the demo, so the demo cannot quietly change the question.

For the wider conversation before a purchase, see questions to ask an AI vendor before buying for your hotel.

What do benchmark and sample size mean in AI testing?#

A benchmark is a fixed set of cases with known answers, used to compare tools or models on the same ground. Sample size is how many cases it holds, and of what kind.

A benchmark fixes three things: the cases, the labels and the scoring rule. Without a fixed set, two scores cannot be compared, because they were measured on different ground. Sample size is about composition as much as total. Two hundred cases with 40 urgent ones say something about urgent maintenance; two hundred cases with three urgent ones say almost nothing, because one case moves recall by a third.

Composition matters more than the total. The test set must contain the rare, expensive cases in realistic numbers, not just the easy majority. Contamination is the next question: ask whether the cases were used while the tool was being built. A tool tested on cases it has already seen will look better than it is. Traps, meaning deliberately hard cases such as a message in another language, a joke that sounds like a complaint, or a complaint that is also a repair request, show whether a score survives real guests.

AI Otel keeps the same discipline on itself. We built a 30-case benchmark to decide which models can serve each layer: 30 cases, 5 traps, not yet run. Results will be published when measured, and we have no trained model yet. Our approach to choosing models is described in AI model evaluation for hotel work, and the hotel operating system the assistant belongs to is described at https://ai-otel.com/platform/pms.

Which error costs your hotel more?#

It depends on the task, not on the tool: for each use, decide whether a false alarm or a missed case is the more expensive mistake, then ask for the number that describes it.

For urgent maintenance escalation, a missed case costs a damaged room and an unhappy guest, while a false alarm costs a technician's walk; tune toward recall and accept more false alarms. For offers and upsell messages, an upgrade offer sent to a guest who has just complained costs goodwill, while a missed offer costs a little revenue; tune toward precision. For reading invoices and delivery notes into the system, a wrong amount posted to the ledger is worse than a document sent for a person to read, so uncertain documents should go to a person by design.

Alerts on hotel reports sit in between. Too many false alarms and staff stop reading them, which turns every later alert into a missed case; too few and the odd number passes unnoticed. Anything with a legal or identity dimension, such as the guest identity notification or a fiscal document, should not be treated as an accuracy problem at all: a person reviews before anything leaves the building.

Write the rule down before you see the score. A sentence such as "for this task a missed case is ten times worse than a false alarm" is a decision the hotel makes, not the vendor. After go-live the same numbers come back weekly, from real cases rather than a test set, so keep the review routine short and repeatable; what is anomaly detection and how it catches the odd number in hotel reports covers the alert side of that routine.

What to do next#

Before the next demo, write one line per task saying which error costs you more. Then ask for precision and recall with their counts, the baseline on the same cases, and the size and composition of the test set. Those four answers turn a sales percentage into a decision you can defend, and the same four questions keep working after go-live, when the numbers come from real guests instead of a test set. If you want to see how we apply this to our own model choices, request a briefing at https://ai-otel.com/contact.