Agent frameworks commonly place a large language model in the control loop: the model chooses tools, sequences them, and decides when a task is complete. We report on operating such a design for meeting scheduling, a domain where most errors are visible to people outside the organization and cannot be undone. From our engineering record we derive a taxonomy of eight recurring failure modes, and observe that every mitigation we shipped took a decision away from the model and added latency and cost. We then describe a replacement design in which the model's only role is to read a request into a fixed, validated form, and all planning, time arithmetic and execution are ordinary code behind a single human approval gate. Finally we describe an evaluation protocol built to resist self-deception (situations rather than sentences, independently written phrasings, a held-back half and a blind final set, three runs per phrasing, and release-blocking serious errors) and report current results without adjustment: 93.8% on everyday requests and 94.0% on requests involving outside guests on the blind final set, against our own launch target of 99.5%, with zero serious errors in either suite.
Contributions
1. Introduction
A scheduling request looks small. "Move my 2pm with Sam to Thursday" is, in practice, a chain of five or six decisions: which meeting, which Sam, which Thursday, whose calendars to read, which time to propose, and who must be told. The result is not text for a person to read and discard. It is an event on somebody else's calendar or an email in somebody else's inbox, and it cannot be taken back.
Errors also compound across that chain. If each of five decisions is right 95% of the time, the whole request is right about 77% of the time. For an assistant handling twenty requests a day, a 95%-accurate tool produces roughly one visible mistake every day.
This paper compares two designs we built and operated for the same product. System A placed the model in the control loop. System B confines the model to reading. Section 2 describes System A, and Section 3 catalogues how it failed. Section 4 describes System B. Sections 5 to 7 describe how we measure System B and what the measurements say. Section 8 lists the limits of this work.
2. System A: the model in the control loop
System A ran a Claude model in a tool-use loop. A meeting request exposed more than twenty tools, including intent classification, attendee lookup, availability search, host clarification, plan construction, plan review and booking. The model chose which tool to call and in what order, read each result, and decided when the request was finished. The loop ran under a cap on turns per activation and a budget cap of $1 per request.
On short, clean requests System A performed well, often finishing in a few turns. The failures described next appeared on real phrasings, multi-person requests, replies arriving mid-flow, and time-zone boundaries.
3. Failure modes of the model-driven loop
The taxonomy below is drawn from our engineering record: bug reports, root-cause analyses and an end-to-end test campaign. The examples in F5 to F8 come from a campaign that ran while the system was on a smaller fallback model (GPT-4.1-mini), so we use them to illustrate mechanisms, never as accuracy figures.
| Failure mode | Observed example | How System B removes it |
|---|---|---|
| F1. Skipped mandatory step | Asked to meet "the Product team and financ", the model sometimes skipped the required name lookup and asked the host "did you mean the Finance team?" A hard rule in the prompt reduced this but did not stop it; a code guard that refused the question until the lookup had run did. | Lookups are code and always run. There is no step for the model to skip. |
| F2. Silent termination | Asked "who is in financ", the model received a vague lookup result and ended its turn. No reply was ever sent. The fix was a code fallback that detected a finished loop with no reply. | Every path ends in a reply that code composes. |
| F3. Reference drift | The host was emailed a numbered list of times sorted by closeness to the requested time, while the list used to interpret the reply was in date order. A reply of "Option 1" was left to the model to interpret over two lists that disagreed, and could resolve to a time the host never saw. | One typed list is shown and stored, and code maps the reply to it. |
| F4. Frame errors in model-supplied values | The model passed a time with a UTC offset where the availability data was in the host's local wall-clock time; separately, the fetch window was derived from fields a phrase like "next Monday" never filled. Free slots were reported as conflicts and then offered back as alternatives. | The model quotes the words; date and clock arithmetic is code, in one frame. |
| F5. Split-brain dates | The model fixed "next Friday" as one date in its message to the host while the extracted field, correctly, held another; code reconciled the search to the field but never told the model, so the email's text and its slot list named different days. | All host-facing text is rendered from one validated form. |
| F6. Unreachable tool | The prompt told the model to present a slot picker while the picker had been filtered out of the tools offered that turn. The model substituted the nearest available tool and asked the host an open question with no times attached. | The model selects no tools. |
| F7. Reported actions not taken | Messages told the host a meeting was "scheduled" or "tentatively scheduled" while no slot had been selected and nothing had been booked. | Confirmation text is produced only after code verifies the calendar write. |
| F8. Instruction echo | A host's standing preference ("book first available between 09:00 and 12:00") was injected into an extraction prompt as mandatory; the extractor returned 09:00 as the requested time, which a later guard flagged as already past, so the host was asked about a time they never gave. | Every extracted value must point at a span of the request's own words. |
The shape of the fixes. Each mode was mitigated, and the mitigations had a common form: a gate that hid every tool but one until a review step ran; a gate that hid the booking tool until required fields were filled; a handler that refused a question; a fallback for a reply that never came. Every mitigation was code taking a decision back from the model. The forced steps also added one or two model calls per request, so latency and cost rose with each fix. After enough of these, the model was no longer meaningfully in charge of the loop; the loop was a set of guards around it.
4. System B: a read-only model
System B starts from one question: which part of scheduling actually needs a language model? Our answer is reading, turning free text such as "can u set something up w Dan from Acme next week" into a structured request. Everything after that is a function of the request and the calendars, and is written as ordinary, tested code.
Everything above the approval gate is reversible: reading a calendar changes nothing, and an unsent draft can be discarded.
Read. A single model call fills a fixed form: one of a short list of request types (schedule, reschedule, cancel, find a time, block time and others), plus the people, the meeting and the time. The model has no tools that act; it cannot book, send or cancel. Values must point back at the words in the message they came from.
Check. Code validates the form against the message. When the message plainly contains time-shaped text such as "Friday 2pm" but the form came back without a time, or names a person but the form has no attendee, a check returns the exact words and asks the model to look again. The check never writes a value. If the second read still misses, the system asks the person. When the model points at an existing meeting from a menu, code accepts the pick only if it shares real words, names or dates with the message.
Plan. Code turns the form into steps. Relations such as before, after and between, time zones and working hours are computed, not generated. For a message containing several requests, the model may write a short plan in a constrained language, at most three steps, which a checker rejects if any step refers to nothing, to itself, or in a circle. For a single request, the model never writes a plan.
Approve and execute. One component is the only route to an irreversible action. The assistant sees one card naming the meeting, the time and who will be told, and nothing leaves until they approve. Code then executes and confirms the write. This is the commit-point pattern described in our companion paper.
5. Evaluation protocol
Language-model systems are easy to over-measure. A team fixes the failures it sees, and the test set quietly becomes a training set. Our protocol is designed against that.
Situations, not sentences. The unit of evaluation is a situation: a plain-language case with one fixed expected outcome, for example "the host asks to meet an outsider whose calendar we cannot see; expect the host's free times, an offer to email the outsider, and nothing sent without a yes." Many phrasings hang off each situation, and one expectation grades them all. Phrasings vary in register: formal, terse, misspelled, voice-transcribed, written by an assistant on an executive's behalf, written by non-native speakers, and two requests in one message.
Three splits. Phrasings are divided into a dev half, which engineers may read and fix against; a held-back half, which is scored but never used to choose a fix; and a blind final set, which is scored and whose misses are shown only as counts. For the two launch-critical suites, new phrasings were written by two independent writers who saw only the plain-language request and the test world: no code, no expected output, and not the other half. Any held-back phrasing 85% or more similar to a dev phrasing was rewritten, so a fix cannot carry over through a near-copy.
All runs must pass. The reading model is not deterministic, even at temperature 0, so each phrasing is run three times and passes only if all three runs pass.
What counts as a pass. The right action with the right people and time, or the right question. Asking is a pass when the request is genuinely ambiguous, and we separately track how often the system asks when it need not, against a ceiling of 15%.
Serious errors. Four outcomes block a release regardless of overall accuracy: booking without the assistant's approval; inviting the wrong person; sending mail the assistant did not approve; and offering a time as free that was not read from a real calendar.
Controlled world. Suites run against fixed calendars and a frozen clock, so a result can be reproduced. The answering model is recorded for every message, and a silent fallback to a different model is reported, not averaged in.
6. Results
The figures below come from the full run on November 13, 2026, with Claude Haiku 4.5 as the reading model (21,209 model calls; no fallback). Percentages are of phrasings, each passing all three runs; intervals are 95% Wilson intervals.
| Suite | Blind final | Held-back | Serious errors | Asked when it need not |
|---|---|---|---|---|
| Everyday book, find, move, cancel with known people | 93.8% (136/145) 88.6–96.7 | 94.3% (365/387) 91.5–96.2 | 0 | 1% |
| Outside guests outsiders, their time zones, outreach | 94.0% (109/116) 88.1–97.0 | 90.5% (324/358) 87.0–93.1 | 0 | 8% |
| Multi-part several requests in one message | – | 84.8% (39/46) 71.8–92.4 | 0 | 3% |
| Guest replies reading an outsider's answer | – | 82.1% (32/39) 67.3–91.0 | 0 | 21% |
| Executives and assistants acting for another person | – | 100% (11/11) 74.1–100 | 0 | 13% |
Reading the table. Neither launch-critical suite meets our 99.5% target, and the intervals are wide because the blind sets are small. The held-back and blind figures for outside guests differ by 3.5 points, inside their intervals. The guest-replies suite exceeds the 15% ceiling for needless questions. The multi-part, guest-replies and executives suites are small, and we report them for completeness rather than as estimates.
Serious errors. The everyday and outside-guest suites recorded no serious errors over 3,018 graded runs (1,596 and 1,422). By the rule of three, that bounds the per-run serious-error rate in these situations below about 0.2% at 95% confidence. It does not bound behaviour outside the situations we test. For transparency: a further suite built from a public email corpus, still in development and to be reported separately, recorded 3 serious errors in its held-back half. Those runs are not included above, and the release rule applies to them in full.
What is excluded. Suites for features not yet built are excluded, as is a regression board whose phrasings pin known readings and therefore does not measure understanding.
7. Where System B still misses
We reviewed every distinct held-back miss in the two launch-critical suites. They fall into five groups.
| Group | Typical phrasing | What the system did |
|---|---|---|
| Asked when it could have acted (largest group) | "Is Vidya available for lunch on Friday or one day next week?"; "advise me of her availability in the next few days" | Asked which day was meant instead of offering times across the stated range. |
| A detail wrong on the approval card | "rescheduled for Thursday, July 16th from 1-1:30 pm"; "next Thursday (Jul. 23)" | Proposed 1:30 instead of 1:00, or the wrong Thursday, on a card awaiting approval. |
| Missed the referenced meeting | "move acme intro to mon pls" | The reader did not point at the meeting; in most cases the software still found it, but we grade the reading itself and count these as misses. |
| Ambiguous guest reply | "The Tuesday one works for me." | Asked the host which offered time the guest meant, instead of booking. |
| Forwarded notice, no action | A pasted calendar cancellation notice | Answered or did nothing instead of offering the cancellation for approval. |
The common property matters more than the categories. Every miss ended either as a question to the assistant or as a card the assistant had not approved. None changed a calendar or sent mail. That is the design goal of System B: it does not make the reader perfect, it decides where the reader's mistakes land.
8. Discussion and limitations
What System B gives up. Range: it handles the request types we have built, and says so or asks outside them, where System A would attempt anything. Questions: when a reading is uncertain it asks, and some of those questions are ones a good human assistant would not need to ask. Velocity: a new capability is a form field, code and a suite of phrasings, not a new tool and a paragraph of prompt. Ingenuity: the old loop occasionally found a creative path through an unusual request. For scheduling we consider these the right trades, because a wrong booking costs a relationship and an extra question costs seconds. In domains where errors are cheap and reversible, a model-driven loop may well be the better choice.
Limitations. The comparison is between two systems built by one team for one product, not a controlled experiment. System A's failures are drawn from an engineering record rather than a matched benchmark, so we make no accuracy comparison between the two systems. System B is evaluated on fixed calendars with a frozen clock, not live traffic. Phrasings come from real requests and two writers, and cannot cover every way people write. The blind final sets are small, so their intervals are wide. Results are for one reading model at one point in time. The grader is strict in both directions: it fails a correct outcome when the reading was wrong, and passes a well-chosen question. Finally, this work is reported by the company that builds the system, and has not been independently replicated.
9. Conclusion
When the model ran our agent's loop, the dominant failures were not invented facts but plausible early decisions that every later step trusted, and each fix moved a decision from the model into code. Confining the model to reading, and putting a person at the single point where actions become irreversible, has produced a system whose misses so far stay inside the organization. It is not yet as accurate as we require. We will keep publishing the numbers the same way: on phrasings we did not tune against, with the misses shown.
Cite this paper
@techreport{lal2026measuring,
title = {Measuring a Scheduling Agent Honestly: Failure Modes of
Model-Driven Agent Loops and a Blind-Set Evaluation
of a Read-Only-Model Design},
author = {Lal, Rajesh},
institution = {ANCI AI},
year = {2026},
month = {November},
url = {https://anci.app/research/measuring-scheduling-agents-2026}
}