Our first AI scheduling agent got about 95% of requests right. We threw it away.
That sounds like a strange thing to do with something that worked most of the time. This is the story of why we did it, what we built instead, and what it taught us about the people who do this job every day: executive assistants.
The AI that was supposed to do it all
When we started building an AI scheduling agent, we aimed high. Our first agent, Zara, would read a request like "move my 2pm with Sam to Thursday," work out what to do, and do it. We believed the AI could handle the whole job on its own.
So we gave the AI model the steering wheel. It read the request, chose which tools to use, decided when it was done, and sent the emails.
What happened with real requests
Demos are kind to AI. Real requests aren't.
People write "the usual time," "not Thursday, I'm flying," or "loop in Dana but don't make it awkward." When we tested Zara on ways of asking it had never seen before, it got far fewer right than we expected.
We could fix each failure as it came up. But every fix was a patch for one way of saying something, and the next way of saying it broke something else.
Why 95% isn't good enough
Most software can live with 95%. A music app that picks the wrong song 1 time in 20 is fine.
Scheduling for an executive isn't. Think about what a mistake looks like:
Each of these costs trust, and it's the EA's trust that's lost, not the software's.
Now do the math. An EA handling 20 requests a day with a tool that's 95% right would see one mistake every single day.
Then there's the compounding problem. A single request isn't one decision. It's five or six in a row: understand it, find the meeting, check the calendars, write to the guest, book. Even if every step is 95% right, a five-step request comes out right only about 3 times in 4.
An agent that improvises its way through every step multiplies its own uncertainty.
What we learned about the job
The longer we watched EAs work, the more we realized we had misunderstood their job.
Scheduling looks like logistics. But great EAs aren't logistics coordinators. They're relationship managers, gatekeepers, negotiators and trusted advisors.
They know which meeting can move, which person needs a personal touch, when to push back, and when an ordinary-looking calendar request has bigger implications.
That judgment doesn't come from reading a calendar. It comes from knowing the people behind it.
Rebuilding around the EA
So we started again, with a different goal: take the busywork away from the EA and leave the judgment with them.
That became ANCI, the AI scheduling agent that works for the executive assistant. Same desk, same calendar, a very different arrangement.
Take one ordinary request: "Move my 2pm with Sam to Thursday."
Same request, same calendar. The difference is who gets the last word.
That shift, from an AI that acts to an AI that prepares, is the whole design. It rests on one simple idea:
The EA isn't an approval button at the end of the process. They're the reason the process can be trusted. An EA knows:
- which executives can be interrupted, and which can't
- which outside relationships need special handling
- which meetings are politically sensitive
- when a request needs a question, not an action
ANCI's job is to amplify that judgment, not imitate it.
When ANCI isn't sure, it doesn't guess. It asks one clear question and names the possible answers:
EMAIL: ask them what time worksBOOK: send the invite now, without checking with them
An extra message costs a few seconds. Inviting the wrong person costs a relationship.
ANCI isn't perfect either
We should be honest about this. On requests ANCI has never seen before, it still gets about 1 in 16 wrong.
The difference is what "wrong" looks like.
Zara's mistakes went out the door. ANCI's mistakes stop at the EA, as an extra question or a summary the EA corrects before anything is sent. A miss costs a minute, not a relationship.
Measuring honestly
We also changed how we keep score, because it's easy to fool yourself with AI.
- Situations, not sentences. Each test is a real situation with one correct outcome, written many different ways: formal, rushed, full of typos, or written by an EA for their boss.
- A blind test set. We fix problems on one set of phrasings and measure on another we never look at. Otherwise we'd just be tuning to the test.
- Three runs each. AI models don't always give the same answer twice, so a phrasing passes only if it's right all three times.
- Serious errors block a release. Booking without a yes, inviting the wrong person, sending mail nobody approved, or making up availability: one of these stops a release, whatever the overall score.
- A higher bar. Our target for everyday requests and meetings with outside guests is 99.5%. We aren't there yet, and we'll keep sharing where we stand. Asking the right question counts as a pass. Guessing wrong counts as a failure.
What I'd tell anyone building AI agents
If your AI works where one mistake costs trust, such as money, health, legal work or someone's calendar, don't just ask how often it's right. Ask what happens when it's wrong, and who catches it.
Here's what we learned the hard way.
For the engineering side of this, see AI Agents Should Be Mostly Deterministic. Here's Why.
What I'd tell every EA evaluating an AI tool
Before you trust any AI with your executive's calendar, ask the vendor these questions:
If the answers are vague, the risk falls on you, because when something goes wrong, the guest won't blame the software. They'll remember who sent it.
For us, the answer was clear. Let AI do the reading, let tested software do the work, and leave the decisions with the person who knows the people.
We weren't building software to replace that judgment. We were building software that needed it.
EAs: what's the one mistake an assistant can never make? Tell me on LinkedIn, or ask the Academy.