Whissle conversation-flow suite

Flow-edit sensitivity

Whether an edit made in the flow designer actually reaches a live conversation — and whether a staged draft correctly reaches none. Probed over the text channel, so it says nothing about the speech path.

100.0%8 of 8 tasks

95% CI 67.6–100.0%

TextPreliminaryOur own judge — Whissle deterministic flow analyzer

Ran on

claude-haiku-4-5

the cheapest and fastest model in its family

provider Anthropic

The harness pinned nothing — no model, no provider, no reasoning-depth or speed setting. Those request controls did not exist on the benchmark endpoint when these runs were made, so every arm inherited the production deployment default.

Read from the deployment default in force on each run date, not from the run's own record: the endpoint did not report which model served a turn until after these runs, and no run artifact names one. Automatic failover to the secondary provider (Gemini 2.5 Flash) is logged but not recorded per turn, so we cannot rule out that a small fraction of turns was served by it.

No published comparator

Run 6 August 2026.

Results

Every metric this run produced

Headline first, then the breakdown. Each row carries the sample it was computed over and its interval — or an explicit statement that there isn't one, which on a small run is the more useful fact.

Headline and per-dimension metrics for this run, with sample size and confidence interval for each.
Metric	Value	Sample	Interval
Edits reaching the live conversationHeadline95% Wilson score interval, computed here from the pass count. The harness does not emit an interval; this is derived, not reported.	100.0%	8 of 8 tasks	95% CI 67.6–100.0%

Reproduce

Run this yourself

The harness, the task suites and the raw trajectories are public, and the API these ran against is the same one your agents run on. You do not have to take the number.

You will need: Node 18+ and uv, and a Whissle workspace key (Settings → API keys on whissle.ai). Nothing else — the simulated caller and the flow analyzer both route through Whissle's own model API, so there is no third-party key to obtain.

git clone https://github.com/WhissleAI/tau2-bench-w && cd tau2-bench-w
uv sync
export WHISSLE_BASE=... WHISSLE_API_KEY=...

./run_flow_mutation.sh run --agent-type headache_enrollment

Each mutation is applied through the public flow-designer API, so this exercises the same path a customer's designer uses.

Running a benchmark straight from the Whissle CLI is on the roadmap and does not exist yet — the CLI manages agents, calls and records today, and we are not going to print a command on this page that would fail when you paste it. Until it lands, the harness above is the path.

Comparison

What we measured against

A comparison is only a comparison when the two numbers come from the same experiment, on stated arms. We split the figures accordingly — produced on this harness with only the agent swapped, or published elsewhere — and both sides name their model.

Our arm in every row below

Whissle ran on

claude-haiku-4-5

the cheapest and fastest model in its family

provider Anthropic

Scoring 100.0% over 8 scored tasks. Where a row below names a different model, that difference is listed with the others.

No setup-matched baseline yet

We have not yet run another model through this exact harness — same domain, same task set, same user simulator, same pass^1, same concurrency — so there is no head-to-head to show. Published figures for other systems appear below for context, greyed, with the reasons they are not directly comparable. When a matched run lands it will appear here as a real comparison.

Nobody publishes a comparable figure for this benchmark at this modality, so we show none. An invented comparator would be worse than an empty column.

Exclusions

What was thrown away

The easiest way to improve a benchmark score is to drop the cases that went badly. Here is the arithmetic, so you can check we didn't.

Attempted

Scored

Excluded

None excluded — the sample is the whole attempt.

Comparability

What this number can and cannot be put beside

Derived from the run's own configuration rather than written by hand, so it cannot drift away from the result it describes.

A capability check, not a quality score: it proves published flow edits actually reach the runtime. It says nothing about how well the agent then conducts the call — that is what the headache-intake suite measures.

Comparable with

Other runs of this same benchmark on this page, at the same modality (Text).
Runs of this benchmark on the same arm — claude-haiku-4-5 (the cheapest and fastest model in its family). A run on a different model is a different experiment.
Other Pass^1 figures — one attempt per task, no best-of-n.

Not comparable with

Any external leaderboard — no published figure exists for this suite at this modality, so we show none.
Anything requiring a confidence interval. N=8 is diagnostic, not conclusive.
Independently-judged results. A rule-based auditor that replays the session against the flow's declared spec. It is deterministic rather than an opinion — but we built it and we run it, so it is not an independent verdict. Treat these numbers as our own instrumentation, not as third-party validation.

Sample cases

What passing and failing look like

A pass rate tells you how often. These tell you what happened. Excerpts are recorded artifacts from the run — where we have no publishable transcript we say so rather than reconstructing one.

Passes (8)

Spoken line

Pass

Rewrite what a step says. The edit went through the exact API the flow designer uses — validate, stage as a draft, confirm the draft is inert on the live agent, publish — and was then hunted for in a live conversation.