§ Notes / drafting
One mega-tool or five tiny ones? Your agent has a preference.
Same CRUD capability, three different conditions, 1 tool, 2 tools and 5 tools. 1,200 runs on Haiku at temperature 0. The preference is real, subtle but real.
I have been building AI agents for over 18 months: tools, prompts, memory, the entire harness around it. Along the way I hit a long list of design questions, and for most of them I did what everyone else does. I took someone's word for it, knowing it was common knowledge.
Fine tools are better than coarser tools, orchestrator should decide the tool list of worker agent, etc.
But now I decided to actually benchmark these claims.
Nothing wrong with borrowing from something that works. But everything proven in public is a coding agent. Claude Code, Codex, etc. These agents are reasoning-dominated and exploration-shaped, and people run them on flagship frontier models (Opus, GPT-5.5, etc).
The agents most of us actually ship look nothing like that. Finance agents, support agents, etc. They are orchestrator-heavy and pipeline-shaped: direct instructions, a definitive workflow, unlike coding agents. And they do not run on flagship models, mainly because of cost and latency. I wouldn't want my agent to think for 30 seconds before performing a trivial task.
What I want to say is: the harness playbook has been written on one kind of agent (coding), and it gets copied into a very different kind of agent (user facing agents).
I hypothesize that a lot of the decisions we take while building agents are not as trivial as they look. They need proper evals before we call them good or bad.
This and the upcoming blogs are a series of experiments I ran testing my hypothesis and actually finding something out of it. Even if I am wrong, it is still a finding for me.
Project ablate is my attempt to find empirical evidence. Change one design at a time, hold everything constant. One experiment per post.
On this page:
- Experiment A: tool granularity
- The setup
- The metrics I judge my hypothesis on
- Here's what I found
- Split by task type
- The read collapse was a new finding
- Smaller results worth keeping
- Was my hypothesis right?
- What is left
Experiment A: tool granularity
Take one capability (CRUD in this case) and wrap it in three different ways:
| Arm | Tools | Shape |
|---|---|---|
| fine | 5 | add, remove, update, list, search |
| medium | 2 | read, plus one write(op, items) |
| coarse | 1 | a single manage(op, ...) covering all five operations |
An arm is one experimental condition, one way of wrapping the tools.
The capability is a movie watchlist. Basic CRUD operations on a movie watchlist. We judge the state of world before and after tool use; i.e, the final list either matches the expected list or it does not. Read tasks are judged by expecting answer in <answer>...</answer> tags in the final text.
My working principle: everything apart from the design condition is kept invariant. The prompt, the agent loop, the memory, the model, caching, everything.
All three arms (fine, medium and coarse) call the same functions underneath. Every write happens inside one transaction. The only thing that varies here is the schema the model sees.
My hypothesis: changing tool granularity will change the way my agent works.
The setup
Model: Haiku 4.5 at temperature 0.
100 tasks against a seeded watchlist: 50 mutation tasks (add, remove, mixed multi-op), 30 update tasks, 20 read tasks (search, list, ask questions). Two seeds per task. A seed here is just a repeat run. At temperature 0 the repeats come out near identical, so 2 seeds just to filter out any API non-determinism.
Real tasks from the suite, one easy and one adversarial per type, so you know what the agent actually faces:
| task type | easy | adversarial |
|---|---|---|
| mutation | "Add No Country for Old Men and then remove Drive." | "Remove the last film, but actually keep the last one and remove the first instead." |
| update | "The third item, 'Toy Story 2', should be 'Toy Story 2 (1999)'. Leave everything else as-is." | "Standardize these to full titles: 'GOT' → 'Game of Thrones', 'BB' → 'Breaking Bad', 'BCS' → 'Better Call Saul', 'TWD' → 'The Walking Dead'." |
| read | "How many titles are on my watchlist in total?" | "Ugh, I really want to watch Heat tonight - is it actually on my list already?" |
The adversarial ones are traps on purpose: a self-contradicting instruction, update targets that only exist as nicknames, a question tempting the agent to act instead of answer.
Total runs: 100 tasks × 3 conditions × 2 seeds × 2 pollution doses = 1,200 runs.
Pollution mechanism: agents behave differently under load (context pollution).
1st sweep: 600 runs at dose 0, meaning no context bloat, no prompt bloat, only the condition tools.
2nd sweep: 600 runs at dose 10k, meaning conversation history (~9k tokens of an already-resolved movie chat), prompt bloat (behaviour-neutral, ~10k tokens), distractor tools (5 tempting ones like browse_trending, 5 neutral ones like get_weather). The entire clutter comes under the cached prompt prefix. At dose 10k every single call has a ~39k token prefix, and the actual task is the last thing the model sees:
flowchart TB
subgraph prefix["Cached prefix, constant per arm"]
direction TB
A["The arm's real tools"] --> B["10 distractor tools"]
B --> C["Real instructions"]
C --> D["10k tokens of prompt bloat"]
D --> E["9k-token stale conversation"]
end
E --> F["The actual task, sent last"]
The metrics I judge my hypothesis on
- Success rate: M/N tasks passed. A task passes only if the final list exactly matches the expected list, checked by plain code. No LLM judge anywhere.
- Bootstrap CI 95% (read more about it here): the error bars. Resample the 100 tasks with replacement (the same task can get picked twice) a few thousand times, recompute the pass rate each time, take the range the middle 95% lands in. If two arms' ranges overlap heavily, the gap between them is probably noise.
- Sign tests on task pairs: for two arms, throw away every task where both agree (both pass or both fail) and count only the tasks where exactly one arm passed. A 6 vs 0 split is a real difference. A balanced split (8 vs 7) is a coin flip. The p value is the chance of that split happening by luck.
One more term before the numbers: pp means percentage points, the absolute gap between two percentages. 96.5% to 84% is a 12.5pp drop.
Here's what I found
Pass rate @dose-0. Each arm ran all 100 tasks twice (2 seeds), so 200 runs per arm. This table is how many of those 200 passed, with a clean context:
| arm | pass rate @dose-0 |
|---|---|
| fine | 96.5% (193/200) |
| medium | 95.0% (190/200) |
| coarse | 94.5% (189/200) |
A 2 point spread. Of 100 tasks, 94 were solved by every arm, 3 by none, and only 3 were solved by some conditions but not all. A clean null, as we cannot put a verdict with these numbers.
This is our baseline @dose-0.
Under load, pass rate @dose-10k. Same 200 runs per arm, but now every call carries the ~39k token clutter. The change column is the drop from dose 0:
| arm | pass rate @dose-10k | change |
|---|---|---|
| fine | 84.0% | -12.5pp |
| medium | 86.0% | -9.0pp |
| coarse | 83.0% | -11.5pp |
Findings from dose-10k: every arm dropped roughly ~11pp under load. Discriminating tasks jumped from 3 to 16, meaning 16 tasks were now solved by some conditions but not all. And 24 runs finished in a single step, versus 0 at dose 0. The agent answers in one shot or it gives up in one shot.
So granularity matters under load? Still no.
The head-to-head sign tests at dose 10k, across all 100 tasks:
| contrast (all 100 tasks) | tasks won | verdict |
|---|---|---|
| fine vs coarse | 8 vs 7 | coin flip |
| medium vs coarse | 7 vs 3 | not significant |
| fine vs medium | 3 vs 5 | not significant |
Reading the first row: there were 15 tasks where exactly one of the two arms passed. Fine won 8 of them, coarse won 7. That is as tied as it gets.
So the null survived here as well. Tool granularity matters less than we think it does.
This aggregate hides two smaller effects that cancel each other out. Splitting the numbers by task type and both show up.
Split by task type
Same dose-10k runs, now grouped by task kind (50 mutation, 30 update, 20 read). Pass rate per arm:
| task type @dose-10k | fine | medium | coarse |
|---|---|---|---|
| mutation (50) | 94% | 88% | 83% |
| update (30) | 93% | 97% | 95% |
| read (20) | 45% | 65% | 65% |
Mutations first. Fine beats coarse by 11pp, and the sign test is 6 vs 0, p = 0.03. Six tasks where fine passed and coarse failed, zero where coarse passed and fine failed. That is the one statistically clean granularity result in the whole study. When the model has to compose a multi-op state change through heavy clutter, five sharp tools hold up and the single overloaded tool degrades.
%%{init: {"themeVariables": {"xyChart": {"backgroundColor": "transparent", "titleColor": "#1A2018", "xAxisLabelColor": "#3D4736", "xAxisTitleColor": "#3D4736", "xAxisLineColor": "#3D4736", "xAxisTickColor": "#3D4736", "yAxisLabelColor": "#3D4736", "yAxisTitleColor": "#3D4736", "yAxisLineColor": "#3D4736", "yAxisTickColor": "#3D4736", "plotColorPalette": "#7A521E"}}}}%%
xychart-beta
title "Mutation tasks @dose-10k (pass rate, %)"
x-axis [fine, medium, coarse]
y-axis "pass rate" 0 --> 100
bar [94, 88, 83]
Updates: every arm sits between 93% and 97%. Nothing to see.
Reads: fine at 45% looks like a disaster. That number smelled wrong, and it bothered me enough that I went and read every single failing transcript.
The read collapse was a new finding
50 read runs failed at dose 10k. I read all 50 and asked one question: forget the scorer, did the model actually answer correctly?
Mostly, yes.
The biggest finding: correct answer, missing tag. My scoring function says read answers must land inside <answer> tags so plain code can check them. The model wrote "Sorted alphabetically, Dunkirk would come first." Correct, but no tag, scored as a fail. "The last title on your list is Soul." Same story: right answer, no tag, another fail.
A small bucket is my bug: the checker wanted the literal string "no" and rejected <answer>No, Titanic is not on your list.</answer>. 2 runs. That one is on me, not the model.
So I re-scored the same read runs two ways. Strict scorer is the automated check: answer must sit inside <answer> tags. Manual is me reading the transcripts: the correct answer anywhere in the reply counts. Both columns are @dose-10k:
| arm | strict scorer | manual |
|---|---|---|
| fine | 45% | 85% |
| medium | 65% | 95% |
| coarse | 65% | 95% |
Reading tasks barely degraded under load. Medium and coarse sit at 95% either way. What dropped was tag compliance, which my scorer was measuring by accident.
Now forget correctness and measure only one thing: did the reply contain the <answer> tag at all? That rule sits in the system prompt, so this table is a direct read on instruction-following, clean vs polluted:
| arm | dose 0 | dose 10k |
|---|---|---|
| fine | 100% | 60% |
| medium | 100% | 70% |
| coarse | 100% | 75% |
%%{init: {"themeVariables": {"xyChart": {"backgroundColor": "transparent", "titleColor": "#1A2018", "xAxisLabelColor": "#3D4736", "xAxisTitleColor": "#3D4736", "xAxisLineColor": "#3D4736", "xAxisTickColor": "#3D4736", "yAxisLabelColor": "#3D4736", "yAxisTitleColor": "#3D4736", "yAxisLineColor": "#3D4736", "yAxisTickColor": "#3D4736", "plotColorPalette": "#7A521E"}}}}%%
xychart-beta
title "System-prompt format rule obeyed (%)"
x-axis ["fine @0", "medium @0", "coarse @0", "fine @10k", "medium @10k", "coarse @10k"]
y-axis "adherence" 0 --> 100
bar [100, 100, 100, 60, 70, 75]
At dose 0 the model followed the rule 100% of the time. Every arm, every run. Under 39k tokens of pollution it dropped to 60-75%. The model still answers the question. It just stops following the format and replies like it is chatting.
The arm that drifted the most: fine, at 60%. A large tool surface seems to pull the model into a chattier, less instructions obeying mode. So that is the second small granularity effect, and it points the opposite way from the first one. Fine handles mutations better, follows buried instructions worse.
We hang a lot on system-prompt instructions in real harnesses. Output formats, tool protocols, JSON schemas, guardrails. This result says those instructions erode quietly under context load.
Under load, the prompt instructions did not hold up. If a contract matters, enforce it in code, than just relying on prompt.
Smaller results worth keeping
- Fine is the priciest arm in both sweeps. Five tool schemas ride on every single call.
- Caching absorbs pollution's latency. Dose 10k is 15× the input tokens but only ~1.7× the cost, and roughly same latency on model response.
- Coarse was the only non-deterministic arm: on 4 tasks its two seeds disagreed, and the flips cancelled out. Fine and medium reproduced exactly on both seeds.
Was my hypothesis right?
"Changing tool granularity will change the way my agent works."
At the aggregate: no, at both doses. With everything held constant, tool count does not alter the success rate, clean or polluted. A well powered null result.
At the margins, under load: yes, twice, in opposite directions. Fine is better at mutating states (writing tasks) vs coarse (6 vs 0, p = 0.03, paired sign test). Fine is worse at holding an instruction buried in prompt vs coarse (60% vs 75% success rate).
What I am taking away:
- If your agent composes multi step writes inside a heavy context, finer tools buy a small real margin.
- Do not trust a prompt instruction to survive a polluted context. 100% to 60% is the price I measured. Critical instructions belong in code.
- Read your transcripts before believing your own numbers while building your eval system.
What is left
This is one model (Haiku 4.5), one domain (a movie watchlist), and one shape of context load. Open threads:
- A dose curve: 0, 5k, 10k, and watch compliance decay per arm. To see if the curve is linear or not.
- A read-heavy suite. 20 read tasks is thin for an effect this interesting. Will expand it to 50 tasks.
- The one I care most about: the same 1,200 runs on Sonnet 4.6, then on an OpenAI model. If a bigger model flattens the effect and keeps the format instructions inside prompt, the copy-from-coding-agents problem is real, and it is worst for the small models most shipped agents actually run on.
Every run behind this post is committed to the benchmark repo github.com/prnayyay/ablate, databases included, along with the earlier sweeps that got it wrong. Null results are the control group for whatever I claim next.
Next experiment: orchestrator tool authority. What changes when the orchestrating agent decides the toolset for its workers, or let the workers decide their own toolset. This one is going to be interesting.
Related reading: six rules every harness gets right, the model is not the runtime, seen from the opposite direction.