Pranay Varma All learnings

§ Notes / drafting

Who picks your worker's tools? I let the orchestrator decide, and it cost 10 points.

Two agents (orchestrator and worker), one shared world. Three levels of supervisor authority over the worker's toolset. 720 runs on Haiku at temperature 0.

Aug 5, 2026 17 min read multi-agent · evals

At the end of the last post I teased this experiment: what changes when the orchestrating agent decides the toolset for its workers, versus letting the worker pick its own tools?

This is not an academic question. It is the default pattern. Anthropic's own engineering post on their multi-agent research system says each subagent needs "guidance on the tools and sources to use". Cognition's Don't Build Multi-Agents argues the opposite: share full context, don't fragment decisions. Both posts are influential. Neither ships with controlled evidence.

So I built the controlled version. Project ablate, experiment 2.

On this page:

The setup

Multi-agent system - 1 orchestrator agent, 1 worker agent, one run, one shared world.

The orchestrator receives the user's task. It holds exactly one tool, delegate. No data access, no domain tools. The worker gets spawned by that delegate tool call, runs its own agent loop against the same world state the run is scored on, and holds whatever domain tools the arm allows.

The 3 conditions (arms):

  1. Autonomous - Worker receives every tool in the pool for it to decide (here the orchestrator only delegates and forgets).
  2. Pre-filter - Worker receives a pre-filtered list of tools from the orchestrator to achieve the goal. (The orchestrator delegates the task with a set of tools for the worker to work with).
  3. Dictate - Worker receives a completely planned order of tools to execute in order to achieve the goal. (The orchestrator delegates the task along with an ordered plan text - which tool to execute first, etc).

The three conditions differ only in the delegate tool's schema and one paragraph of the orchestrator prompt. Nothing else.

Note: In order to carry out the experiment, everything apart from these 3 arms is byte-identical, from prompt, to agent loop, tool definitions, etc.

arm delegate schema worker receives
autonomous {task} every tool in the pool
prefilter {task, allowed_tools[]} exactly the named tools
dictate {task, tool_plan[]} exactly the planned tools, plus the ordered plan text

An arm is one experimental condition, one level of supervisor authority. Gating is programmatic, not prose: a pruned (filtered out) tool is absent from the worker's LLM request. It is not "discouraged". It does not exist as far as the model can see.

Three properties make the comparison mean anything:

  1. The orchestrator's task prose is discarded; the worker always receives the original instruction verbatim. Otherwise the orchestrator could smuggle tool advice while delegating. Note that the orchestrator's task is not deleted but recorded, just never sent to the worker agent.
  2. The worker never learns an orchestrator exists. Its system prompt is byte-identical in all three arms.
  3. All three orchestrators see the same tool catalog to delegate the worker, but the orchestrator cannot use those tools, it can only delegate. I vary authority, not knowledge.

The domain is a personal assistant: 19 tools over email, calendar, contacts, todos, and notes. Information retrieval is deliberately two step (search tool returns ids and titles, never bodies), so getting content is always a second explicit call.

Check out the complete task suite breakdown here: tasks.data.json

How do I score: Scoring is a whole-world diff checked by plain code, no LLM judge. Read tasks answer through a submit_answer tool that every arm always holds. I check the world before and after the task, to score if a run is a pass or a fail.

Two terms before the numbers.

A sign test takes only the tasks where two arms disagree (1 pass, 1 fail) and asks whether the win split could be a coin flip.

A fatal prune is the orchestrator omitting a tool the task required, so the worker never gets the required tool.

Dose - I usually run the sweep on 2 doses, meaning every task is run on dose 0 (clean agent context, only the system prompt, and tool catalog) and dose 10k (system prompt - instructions + bloat, conversation history, distractor tools).

Here's what I found

Pass rates on a clean context (dose 0), 120 tasks per arm:

arm pass rate @dose-0 tools granted to worker (avg) fatal prunes
autonomous 75.8% (91/120) 19.0 0
prefilter 70.8% (85/120) 2.7 26 (21.7%)
dictate 65.0% (78/120) 2.2 36 (30.0%) 
%%{init: {"themeVariables": {"xyChart": {"backgroundColor": "transparent", "titleColor": "#1A2018", "xAxisLabelColor": "#3D4736", "xAxisTitleColor": "#3D4736", "xAxisLineColor": "#3D4736", "xAxisTickColor": "#3D4736", "yAxisLabelColor": "#3D4736", "yAxisTitleColor": "#3D4736", "yAxisLineColor": "#3D4736", "yAxisTickColor": "#3D4736", "plotColorPalette": "#7A521E"}}}}%%
xychart-beta
    title "Pass rate @dose-0 (%)"
    x-axis [autonomous, prefilter, dictate]
    y-axis "pass rate" 0 --> 100
    bar [75.8, 70.8, 65.0]

The three arms genuinely differ.

The head-to-head sign tests:

Contrast Wins p-value
Autonomous vs Dictate 16 vs 3 0.0044
Prefilter vs Dictate 9 vs 2 0.065
Autonomous vs Prefilter 9 vs 3 0.146

What is 16v3 here: Out of all the 120 tasks, 16 tasks are won by autonomous where dictate failed, and 3 tasks are won by dictate where autonomous failed. Rest 101 tasks were either pass or fail for both these arms (both failed the same task, or both passed the same task).

The entire gap is one forgotten tool

This is the part worth the price of the experiment.

Split the 120 tasks by whether the orchestrator omitted a required tool:

subset n autonomous prefilter dictate
prune-free 84 91.7% 90.5% 90.5%
≥1 prune 36 38.9% 25.0% 5.6%

Here for ≥1 prunes - autonomous passed on 38.9% of tasks where pruning happened for pre-filter or dictate arms, even after getting the complete tool set. Meaning the tasks were genuinely difficult.

On the 84 tasks where no tool was forgotten, the arms are statistically identical. The entire 10-point gap lives in the 36 tasks where the orchestrator guessed the toolset wrong.

Test one, discordant prunes. Take tasks where dictate pruned but prefilter did not, holding the task fixed. Prefilter passes 80% of them, dictate passes 10%. Seven wins to zero, p = 0.0156. Same task, same worker, same everything; the only difference is whether the required tool was in the grant.

Test two, difficulty control. Restrict to tasks the never-pruned autonomous arm solved, so the task is provably doable. Dictate fails 76.5% of those when it pruned versus 2.7% when it did not (p = 1.2e-10). Prefilter: 70.0% vs 2.5% (p = 5.0e-07).

And there is an ordering inside the mistake itself: prefilter's prune set is a strict subset of dictate's. Ten tasks where only dictate pruned, zero where only prefilter did (p = 0.0020). Writing down an ordered plan forces more commitments than naming an allowed subset, and every extra commitment is another chance to be blind.

For example:

One run shows the whole thing.

arm what happened verdict
autonomous delete_todo(t1), answers "2" PASS
prefilter tries delete_event on a todo id, fails, answers "2" anyway FAIL
dictate never attempts a delete (plan only mentioned to delete event and answer), as nothing was deleted, it answers "3" FAIL

The before world

The after world

The orchestrator assumed "offsite prep" was a calendar event, even though it was a to-do.

For more details, look at this: the full three-arm trace for this run. Every tool call, the prose each orchestrator wrote and never sent, and what each arm finally told the user.

Where the signal lives

Splitting by stratum tells me exactly where the risk is:

stratum @dose-0 autonomous prefilter dictate
obvious 100% 100% 100%
decoy 90.0% 90.0% 86.7%
hidden 70.0% 66.7% 53.3%
state-dependent 43.3% 26.7% 20.0%

When the request names its own tools, authority is completely free: 100 across the board, zero tasks that separate the arms. All the signal sits where the required tool is only inferable from state the router cannot see. The single significant stratum cell is state-dependent, autonomous vs dictate, 7 wins to 0, p = 0.0156.

In real world multi-agent systems - there will be more state dependent tasks, which tells us, autonomous performs better than dictate.

Then I polluted everything

Same 360 runs (120 tasks × 3 arms) again, dose 10k: ten distractor tools added to the pool (29 total), ~13.8k tokens of behaviour-neutral junk in the worker's system prompt, and a stale resolved conversation. We'd assume autonomous will perform the worst given it'll get more distractor tools to choose from.

Pass-rate:

arm @dose-0 @dose-10k change
autonomous 75.8% 65.8% -10.0pp
prefilter 70.8% 65.0% -5.8pp
dictate 65.0% 60.0% -5.0pp 
%%{init: {"themeVariables": {"xyChart": {"backgroundColor": "transparent", "titleColor": "#1A2018", "xAxisLabelColor": "#3D4736", "xAxisTitleColor": "#3D4736", "xAxisLineColor": "#3D4736", "xAxisTickColor": "#3D4736", "yAxisLabelColor": "#3D4736", "yAxisTitleColor": "#3D4736", "yAxisLineColor": "#3D4736", "yAxisTickColor": "#3D4736", "plotColorPalette": "#7A521E, #3D4736"}}}}%%
xychart-beta
    title "Performance by Arm and Dose"
    x-axis ["autonomous", "prefilter", "dictate"]
    y-axis "Performance (%)" 0 --> 100
    bar "@dose-0" [75.8, 70.8, 65.0]
    line "@dose-10k" [65.8, 65.0, 60.0]

■ dose-0   ▬ dose-10k 

%%{init: {"themeVariables": {"xyChart": {"backgroundColor": "transparent", "titleColor": "#1A2018", "xAxisLabelColor": "#3D4736", "xAxisTitleColor": "#3D4736", "xAxisTickColor": "#3D4736", "xAxisLineColor": "#3D4736", "yAxisLabelColor": "#3D4736", "yAxisTitleColor": "#3D4736", "yAxisLineColor": "#3D4736", "yAxisTickColor": "#3D4736", "plotColorPalette": "#7A521E"}}}}%%
xychart-beta
    title "Performance Change (Percentage Points)"
    x-axis ["autonomous", "prefilter", "dictate"]
    y-axis "Change (pp)" -12 --> 0
    bar "Δ" [-10.0, -5.8, -5.0]

Every pairwise sign test between arms at dose 10k: null.

I have to retract my own headline

My first read of that table was: pollution hurts the autonomous arm most, only its 10-point drop is significant within-arm, so autonomy is the fragile choice. I wrote that down as the headline.

It is wrong, and my own analysis killed it.

Look at the task subset comparison:

subset n autonomous prefilter dictate
mutation (world-diff) 79 68.4 → 59.5 (-8.9) 64.6 → 55.7 (-8.9) 63.3 → 55.7 (-7.6)
read (answer channel) 41 90.2 → 78.0 (-12.2) 82.9 → 82.9 (0.0) 68.3 → 68.3 (0.0)

Autonomous does not fail selectively at answering. It loses 8.9 points on state-change tasks, identical to prefilter and slightly more than dictate.

  1. Dictate has the worst answer adherence.

Answer-gate failures at dose 10k (runs where the world state was correct but the answer was missing or incorrect):

arm answer-gate failures world-gate failures
autonomous 9 32
prefilter 7 35
dictate 13 35

The specific adherence failure described, where the worker never called submit_answer:

arm dose 0 dose 10k change
autonomous 0 3 +3
prefilter 1 4 +3
dictate 1 7 +6

Dictate's silent failures increased twice as fast as autonomous's. The arm with the tightest instructions is the one that most often stops answering under load.

Dictate's additional silent failures also landed on read tasks it was already failing. It started at 28/41, compared with autonomous's 37/41, so six extra silent failures did not reduce its overall score. Autonomous simply had more successful read tasks exposed to a failure mode that affected every arm.

What you can say

Context pollution primarily attacks the answer channel rather than state mutation. Silent read failures, where the worker searches, gives up, and never submits an answer, increased under load in every arm, most sharply in dictate (1 → 7).

Pollution's real mechanism: easy surrender

What does pollution do? One thing, significantly, and identically in every arm.

The worker gives up and asks a question. There is no user to ask; the worker is alone by construction.

And the surrender has a precise, monitorable shape. Look at this:

after a search returns no match dose 0 dose 10k p
retried with a different query 67.8% 57.2% 0.030
the miss was the run's last domain call 12.5% 25.1% 0.0012

The agent loses the ability to retry and asks a question as an end-turn, to a non-existent user. (The worker was designed to work alone).

Take a look at this example:

Same tools, same prompt, same result string. The only difference is 13.8k tokens of junk, and the only behavioural difference is dropping one word and searching again.

For more details, look at this: the full six-run trace, both doses side by side. The two details the polluted worker asks the user for are the two fields it had already searched, and both would have matched on the retry it never made.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "primaryColor": "#E9E7D5", "primaryTextColor": "#1A2018", "primaryBorderColor": "#3D4736", "lineColor": "#3D4736", "edgeLabelBackground": "#E9E7D5", "tertiaryColor": "#E9E7D5"}}}%%
flowchart TD
    A["search_inbox('BrightPath newsletter')"]
    B["No inbox messages match"]
    C{"what happens next?"}
    D["drop the qualifier<br/>search_inbox('BrightPath')"]
    E["1 match: e3 from BrightPath"]
    F["archive_email(e3)"]
    G["PASS • 3 of 3"]
    H["stop and ask the user"]
    I["FAIL • 3 of 3<br/>inbox untouched"]
    A --> B
    B --> C
    C -->|"clean context<br/>3 of 3 arms"| D
    D --> E
    E --> F
    F --> G
    C -->|"13.8k tokens of junk<br/>3 of 3 arms"| H
    H --> I
    classDef pass fill:#3D4736,color:#FDFCF5,stroke:#3D4736
    classDef fail fill:#7A521E,color:#FDFCF5,stroke:#7A521E
    class G pass
    class H,I fail

One more thing pollution does, and it fooled me briefly: it reduces mistakes too. Decoy calls drop 40%, mutating calls drop 19%. That is not robustness. The model just does less of everything. Lethargy, not skill.

The ten trap tools

The distractor tool layer was the one piece of pollution the arms could filter. If orchestrator authority earns credit, it is here.

Total distractor calls across dose-10k's 360 runs: three. All on the same task, one per arm.

The cost argument flips sign

Clean, pruning looked like free money:

arm $/run @dose-0
autonomous $0.0323
prefilter $0.0185
dictate $0.0192

2.3× cheaper. Fewer tool schemas on every call. This is the standard argument for curation and I believed it.

Polluted, within the same sweep:

arm $/run @dose-10k runs paying cache creation
autonomous $0.0230 0 / 120
prefilter $0.0382 (+66%) 58 / 120
dictate $0.0424 (+84%) 64 / 120

The mechanism is in the caching docs: tools render at the very front of the prompt, and modifying tool definitions invalidates the entire cache. Autonomous sends a byte-identical 29-tool array on every task, so the whole 14k polluted prefix caches once and is read back forever. A per-task tool subset changes the front of the prefix and pays the cache-write premium on every task.

Per-task tool curation saves money only when there is no prefix worth caching, which is exactly when the savings are smallest.

What the 720-run audit overturned

After both dose sweeps froze, I re-read every run.

My audit found interesting things.

One thing I can say for sure is: while building agent eval systems, judge the LLM's final texts and thoughts as well. Even though the state of the world matters, it is interesting to understand the LLM's thought process, its ability to give false-positives, and how to overcome it.

What I am taking away:

  1. Majority failure is premature commitment and lack of context from the orchestrator. Had the orchestrator been given prior context, it could've performed better in the 2 (pre-filter and dictate) arms. Held prune-free, the arms are identical. The tax is entirely the supervisor committing to tools before anyone has read the data.
  2. An ordered plan is strictly worse than an allowed list. pre-filter >> dictate. Its prunes are a strict superset (p = 0.0020), and the plan prose is a second, ungated channel: a hallucinated tool name survives in the plan text even after the gate drops it, and the worker hunts a tool that does not exist.
  3. Context pollution is an equalizer, and its mechanism is motivational. It does not make the model pick wrong tools. It makes the model stop after one failed search and ask a user who is not there, identically in every arm, and no amount of tool authority touches that. Query length up plus retry rate down is a production-monitorable signature.
  4. Curation's cost argument depends entirely on the cache. 2.3× cheaper with nothing worth caching, 1.8× more expensive with a big cached prefix. "Fewer tools per call is cheaper" is a claim about a pricing regime, not about architecture.

Real world scenario (Very important)

In a real-world multi-agent system, things are way more complex than in this experiment. Context retrieval, tool granularity, forced-worker delegation vs orchestration having an option to delegate, cost vs latency arguments. A lot of users' requests will affect a state (database), meaning a lot of tasks are state-dependent. A lot of moving parts. The real world scenario is harsher than even dose 10k.

What I'd take from this experiment:

Since a lot of tasks are state-dependent, and we have seen the dictate arm fail miserably in state-dependent tasks, either provide enough context to the orchestrator agent if you are planning to go the pre-filter/dictate route, or force the worker agent to gather complete context itself using the allowed tools.

Every run behind this post is committed to github.com/prnayyay/ablate, databases included, along with both aborted sweeps and the audit tooling that overturned my own claims. The retractions stay in.

Next experiment: Deep diving into consumer facing agents. Forced worker delegation v/s letting the orchestrator decide. What I faced while building the assistant.

Related reading: One mega-tool or five tiny ones?, the previous experiment in this series.