Benchmarks
Every metric, with its uncertainty
All numbers come from results.json (produced by scripts/analyze.py). Intervals are 95% bootstrap CIs; configurations trained with several seeds use a hierarchical bootstrap that resamples seeds, then tasks. Configurations that have not been evaluated show no results yet.
Workflow success by split
Episode-level success: every validator check passes. Pooled = all four main splits together.
In-distribution eval_id — held-out instances of training templates
A · Baseline (zero-shot)
5.0 [2.0, 8.0]
A+ · Baseline (few-shot prompt)
12.0 [8.0, 16.5]
B · Teacher-distilled LoRA
86.0 [81.0, 90.7]
C · Independent (oracle) LoRA
89.7 [85.3, 93.5]
D · Combined LoRA (mixed)
91.0 [86.7, 94.7]
D' · Combined LoRA (sequential B→C)
90.3 [86.2, 94.0]
Teacher · Qwen3.5-4B (4-bit)
81.0 [75.5, 86.5]
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
C_matched · Oracle on B's tasks
81.0 [75.5, 86.5]
C_25 · Oracle, 250 episodes
87.0 [82.0, 91.5]
C_50 · Oracle, 500 episodes
92.5 [88.5, 96.0]
C_full · Oracle, 3,000 episodes
90.5 [86.0, 94.5]
Unseen templates eval_ood — templates never seen in training (novel phrasing / task graphs / depth 4-5)
A · Baseline (zero-shot)
0.0 [0.0, 0.0]
A+ · Baseline (few-shot prompt)
3.8 [1.7, 6.3]
B · Teacher-distilled LoRA
30.3 [23.9, 37.5]
C · Independent (oracle) LoRA
30.3 [23.9, 36.8]
D · Combined LoRA (mixed)
35.6 [29.2, 42.8]
D' · Combined LoRA (sequential B→C)
32.8 [27.4, 38.6]
Teacher · Qwen3.5-4B (4-bit)
56.3 [50.0, 62.5]
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
C_matched · Oracle on B's tasks
29.2 [23.3, 35.0]
C_25 · Oracle, 250 episodes
33.3 [27.5, 39.6]
C_50 · Oracle, 500 episodes
32.5 [26.7, 38.8]
C_full · Oracle, 3,000 episodes
30.4 [24.6, 36.3]
Unseen tool domains eval_unseen — tool domains never seen in training (inventory, maps)
A · Baseline (zero-shot)
0.0 [0.0, 0.0]
A+ · Baseline (few-shot prompt)
33.3 [25.0, 41.7]
B · Teacher-distilled LoRA
33.3 [17.5, 48.3]
C · Independent (oracle) LoRA
33.1 [24.2, 42.5]
D · Combined LoRA (mixed)
36.4 [21.1, 53.3]
D' · Combined LoRA (sequential B→C)
9.2 [3.3, 16.7]
Teacher · Qwen3.5-4B (4-bit)
75.0 [66.7, 82.5]
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
C_matched · Oracle on B's tasks
43.3 [34.2, 52.5]
C_25 · Oracle, 250 episodes
28.3 [20.8, 36.7]
C_50 · Oracle, 500 episodes
49.2 [40.0, 58.3]
C_full · Oracle, 3,000 episodes
17.5 [10.8, 24.2]
Renamed tools eval_renamed — training templates with every tool/parameter renamed (held-out alias vocabulary)
A · Baseline (zero-shot)
3.8 [1.3, 6.9]
A+ · Baseline (few-shot prompt)
11.9 [6.9, 16.9]
B · Teacher-distilled LoRA
73.3 [66.5, 80.2]
C · Independent (oracle) LoRA
69.2 [61.3, 77.3]
D · Combined LoRA (mixed)
71.7 [64.4, 78.8]
D' · Combined LoRA (sequential B→C)
71.0 [64.4, 77.7]
Teacher · Qwen3.5-4B (4-bit)
80.6 [74.4, 86.9]
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
C_matched · Oracle on B's tasks
65.6 [58.1, 72.5]
C_25 · Oracle, 250 episodes
67.5 [60.6, 74.4]
C_50 · Oracle, 500 episodes
80.0 [73.8, 86.3]
C_full · Oracle, 3,000 episodes
63.1 [55.6, 70.6]
All four splits (pooled)
A · Baseline (zero-shot)
2.2 [1.3, 3.3]
A+ · Baseline (few-shot prompt)
12.8 [10.4, 15.3]
B · Teacher-distilled LoRA
55.8 [51.5, 60.2]
C · Independent (oracle) LoRA
55.9 [51.6, 60.0]
D · Combined LoRA (mixed)
59.1 [53.3, 64.7]
D' · Combined LoRA (sequential B→C)
53.3 [49.8, 57.0]
Teacher · Qwen3.5-4B (4-bit)
71.7 [68.5, 74.9]
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
C_matched · Oracle on B's tasks
54.0 [50.4, 57.6]
C_25 · Oracle, 250 episodes
55.0 [51.4, 58.8]
C_50 · Oracle, 500 episodes
62.5 [58.9, 66.0]
C_full · Oracle, 3,000 episodes
52.2 [48.6, 55.8]
Diagnostics
Where episodes go wrong. Percentages are per-episode rates; counts are means per episode. Each cell shows mean and [95% CI].
| configuration | n | Success % | Tool selection % | Args correct % | Dependency-safe % | Strict schema % | Halluc. tools | Unneeded calls | Invalid args | Parse errors | Turns | Calls |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A · Baseline (zero-shot) | 720 | 2.2 [1.3, 3.3] | 24.3 [21.3, 27.3] | 59.5 [51.4, 67.2] | 94.3 [92.5, 95.8] | 20.1 [17.2, 23.2] | 0.03 [0.00, 0.08] | 0.46 [0.35, 0.59] | 0.11 [0.06, 0.17] | 1.85 [1.74, 1.96] | 2.74 [2.63, 2.86] | 0.61 [0.49, 0.74] |
| A+ · Baseline (few-shot prompt) | 720 | 12.8 [10.4, 15.3] | 55.2 [52.0, 58.4] | 44.2 [40.1, 48.3] | 75.3 [72.1, 78.3] | 64.7 [61.4, 68.3] | 0.31 [0.21, 0.43] | 1.21 [1.04, 1.39] | 0.31 [0.23, 0.40] | 0.60 [0.52, 0.69] | 2.73 [2.61, 2.84] | 1.61 [1.45, 1.78] |
| B · Teacher-distilled LoRA | 720 | 55.8 [51.5, 60.2] | 79.4 [76.8, 82.1] | 78.0 [73.7, 82.5] | 90.8 [87.5, 93.7] | 99.3 [98.7, 99.7] | 0.13 [0.06, 0.23] | 1.05 [0.87, 1.26] | 0.16 [0.09, 0.24] | 0.01 [0.00, 0.02] | 2.94 [2.73, 3.16] | 2.14 [1.92, 2.36] |
| C · Independent (oracle) LoRA | 720 | 55.9 [51.6, 60.0] | 80.9 [78.3, 83.4] | 79.6 [76.6, 82.5] | 91.9 [89.6, 94.1] | 98.6 [97.1, 100.0] | 0.09 [0.05, 0.14] | 0.44 [0.36, 0.55] | 0.05 [0.03, 0.07] | 0.01 [0.00, 0.03] | 2.43 [2.34, 2.52] | 1.56 [1.47, 1.66] |
| D · Combined LoRA (mixed) | 720 | 59.1 [53.3, 64.7] | 81.4 [78.5, 84.3] | 80.9 [75.4, 85.4] | 92.1 [89.4, 94.5] | 99.1 [98.0, 99.8] | 0.06 [0.04, 0.10] | 0.73 [0.57, 0.96] | 0.09 [0.06, 0.14] | 0.00 [0.00, 0.01] | 2.73 [2.55, 2.98] | 1.88 [1.67, 2.16] |
| D' · Combined LoRA (sequential B→C) | 720 | 53.3 [49.8, 57.0] | 82.2 [78.9, 85.6] | 76.9 [72.6, 81.0] | 94.3 [92.5, 96.1] | 99.1 [98.0, 100.0] | 0.09 [0.05, 0.14] | 0.85 [0.73, 0.97] | 0.10 [0.05, 0.18] | 0.00 [0.00, 0.01] | 2.77 [2.68, 2.88] | 1.94 [1.83, 2.08] |
| Teacher · Qwen3.5-4B (4-bit) | 720 | 71.7 [68.5, 74.9] | 85.2 [83.0, 87.4] | 86.3 [83.6, 88.9] | 86.8 [84.3, 89.3] | 81.9 [79.2, 84.7] | 0.00 [0.00, 0.00] | 0.31 [0.26, 0.37] | 0.09 [0.06, 0.12] | 0.23 [0.18, 0.28] | 2.49 [2.42, 2.57] | 1.50 [1.43, 1.58] |
| Oracle (programmatic upper bound) | 720 | 100.0 [100.0, 100.0] | 100.0 [100.0, 100.0] | 100.0 [100.0, 100.0] | 100.0 [100.0, 100.0] | 100.0 [100.0, 100.0] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 2.46 [2.40, 2.53] | 1.68 [1.61, 1.75] |
| C_matched · Oracle on B's tasks | 720 | 54.0 [50.4, 57.6] | 80.4 [78.1, 82.8] | 77.1 [73.9, 80.1] | 89.0 [86.7, 91.3] | 97.8 [96.7, 98.8] | 0.05 [0.03, 0.07] | 0.53 [0.47, 0.59] | 0.15 [0.12, 0.18] | 0.02 [0.01, 0.03] | 2.47 [2.41, 2.53] | 1.61 [1.55, 1.68] |
| C_25 · Oracle, 250 episodes | 720 | 55.0 [51.4, 58.8] | 80.2 [77.7, 82.6] | 80.1 [77.0, 83.1] | 92.6 [90.7, 94.4] | 97.8 [96.7, 98.8] | 0.03 [0.02, 0.04] | 0.48 [0.42, 0.53] | 0.05 [0.04, 0.07] | 0.02 [0.01, 0.03] | 2.43 [2.38, 2.48] | 1.57 [1.52, 1.63] |
| C_50 · Oracle, 500 episodes | 720 | 62.5 [58.9, 66.0] | 83.5 [81.2, 85.7] | 84.2 [81.5, 86.9] | 91.1 [89.0, 93.1] | 96.3 [94.7, 97.5] | 0.04 [0.02, 0.06] | 0.48 [0.42, 0.54] | 0.06 [0.04, 0.08] | 0.04 [0.02, 0.05] | 2.56 [2.50, 2.61] | 1.68 [1.63, 1.74] |
| C_full · Oracle, 3,000 episodes | 720 | 52.2 [48.6, 55.8] | 78.0 [75.4, 80.5] | 75.3 [72.1, 78.5] | 87.9 [85.4, 90.3] | 97.8 [96.7, 98.8] | 0.11 [0.08, 0.15] | 0.56 [0.50, 0.62] | 0.04 [0.03, 0.06] | 0.03 [0.01, 0.04] | 2.45 [2.40, 2.50] | 1.60 [1.55, 1.66] |
- Success %
- All validator checks pass
- Tool selection %
- Set-level F1 between tools called and gold tools
- Args correct %
- Gold calls matched with correct arguments, given the right tool was called
- Dependency-safe %
- No invented identifiers; no dependent call before its prerequisite returned
- Strict schema %
- Every turn is exactly one well-formed JSON action
- Halluc. tools
- Calls to tools not in the catalogue, per episode
- Unneeded calls
- Calls beyond the gold workflow, per episode
- Invalid args
- Calls rejected for invalid arguments, per episode
- Parse errors
- Unparseable turns per episode
- Turns
- Model turns per episode
- Calls
- Tool calls per episode
Paired comparisons
Difference in success rate (percentage points), computed per task on identical tasks (seed-averaged), with a bootstrap 95% CI and a two-sided sign-flip permutation p-value. Positive means the first configuration is better. Dark marks = CI excludes zero. No correction for multiple comparisons is applied.
| comparison | split | Δ success (pp) | 95% CI | −100 · 0 · +100 pp | p | n tasks |
|---|---|---|---|---|---|---|
| A+ vs A | In-distribution | +7.0 | [+1.5, +12.5] | 0.020 | 200 | |
| A+ vs A | Unseen templates | +3.8 | [+1.7, +6.3] | 0.005 | 240 | |
| A+ vs A | Unseen tool domains | +33.3 | [+25.0, +41.7] | <0.001 | 120 | |
| A+ vs A | Renamed tools | +8.1 | [+2.5, +13.8] | 0.016 | 160 | |
| B vs A | In-distribution | +81.0 | [+75.7, +86.0] | <0.001 | 200 | |
| B vs A | Unseen templates | +30.3 | [+25.7, +35.3] | <0.001 | 240 | |
| B vs A | Unseen tool domains | +33.3 | [+26.4, +40.3] | <0.001 | 120 | |
| B vs A | Renamed tools | +69.6 | [+63.1, +76.0] | <0.001 | 160 | |
| C vs A | In-distribution | +84.7 | [+79.7, +89.2] | <0.001 | 200 | |
| C vs A | Unseen templates | +30.3 | [+25.1, +35.4] | <0.001 | 240 | |
| C vs A | Unseen tool domains | +33.1 | [+26.7, +40.0] | <0.001 | 120 | |
| C vs A | Renamed tools | +65.4 | [+59.4, +71.7] | <0.001 | 160 | |
| C_25 vs A | In-distribution | +82.0 | [+76.5, +87.0] | <0.001 | 200 | |
| C_25 vs A | Unseen templates | +33.3 | [+27.5, +39.6] | <0.001 | 240 | |
| C_25 vs A | Unseen tool domains | +28.3 | [+20.8, +36.7] | <0.001 | 120 | |
| C_25 vs A | Renamed tools | +63.7 | [+56.3, +71.3] | <0.001 | 160 | |
| C_50 vs A | In-distribution | +87.5 | [+82.5, +92.0] | <0.001 | 200 | |
| C_50 vs A | Unseen templates | +32.5 | [+26.7, +38.8] | <0.001 | 240 | |
| C_50 vs A | Unseen tool domains | +49.2 | [+40.0, +58.3] | <0.001 | 120 | |
| C_50 vs A | Renamed tools | +76.3 | [+69.4, +83.1] | <0.001 | 160 | |
| C_full vs A | In-distribution | +85.5 | [+80.5, +90.0] | <0.001 | 200 | |
| C_full vs A | Unseen templates | +30.4 | [+24.6, +36.3] | <0.001 | 240 | |
| C_full vs A | Unseen tool domains | +17.5 | [+10.8, +24.2] | <0.001 | 120 | |
| C_full vs A | Renamed tools | +59.4 | [+51.9, +67.5] | <0.001 | 160 | |
| C_matched vs A | In-distribution | +76.0 | [+70.0, +81.5] | <0.001 | 200 | |
| C_matched vs A | Unseen templates | +29.2 | [+23.3, +35.0] | <0.001 | 240 | |
| C_matched vs A | Unseen tool domains | +43.3 | [+34.2, +52.5] | <0.001 | 120 | |
| C_matched vs A | Renamed tools | +61.9 | [+54.4, +69.4] | <0.001 | 160 | |
| D vs A | In-distribution | +86.0 | [+81.2, +90.3] | <0.001 | 200 | |
| D vs A | Unseen templates | +35.6 | [+30.1, +40.7] | <0.001 | 240 | |
| D vs A | Unseen tool domains | +36.4 | [+29.7, +43.3] | <0.001 | 120 | |
| D vs A | Renamed tools | +67.9 | [+61.3, +74.2] | <0.001 | 160 | |
| D' vs A | In-distribution | +85.3 | [+80.8, +89.5] | <0.001 | 200 | |
| D' vs A | Unseen templates | +32.8 | [+27.9, +37.8] | <0.001 | 240 | |
| D' vs A | Unseen tool domains | +9.2 | [+5.3, +13.3] | <0.001 | 120 | |
| D' vs A | Renamed tools | +67.3 | [+60.8, +73.8] | <0.001 | 160 | |
| Oracle vs A | In-distribution | +95.0 | [+92.0, +98.0] | <0.001 | 200 | |
| Oracle vs A | Unseen templates | +100.0 | [+100.0, +100.0] | <0.001 | 240 | |
| Oracle vs A | Unseen tool domains | +100.0 | [+100.0, +100.0] | <0.001 | 120 | |
| Oracle vs A | Renamed tools | +96.3 | [+93.1, +98.8] | <0.001 | 160 | |
| Teacher vs A | In-distribution | +76.0 | [+70.0, +82.0] | <0.001 | 200 | |
| Teacher vs A | Unseen templates | +56.3 | [+50.0, +62.5] | <0.001 | 240 | |
| Teacher vs A | Unseen tool domains | +75.0 | [+66.7, +82.5] | <0.001 | 120 | |
| Teacher vs A | Renamed tools | +76.9 | [+70.0, +83.1] | <0.001 | 160 | |
| B vs A+ | In-distribution | +74.0 | [+68.0, +79.7] | <0.001 | 200 | |
| B vs A+ | Unseen templates | +26.5 | [+21.9, +31.3] | <0.001 | 240 | |
| B vs A+ | Unseen tool domains | −0.0 | [−8.6, +8.9] | 1.000 | 120 | |
| B vs A+ | Renamed tools | +61.5 | [+54.6, +68.5] | <0.001 | 160 | |
| C vs A+ | In-distribution | +77.7 | [+72.0, +83.0] | <0.001 | 200 | |
| C vs A+ | Unseen templates | +26.5 | [+21.4, +31.7] | <0.001 | 240 | |
| C vs A+ | Unseen tool domains | −0.3 | [−8.6, +8.3] | 1.000 | 120 | |
| C vs A+ | Renamed tools | +57.3 | [+50.4, +64.4] | <0.001 | 160 | |
| D vs A+ | In-distribution | +79.0 | [+73.3, +84.3] | <0.001 | 200 | |
| D vs A+ | Unseen templates | +31.8 | [+26.7, +37.1] | <0.001 | 240 | |
| D vs A+ | Unseen tool domains | +3.1 | [−5.3, +11.7] | 0.514 | 120 | |
| D vs A+ | Renamed tools | +59.8 | [+52.7, +66.9] | <0.001 | 160 | |
| D' vs A+ | In-distribution | +78.3 | [+72.8, +83.3] | <0.001 | 200 | |
| D' vs A+ | Unseen templates | +29.0 | [+24.0, +34.0] | <0.001 | 240 | |
| D' vs A+ | Unseen tool domains | −24.2 | [−32.8, −15.6] | <0.001 | 120 | |
| D' vs A+ | Renamed tools | +59.2 | [+52.1, +66.3] | <0.001 | 160 | |
| C vs B | In-distribution | +3.7 | [−1.5, +8.8] | 0.184 | 200 | |
| C vs B | Unseen templates | ±0.0 | [−5.3, +5.1] | 1.000 | 240 | |
| C vs B | Unseen tool domains | −0.3 | [−5.6, +5.0] | 1.000 | 120 | |
| C vs B | Renamed tools | −4.2 | [−9.2, +0.8] | 0.108 | 160 | |
| C_matched vs B | In-distribution | −5.0 | [−9.7, −0.5] | 0.036 | 200 | |
| C_matched vs B | Unseen templates | −1.1 | [−6.7, +4.7] | 0.742 | 240 | |
| C_matched vs B | Unseen tool domains | +10.0 | [+2.2, +17.8] | 0.014 | 120 | |
| C_matched vs B | Renamed tools | −7.7 | [−12.3, −3.5] | <0.001 | 160 | |
| D vs B | In-distribution | +5.0 | [+0.2, +10.0] | 0.051 | 200 | |
| D vs B | Unseen templates | +5.3 | [+1.1, +9.7] | 0.015 | 240 | |
| D vs B | Unseen tool domains | +3.1 | [−2.2, +8.3] | 0.310 | 120 | |
| D vs B | Renamed tools | −1.7 | [−6.5, +2.7] | 0.520 | 160 | |
| C_matched vs C | In-distribution | −8.7 | [−13.2, −4.2] | <0.001 | 200 | |
| C_matched vs C | Unseen templates | −1.1 | [−5.4, +3.1] | 0.648 | 240 | |
| C_matched vs C | Unseen tool domains | +10.3 | [+3.1, +17.5] | 0.009 | 120 | |
| C_matched vs C | Renamed tools | −3.5 | [−8.5, +1.5] | 0.202 | 160 | |
| D vs C | In-distribution | +1.3 | [−0.5, +3.3] | 0.241 | 200 | |
| D vs C | Unseen templates | +5.3 | [+1.1, +9.4] | 0.016 | 240 | |
| D vs C | Unseen tool domains | +3.3 | [−1.7, +8.6] | 0.254 | 120 | |
| D vs C | Renamed tools | +2.5 | [−1.3, +6.3] | 0.240 | 160 | |
| D' vs D | In-distribution | −0.7 | [−3.3, +2.0] | 0.702 | 200 | |
| D' vs D | Unseen templates | −2.8 | [−5.8, +0.1] | 0.084 | 240 | |
| D' vs D | Unseen tool domains | −27.2 | [−33.9, −20.3] | <0.001 | 120 | |
| D' vs D | Renamed tools | −0.6 | [−3.7, +2.5] | 0.796 | 160 |
Breakdowns
Success pooled over the four main splits and averaged over seeds, sliced by task property. No intervals here — small cells are noisy, so read them with their n.
| configuration | approval | args | chain | parallel | permission | recovery | select | termination |
|---|---|---|---|---|---|---|---|---|
| A · Baseline (zero-shot) | 0.0% n=38 | 0.0% n=54 | 0.0% n=189 | 0.0% n=106 | 0.0% n=30 | 0.0% n=105 | 0.6% n=160 | 39.5% n=38 |
| A+ · Baseline (few-shot prompt) | 0.0% n=38 | 13.0% n=54 | 12.7% n=189 | 8.5% n=106 | 3.3% n=30 | 3.8% n=105 | 29.4% n=160 | 0.0% n=38 |
| B · Teacher-distilled LoRA | 0.9% n=38 | 54.9% n=54 | 40.0% n=189 | 64.2% n=106 | 20.0% n=30 | 66.7% n=105 | 80.8% n=160 | 60.5% n=38 |
| C · Independent (oracle) LoRA | 43.9% n=38 | 56.8% n=54 | 35.3% n=189 | 46.5% n=106 | 0.0% n=30 | 87.0% n=105 | 77.9% n=160 | 60.5% n=38 |
| D · Combined LoRA (mixed) | 43.9% n=38 | 60.5% n=54 | 42.5% n=189 | 46.5% n=106 | 23.3% n=30 | 85.1% n=105 | 79.6% n=160 | 60.5% n=38 |
| D' · Combined LoRA (sequential B→C) | 47.4% n=38 | 54.3% n=54 | 34.6% n=189 | 40.9% n=106 | 21.1% n=30 | 80.6% n=105 | 71.0% n=160 | 61.4% n=38 |
| Teacher · Qwen3.5-4B (4-bit) | 36.8% n=38 | 74.1% n=54 | 53.4% n=189 | 88.7% n=106 | 43.3% n=30 | 68.6% n=105 | 95.0% n=160 | 78.9% n=38 |
| Oracle (programmatic upper bound) | 100.0% n=38 | 100.0% n=54 | 100.0% n=189 | 100.0% n=106 | 100.0% n=30 | 100.0% n=105 | 100.0% n=160 | 100.0% n=38 |
| C_matched · Oracle on B's tasks | 13.2% n=38 | 57.4% n=54 | 24.9% n=189 | 61.3% n=106 | 3.3% n=30 | 81.9% n=105 | 81.9% n=160 | 60.5% n=38 |
| C_25 · Oracle, 250 episodes | 47.4% n=38 | 55.6% n=54 | 33.3% n=189 | 50.9% n=106 | 10.0% n=30 | 81.0% n=105 | 75.0% n=160 | 60.5% n=38 |
| C_50 · Oracle, 500 episodes | 50.0% n=38 | 68.5% n=54 | 40.2% n=189 | 64.2% n=106 | 40.0% n=30 | 79.0% n=105 | 82.5% n=160 | 60.5% n=38 |
| C_full · Oracle, 3,000 episodes | 34.2% n=38 | 59.3% n=54 | 24.3% n=189 | 51.9% n=106 | 0.0% n=30 | 82.9% n=105 | 75.0% n=160 | 60.5% n=38 |
Catalogue-size ablation
The same tasks with k irrelevant tools added to the catalogue. A model that selects tools by reading descriptions should be flat; one that pattern-matches will degrade as k grows.
0 distractor tools eval_catalog_0
A · Baseline (zero-shot)
3.1 [0.0, 7.3]
A+ · Baseline (few-shot prompt)
16.7 [9.4, 24.0]
B · Teacher-distilled LoRA
85.4 [78.1, 91.7]
C · Independent (oracle) LoRA
84.4 [77.1, 91.7]
D · Combined LoRA (mixed)
91.7 [85.4, 96.9]
D' · Combined LoRA (sequential B→C)
91.7 [85.4, 96.9]
Teacher · Qwen3.5-4B (4-bit)
no results yet
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
5 distractor tools eval_catalog_5
A · Baseline (zero-shot)
4.2 [1.0, 8.3]
A+ · Baseline (few-shot prompt)
12.5 [6.3, 19.8]
B · Teacher-distilled LoRA
82.3 [75.0, 89.6]
C · Independent (oracle) LoRA
82.3 [74.0, 89.6]
D · Combined LoRA (mixed)
90.6 [84.4, 95.8]
D' · Combined LoRA (sequential B→C)
90.6 [84.4, 95.8]
Teacher · Qwen3.5-4B (4-bit)
no results yet
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
12 distractor tools eval_catalog_12
A · Baseline (zero-shot)
5.2 [1.0, 10.4]
A+ · Baseline (few-shot prompt)
17.7 [10.4, 26.0]
B · Teacher-distilled LoRA
80.2 [71.9, 87.5]
C · Independent (oracle) LoRA
84.4 [77.1, 91.7]
D · Combined LoRA (mixed)
89.6 [83.3, 94.8]
D' · Combined LoRA (sequential B→C)
88.5 [82.3, 94.8]
Teacher · Qwen3.5-4B (4-bit)
no results yet
Oracle (programmatic upper bound)
100.0 [100.0, 100.0]
Efficiency
Per evaluation run, measured on the local machine (Apple silicon, MLX). Latency is end-to-end per episode, including every turn.
| run | configuration | episodes | mean turns | latency / episode (s) | gen tok/s | total tok/s | gen tok / episode | peak mem (GB) | wall time (s) |
|---|---|---|---|---|---|---|---|---|---|
| A_base | A · Baseline (zero-shot) | 1008 | 2.68 | 0.85 | 126 | 1883 | 106.4 | 2.98 | 856.6 |
| A_fewshot | A+ · Baseline (few-shot prompt) | 1008 | 2.71 | 1.12 | 87 | 1904 | 97.5 | 3.90 | 1137.1 |
| B_s0 | B · Teacher-distilled LoRA | 1008 | 2.61 | 1.01 | 93 | 1464 | 94.4 | 3.49 | 1022.8 |
| B_s1 | B · Teacher-distilled LoRA | 720 | 3.14 | 1.23 | 88 | 1505 | 107.4 | 3.83 | 882.6 |
| B_s2 | B · Teacher-distilled LoRA | 720 | 2.93 | 1.13 | 94 | 1525 | 105.6 | 3.43 | 811.2 |
| C_25_s0 | C_25 · Oracle, 250 episodes | 720 | 2.43 | 0.87 | 90 | 1516 | 77.5 | 3.03 | 624.0 |
| C_50_s0 | C_50 · Oracle, 500 episodes | 720 | 2.56 | 0.86 | 97 | 1625 | 83.7 | 3.10 | 620.3 |
| C_full_s0 | C_full · Oracle, 3,000 episodes | 720 | 2.45 | 0.80 | 100 | 1659 | 79.5 | 3.01 | 574.6 |
| C_matched_s0 | C_matched · Oracle on B's tasks | 720 | 2.47 | 0.90 | 91 | 1498 | 81.4 | 3.24 | 648.0 |
| C_s0 | C · Independent (oracle) LoRA | 1008 | 2.39 | 0.86 | 91 | 1524 | 78.6 | 3.16 | 870.2 |
| C_s1 | C · Independent (oracle) LoRA | 720 | 2.36 | 0.83 | 91 | 1525 | 75.7 | 3.03 | 597.1 |
| C_s2 | C · Independent (oracle) LoRA | 720 | 2.51 | 0.91 | 89 | 1503 | 80.8 | 3.33 | 657.9 |
| Dmix_s0 | D · Combined LoRA (mixed) | 1008 | 2.53 | 0.94 | 92 | 1504 | 86.4 | 3.10 | 949.0 |
| Dmix_s1 | D · Combined LoRA (mixed) | 720 | 2.57 | 0.95 | 90 | 1483 | 85.6 | 3.12 | 685.2 |
| Dmix_s2 | D · Combined LoRA (mixed) | 720 | 3.00 | 1.13 | 88 | 1540 | 98.8 | 3.49 | 810.8 |
| Dseq_s0 | D' · Combined LoRA (sequential B→C) | 1008 | 2.64 | 0.97 | 87 | 1539 | 84.0 | 3.14 | 975.4 |
| Dseq_s1 | D' · Combined LoRA (sequential B→C) | 720 | 2.75 | 0.93 | 94 | 1659 | 86.6 | 3.12 | 666.8 |
| Dseq_s2 | D' · Combined LoRA (sequential B→C) | 720 | 2.82 | 1.04 | 87 | 1560 | 90.4 | 3.32 | 749.0 |
| oracle | Oracle (programmatic upper bound) programmatic — no model inference | 1008 | 2.42 | 0.00 | — | — | 0.0 | 0.00 | 0.2 |
| teacher | Teacher · Qwen3.5-4B (4-bit) | 720 | 2.49 | 8.05 | 12 | 180 | 97.4 | 13.17 | 5801.0 |
Training curves
LoRA training loss per step (line) and validation loss at evaluation checkpoints (dots), one panel per run, on a shared y-axis.
B_s0
B- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2281
- final val loss
- 0.4115
- peak mem (GB)
- 4.01
- adapter sha256
- 9ff12841046593b3…
B_s1
B- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2226
- final val loss
- 0.4062
- peak mem (GB)
- 4.01
- adapter sha256
- df5a7ea4024885cb…
B_s2
B- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2266
- final val loss
- 0.4319
- peak mem (GB)
- 4.01
- adapter sha256
- 4e81a988bdca3969…
C_25_s0
C_25- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2312
- final val loss
- 0.0095
- peak mem (GB)
- 3.80
- adapter sha256
- 1fa38da12bef3681…
C_50_s0
C_50- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2252
- final val loss
- 0.0042
- peak mem (GB)
- 3.80
- adapter sha256
- 9f26700b3b26828e…
C_full_s0
C_full- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2088
- final val loss
- 0.0024
- peak mem (GB)
- 3.90
- adapter sha256
- 76551ac500107ba5…
C_matched_s0
C_matched- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2255
- final val loss
- 0.0095
- peak mem (GB)
- 3.90
- adapter sha256
- 87ce5656ffef10ee…
C_s0
C- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2256
- final val loss
- 0.0031
- peak mem (GB)
- 3.90
- adapter sha256
- 7354e6f5c4bc5ea6…
C_s1
C- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2220
- final val loss
- 0.0039
- peak mem (GB)
- 3.90
- adapter sha256
- 029a043ed73f2f86…
C_s2
C- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2273
- final val loss
- 0.0028
- peak mem (GB)
- 3.90
- adapter sha256
- 28539a4006dc709b…
Dmix_s0
D- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2235
- final val loss
- 0.0143
- peak mem (GB)
- 4.01
- adapter sha256
- 3533d0103d9ef3a4…
Dmix_s1
D- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2453
- final val loss
- 0.0120
- peak mem (GB)
- 4.01
- adapter sha256
- 4e6d38c2605a0495…
Dmix_s2
D- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2258
- final val loss
- 0.0137
- peak mem (GB)
- 4.01
- adapter sha256
- 67fc1ebaf6b6d1e7…
Dseq_s0
D'- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2234
- final val loss
- 0.0033
- peak mem (GB)
- 4.01
- adapter sha256
- dac1f13925e0bb76…
Dseq_s1
D'- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2476
- final val loss
- 0.0039
- peak mem (GB)
- 4.01
- adapter sha256
- 9ddb37cae95ec339…
Dseq_s2
D'- steps
- 250
- trainable params
- 5865472
- train time (s)
- 2086
- final val loss
- 0.0037
- peak mem (GB)
- 4.01
- adapter sha256
- 61a179b00eed1088…
training loss (colour = configuration)validation loss