Specialised LoRA training for tool selection and multi-step orchestration in a 0.5B model

Technical report — Cognitive Core Lab, November 2026. All numbers in tables are generated from run artifacts by scripts/analyze.py and scripts/build_report.py.

Summary

Result. LoRA training of a 0.5B model on 1,000 verified workflow episodes (about 38 minutes on a 16 GB Apple M4) raised workflow success on held-out tasks from 2.2% (untrained) and 12.8% (few-shot prompted) to 53–59% pooled over the four held-out splits, and to 86–91% on new instances of training templates. Every trained configuration beats both baselines on in-distribution, unseen-template and renamed-catalogue tasks (Holm-adjusted p ≤ 0.021).

Where it stops. Gains shrink sharply outside the training distribution. On 16 task templates never seen in training, success is 30–36% (the 4B teacher: 56%). On tool domains absent from training, trained models are no better than the few-shot prompt (≈33% for both), and one template in that split turned out to be flawed (§7). Success on tasks whose dependency graph was seen in training (72–74%) is roughly double that on novel graphs (26–38%): much of what is learned is pattern-specific.

Signal source. Teacher demonstrations (B) and oracle trajectories (C) give statistically indistinguishable aggregate results (55.8% vs 55.9%), with different strengths that track what each data source contains: B almost never handles human approval (0.9%; the teacher produced one accepted approval episode), C is stronger on failure recovery (87.0% vs 66.7%). Mixing both (Dmix, 59.1%) is numerically best, but its advantages do not survive multiple-comparison correction. Training on the two sources sequentially (Dseq) collapses on unseen tool domains (9.2% vs 36.4% for Dmix; −27.2 pts, Holm-adjusted p = 0.021).

Students vs teacher. Distilled students exceed the teacher on in-distribution tasks (86.0% vs 81.0%), because only verified teacher episodes were kept. They use about a quarter of the teacher's peak memory (3.0–3.9 GB vs 13.2 GB) and have 7–10× lower mean latency per episode (measured at different batch sizes, §7). They do not inherit the teacher's generalisation.

Failures. The first two training configurations diverged (§4.1) and are preserved as negative results. All numbers below come from results/ and runs/, generated by scripts/analyze.py.

1. Question and hypotheses

Can a 0.5B-parameter open-weight model be made into a usable tool orchestrator through cheap LoRA training, and does it matter whether the training signal comes from a stronger model or from an independent, programmatic source?

• H1 (training helps). Every LoRA configuration (B, C, D) beats the unmodified baseline (A) on workflow success on held-out tasks.
• H2 (beyond format). The gain is not only output-format compliance: trained models also beat a few-shot-prompted baseline (A⁺), and they improve on diagnostics that format alone cannot fix (dependency safety, argument correctness, recovery, permission handling).
• H3 (generalisation vs memorisation). Gains carry over, with some loss, to unseen task templates (eval_ood), unseen tool domains (eval_unseen) and renamed catalogues (eval_renamed). A model that only memorised its training graphs would collapse on these splits.
• H4 (signal source). With the same budget, teacher demonstrations (B) and oracle trajectories (C) produce different strengths. The comparison is open-ended: B inherits the teacher's blind spots and the coverage bias of rejection sampling, while C is exhaustive but stylistically uniform.
• H5 (combination). Combining sources (D) is at least as good as the better single source, and the combination strategy (mixed vs sequential) matters little.

2. Setup

2.1 Models

• Student / foundation checkpoint: Qwen/Qwen2.5-0.5B-Instruct @ 7ae557604adf (Apache-2.0). Every configuration starts from these exact weights. The selection rationale and measured alternatives are in MODEL_SELECTION.md.
• Teacher: Qwen/Qwen3.5-4B as the MLX 4-bit conversion mlx-community/Qwen3.5-4B-MLX-4bit @ 32f3e8ecf654 (Apache-2.0), run locally in non-thinking mode. No hosted API was available.
• Oracle: a programmatic policy (ccl/oracle.py) that reads the task's gold dependency graph and reacts to real observations. It is the source of config-C data and the validator's ceiling. It is not a model.

2.2 Environment: tools, runtime, action schema

21 typed tools in 12 domains (contacts, calendar, email, weather, FX, orders, files, tickets, maths, flights; held-out: inventory, maps), each simulated deterministically over a seeded world (ccl/tools.py). Four tools need human approval (send_email, issue_refund, delete_file, book_flight), and two are role-gated (issue_refund → support_agent, delete_file → admin). Faults are injected deterministically: transient retryable timeouts, permanent service_unavailable with a declared fallback tool, and not-found entities.

Each turn the model emits exactly one JSON object:

{"calls": [{"tool": "<name>", "args": {...}}, ...]}                 // >1 call = parallel, independent
{"final": "<answer>", "status": "done|refused|blocked|failed"}

The runtime (ccl/runtime.py) parses the action leniently: it accepts code fences, <think> blocks and a bare call object, but records strict-schema compliance separately. It validates arguments against the typed schema, enforces role permissions itself (a forbidden call is never executed and is scored as a violation), asks an approver before consequential calls, and executes calls with deterministic limits: 8 turns, 6 calls per turn, 16 calls total, 2 parse errors, a 2 s tool timeout. Observations are appended as a user message. The same loop drives teacher generation, oracle generation, evaluation, the CLI (scripts/agent.py, with a real terminal approval prompt) and the live server (scripts/serve.py).

2.3 Tasks

44 scenario templates in 8 families: tool selection, args construction (relative dates, 12h→24h times, currency names and symbols to ISO codes, urgency to priority), sequential chains up to depth 5, parallel fan-out/fan-in, recovery (retry, fallback, not-found), permission boundaries, human approval (including denial), and termination (no-tool requests, stop when done). Every task has a gold DAG with per-argument matching rules, an expected final status, required answer facts and expected side effects. Catalogues include 2–6 random distractor tools. 25% of training tasks have tools and parameters renamed from a training alias vocabulary, so models must read the catalogue rather than memorise names.

Validation is executable (ccl/evaluate.py). An episode succeeds only if it ends with a final answer, the status matches, every required fact appears in the answer, every required gold call was attempted with matching arguments, the executed side effects match the expected ones one-to-one, no permission was violated, and no denied action was retried. No model judge is used anywhere.

2.4 Splits and leakage control

• train_pool (3,000) and val_pool (150): the 24 training templates only, never the held-out tool domains.
• eval_id (200): new instances of training templates.
• eval_ood (240): 16 templates never used in training. These include new phrasings, new graphs (fan-in, chain-into-conversion, book-the-cheapest), depth 4–5 chains (training goes to depth 3) and new permission and approval situations.
• eval_unseen (120): 4 templates over tool domains absent from training (inventory, maps).
• eval_renamed (160): training templates with every tool and parameter renamed from a held-out alias vocabulary.
• eval_catalog_{0,5,12} (96 each): identical in-distribution tasks with fixed distractor counts.

Controls: (i) templates belong to exactly one split; (ii) any eval task whose request text occurs verbatim in the train or validation pool is dropped; (iii) a value-free gold-graph signature is computed per task, and its overlap with training is reported. Eval tasks are tagged graph_seen or graph_novel and results are stratified by that tag. (iv) Held-out tools never appear in training, not even as distractors. (v) Eval and train use different RNG seed ranges, and the name and city pools for training are a strict subset of those for evaluation. The audit is in data/v1/leakage_audit.json:

<!-- BEGIN:data -->

<!-- END:data -->

scripts/build_datasets.py reproduces every task file byte-for-byte; this was verified by rebuilding and comparing all 11 SHA-256 hashes.

3. Training data

3.1 B — teacher demonstrations (distillation through data)

The teacher ran the same runtime, with the same system prompt and catalogue rendering, on the first 1,600 tasks of the training pool. Pass 1 decoded greedily. Pass 2 drew one T=0.7 sample for every task rejected in pass 1 (rejection sampling). An episode was accepted only if it passed the executable validator and had no parse errors, hallucinated tools, invalid-argument errors, unnecessary calls or dependency-unsafe calls. Every episode, accepted or not, is kept with its rejection reasons in data/v1/teacher/pass{1,2}.jsonl.gz. No synthetic label was ever substituted for a missing teacher demonstration. Tasks the teacher could not solve are simply absent from B.

<!-- BEGIN:teacher -->

Teacher: mlx-community/Qwen3.5-4B-MLX-4bit (upstream Qwen/Qwen3.5-4B, apache-2.0). Tasks offered: 1600; accepted episodes: 1011 (63.2% of tasks). Accepted episodes by family: approval 1, args 122, chain 146, parallel 124, permission 47, recovery 132, select 308, termination 131. Wall-clock: pass 1 3.70 h, pass 2 1.96 h. API cost: $0 (local). <!-- END:teacher -->

3.2 C — independently generated, execution-verified trajectories

The oracle (§2.1) ran on the same task pool. Its trajectories pass through the same runtime and the same acceptance filter (3,000/3,000 accepted). C contains no model outputs.

3.3 Normalisation and budget matching

Assistant turns from both sources are re-serialised to canonical JSON, both as targets and in the history, so B and C differ only in their decisions (which tools, which arguments, sequential vs parallel, answer wording, status), not in surface format. Each single-source variant has exactly 1,000 episodes: B is a seeded random sample of accepted teacher episodes, and C covers 1,000 random tasks from the same 1,600 offered to the teacher. Every configuration trains for 250 optimizer steps × 8 episodes with identical hyperparameters. Dmix (B ∪ C, 2,000 episodes) therefore sees each episode about once; B and C see theirs about twice. Dseq runs B for 125 steps, then C for 125 steps, on one cosine schedule.

Ablations (single seed): C_matched, the oracle on exactly B's 1,000 tasks, isolates label source from task coverage; C_25 / C_50 / C_full train on 250, 500 and 3,000 oracle episodes at the same step budget.

4. Training

Final configuration (used for every reported run): LoRA rank 16, scale 20, dropout 0, applied to every linear layer of the last 16 of 24 blocks (5,865,472 trainable parameters, ~1.2% of the 494M model). AdamW, lr 1e-5, 20 warmup steps, cosine decay to 1e-6, global gradient-norm clipping at 1.0, batch 4 × 2 gradient-accumulation, max length 2,048, gradient checkpointing. The loss covers assistant tokens only. Each episode is packed into one sequence with every assistant turn supervised; this gives the same loss as one example per turn under teacher forcing, but processes the shared prefix once. Batch order is a deterministic function of the seed. Checkpoints hold adapter, optimizer state and step, and are written atomically; a test (tests/test_train.py) checks that an interrupted run, once resumed, reproduces the uninterrupted adapter. B, C, Dmix and Dseq use seeds 0, 1 and 2; the ablations (C_matched, C_25, C_50, C_full) use seed 0 only (see §7). All 16 reported runs pass the stability rule below (logs/stability.jsonl).

4.1 Failed training configurations (preserved)

The final configuration is the third one tried. The first two diverged; both are kept with their loss curves and evaluations in archive/, and none of their numbers is used in any comparison.

Diagnosis (experiments/grad_diag/init_grads*.log): at initialisation, before any update, the gradient is dominated by v_proj.lora_b in the earliest adapted layers (layer 0 when all layers are adapted) and varies ~20× between batches. With LoRA scale 20, small updates to these layers move the model sharply; restricting LoRA to the last 16 blocks (MLX-LM's default) removes the most sensitive layers. Selection rules were written before each pilot ran (PLAN.md in each folder). One weakness: the lr pilot's raw-loss rule was noise-sensitive (single-batch loss at initialisation already spans 0.32–0.80) and only 60 steps long, which is why it passed an lr that later diverged; the stability study uses a smoothed loss and a gradient-norm criterion over 100 steps.

<!-- BEGIN:training -->

Total LoRA training wall-clock: 10.05 h on one Apple M4. <!-- END:training -->

5. Evaluation protocol

All configurations run on identical task files with greedy decoding, batched (64 episodes in lock-step, max 256 new tokens per turn), on the same machine. Configuration A⁺ uses the same weights as A, plus two worked examples (with tools that are not available) appended to the system prompt. Metrics are computed per episode (§2.3 for success):

Uncertainty. Each trained configuration's CI comes from a hierarchical bootstrap: resample training seeds, then tasks (4,000 resamples, 95% percentile intervals). Paired comparisons use per-task outcomes averaged over seeds, with a bootstrap CI on the difference and a two-sided sign-flip permutation p-value. Single-run configurations (A, A⁺, teacher, ablations) carry task-level bootstrap CIs only. Those CIs do not include training-seed variance, so they understate uncertainty.

6. Results

<!-- BEGIN:tables -->

Values: mean [95% CI]. Trained configs pool seeds via a hierarchical bootstrap (resample seeds, then tasks).

Workflow success (%) by split

Diagnostics (all main splits pooled)

Paired differences in success (percentage points; seed-averaged per task)

p_holm: Holm-Bonferroni adjusted over all 84 tests in this table. The smallest attainable permutation p is ~0.0002 (5,000 sign-flips).

Post-hoc: eval_unseen excluding the flawed un_timezone template (success %, n=90 tasks)

Added after results were seen; reported alongside, never instead of, the pre-registered eval_unseen.

Catalogue-size ablation (success %, eval_catalog_k: k distractor tools)

Success (%) by family (pooled main splits, seed-averaged)

Success (%) by gold workflow depth (pooled main splits, seed-averaged)

Success (%) by graph seen in train (pooled main splits, seed-averaged)

Success (%) by fault-injected / not-found (pooled main splits, seed-averaged)

Efficiency (per evaluation run)

<!-- END:tables -->

6.1 Findings by hypothesis

Paired p-values below are Holm-adjusted over all 84 tests in the table above. Paired CIs average each task over seeds and so exclude training-seed variance; the hierarchical CIs in the first table are the conservative ones.

• H1 (training helps): supported. Every trained configuration beats A on every split (e.g. B on eval_id +81.0 pts [+75.7, +86.0]). Pooled success: A 2.2%, B 55.8%, C 55.9%, Dmix 59.1%, Dseq 53.3%.
• H2 (beyond format): supported, except on unseen tools. Against the few-shot prompt (A⁺), B gains +74.0 pts on eval_id, +26.5 on eval_ood and +61.5 on eval_renamed; C, Dmix and Dseq are similar. Diagnostics that format cannot fix also improve: argument correctness 44% (A⁺) → 77–81%, dependency safety 75% → 91–94%, tool-selection F1 55% → 79–82%. On eval_unseen B, C and Dmix are level with A⁺ (differences −0.3 to +3.1 pts, all n.s.), and Dseq is worse (−24.2).
• H3 (generalisation): partly supported. Renaming every tool and parameter costs only ~13–21 pts (eval_id → eval_renamed), so models read the catalogue rather than memorise names. But unseen templates drop to 30–36%, and success on novel dependency graphs (26–38%) is about half that on graphs seen in training (72–74%). Depth-5 chains are almost never solved (≤4.4%, n=15). Most of the gain is pattern-specific.
• H4 (signal source): no aggregate difference; different profiles. C − B is between −4.2 and +3.7 pts on every split (all n.s.). By family (descriptive, not tested): B is near zero on approval (0.9% vs C 43.9%), which matches the single accepted approval episode in the teacher data; C is stronger on recovery (87.0% vs 66.7%) and makes fewer unnecessary calls (0.4 vs 1.1 per episode); B is stronger on parallel calls (64.2% vs 46.5%) and permission (20.0% vs 0.0%, n=30). C_matched (oracle labels on exactly B's tasks) loses 8.7 pts to C on eval_id (p_holm 0.021) and 7.7 to B on eval_renamed (p_holm 0.021); on one seed this suggests that which tasks are covered matters at least as much as who labelled them.
• H5 (combination): partly refuted. Dmix is numerically best pooled (59.1%) and beats B on eval_id (+5.0) and both B and C on eval_ood (+5.3 each), but none of these survives correction (p_holm ≥ 0.37). Strategy matters: sequential B→C (Dseq) matches Dmix in-distribution but falls on eval_unseen by −27.2 pts [−33.9, −20.3] (p_holm 0.021), consistently across all three seeds (8.3%, 4.2% and 15.0% per seed vs 22–53% for Dmix). On the inventory templates Dseq often mangles identifiers (e.g. calling reserve_stock with "535" for SKU-535); the cause is not established.

6.2 Ablations (single seed; seed-to-seed spread of C pooled success is 53.6–58.6%)

• Data quantity (oracle data, fixed 250 × 8 episode budget): C_25 55.0%, C (1,000) 55.9%, C_50 62.5%, C_full 52.2%. No monotonic trend; C_50 is the best single run, mainly on eval_unseen and eval_renamed. With one seed per point, only C_50's lead is outside C's own seed spread, and only modestly. C_full sees each of its 3,000 episodes 0.67 times, so it is not "more training".
• Catalogue size (0, 5, 12 distractor tools; seed 0): trained models lose at most 5.2 pts from 0 to 12 distractors, with overlapping CIs; distractors are not a bottleneck at this scale.
• Unseen tools / combination strategy: see H2 and H5.

7. Limitations

1. Synthetic, simulated environment. 21 deterministic tools and 44 templated scenarios. Requests are template-generated English; real APIs, real users and open-ended requests are not tested.
2. Weak, local teacher. No hosted API was available, so the teacher is a 4-bit 4B model that solves 71.7% of held-out tasks. B is bounded by it: only 63.2% of offered tasks yielded an accepted episode, and approval tasks yielded one. B-vs-C differences are therefore confounded with task coverage (C_matched isolates this on one seed only).
3. Flawed template. un_timezone (30 of 120 eval_unseen tasks) is answerable from world knowledge, and its gold graph requires a geocode call the system prompt never demands; every model, including the teacher, scores 0%. It was found after results were seen. The pre-registered split is reported unchanged; a clearly labelled post-hoc table excludes the template.
4. Uneven coverage of behaviours. eval_id and eval_renamed contain no permission tasks (all such requests overlapped the training pool verbatim and were removed), so permission handling is measured only on eval_ood (n=30). Approval (n=38), depth-4 and depth-5 strata (n=15 each) are small.
5. Seeds. Main configurations have 3 seeds each; ablations 1. Seed-to-seed spread is large (e.g. Dmix eval_unseen 22–53%), so single-seed ablation differences of a few points are not evidence.
6. Hyperparameters were not swept per configuration. The final setting was chosen after two diverged configurations, using pilots on Dmix and B (§4.1); all configurations share it. Other configurations might prefer different settings. Selection used validation loss on oracle data.
7. Multiple comparisons. 84 paired tests are reported; Holm-adjusted p-values are given. The smallest attainable permutation p (~0.0002) limits how small adjusted values can be.
8. Decoding and measurement. Greedy decoding, batched (64 episodes; teacher 24 because of memory), which is not bit-for-bit deterministic. Latency and throughput are measured at those different batch sizes and are not a like-for-like speed comparison.
9. Run history. Teacher pass 1 was interrupted once (Metal out-of-memory while other GPU jobs ran) and pass 2 once (a simulator crash on malformed teacher arguments, fixed by validating array items; outcomes of earlier episodes were unaffected). The prefill batch changed from 8 to 16 for the last 75 pass-2 episodes. The Mac slept during Dseq_s1 training; the trainer's clock excludes the sleep, the queue's wall-clock does not. All student evaluations ran after the simulator fix.
10. Provenance. The teacher's 4-bit weights were not revision-pinned at download time; the recorded revision is the Hub HEAD at the time (see MODEL_SELECTION.md). The few-shot baseline uses one prompt design; a better prompt could narrow the gap.

8. Compute and cost

<!-- BEGIN:queue --> Successful queue steps: 33 (training 11.18 h, evaluation 4.99 h wall-clock). <!-- END:queue -->

All work ran on one Apple M4 laptop with 16 GB of unified memory; no cloud compute and no API spend.

Peak memory: ≈ 4.0 GB for training, 3.0–3.9 GB for student evaluation, 13.2 GB for teacher evaluation. Cost: electricity only.

9. Reproducibility

See the README for the full sequence of commands. Every stage is deterministic given its seed and resumable; the remaining non-determinism is in batched GPU arithmetic. Dataset hashes are in data/v1/manifest.json and data/v1/sft/variants.json, adapter hashes in §4, and the pinned environment in requirements.lock. Model weights and adapters are not committed (size); adapter hashes allow verification of re-trained or shared adapters.