cognitive-core-lab
Research question

Can LoRA fine-tuning teach a 0.5B model to choose and orchestrate tools?

Small models are cheap enough to run an agent loop locally, but they tend to pick the wrong tool, invent arguments and lose track of multi-step plans. This project measures whether low-rank fine-tuning of Qwen2.5-0.5B-Instruct fixes that — and whether it matters where the training trajectories come from — on identical held-out tasks, with confidence intervals, under a strict runtime.

Explore tracesBenchmarksMethods & data
8/8configurations with results
4held-out evaluation splits
2026-11-10 08:24 UTCdata exported
364c676git commit of export

Workflow success by evaluation split

Share of episodes where every validator check passes (right calls, right arguments, safe ordering, right side effects, right final status and answer). Dots are means; whiskers are 95% bootstrap confidence intervals over tasks (and over training seeds for LoRA configurations).

All metrics →
success by split with 95% confidence intervals
In-distribution eval_id — held-out instances of training templates
Unseen templates eval_ood — templates never seen in training (novel phrasing / task graphs / depth 4-5)
Unseen tool domains eval_unseen — tool domains never seen in training (inventory, maps)
Renamed tools eval_renamed — training templates with every tool/parameter renamed (held-out alias vocabulary)

Configurations compared

All student configurations share the same base weights, system prompt, tool-catalogue rendering, decoding settings and runtime limits. Only the LoRA adapter (or the prompt, for A+) differs.

Baselines (no weight change)

What the 0.5B model does out of the box, with and without worked examples in the prompt.

A · Baseline (zero-shot)

Unmodified Qwen2.5-0.5B-Instruct with the shared system prompt.

results · 1 seedA_base

A+ · Baseline (few-shot prompt)

Same weights as A; the system prompt adds two worked examples (prompting, no weight change).

results · 1 seedA_fewshot

LoRA on teacher demonstrations

Distillation: trajectories from a 4B teacher, kept only if they execute and validate.

B · Teacher-distilled LoRA

LoRA on execution-verified demonstrations generated by the Qwen3.5-4B teacher.

results · 3 seedsB_s0

LoRA on independent trajectories

Trajectories produced by a programmatic policy, also execution-verified — no stronger model involved.

C · Independent (oracle) LoRA

LoRA on trajectories from an independent programmatic oracle, executed and validated.

results · 3 seedsC_s0

Combined

Both data sources: shuffled together, or one after the other.

D · Combined LoRA (mixed)

LoRA on the union of B and C data, shuffled together (same step budget).

results · 3 seedsDmix_s0

D' · Combined LoRA (sequential B→C)

LoRA trained on B data for half the steps, then C data for the other half.

results · 3 seedsDseq_s0

References

Not candidates — context for how far the student is from a larger model and from the ceiling.

Teacher · Qwen3.5-4B (4-bit)

Qwen3.5-4B (MLX 4-bit) — reference only; it produced the B data.

results · 1 seedteacher

Oracle (programmatic upper bound)

Programmatic policy reading the gold task graph — the validator's ceiling, not a model.

results · 1 seedoracle

How an episode runs

The model sees a request, the user's role and a tool catalogue, and must reply with exactly one JSON action per turn. A runtime parses it, enforces permissions and human approval, executes calls on deterministic simulators and returns observations until the model finishes.

1 · Call tools

{"calls": [
  {"tool": "find_contact", "args": {"name": "…"}}
]}

Several calls in one list are independent and run in parallel. Unknown tools, bad arguments, missing roles and denied approvals come back as error observations.

2 · Read observations

{"tool": "find_contact",
 "result": {…}}
{"error": "approval_denied", …}

Later arguments must come from earlier observations — inventing an identifier counts as a dependency violation even if it happens to be right.

3 · Finish

{"final": "…",
 "status": "done | refused |
            blocked | failed"}

The status matters: refusing a forbidden action or reporting a blocked approval is the correct outcome for some tasks.

Evaluation splits

Each split probes a different kind of generalisation beyond the training distribution.

splitwhat it tests
eval_idheld-out instances of training templates
eval_oodtemplates never seen in training (novel phrasing / task graphs / depth 4-5)
eval_unseentool domains never seen in training (inventory, maps)
eval_renamedtraining templates with every tool/parameter renamed (held-out alias vocabulary)

Read the numbers with these caveats

  • Synthetic, deterministic tools. Tasks run against simulators, not real APIs. That makes results exactly reproducible and every episode checkable, but real tools are messier.
  • The oracle is a ceiling, not a model. It reads the gold task graph. Its role is to show that every task is solvable under the validator.
  • The teacher is filtered. Configuration B only trains on teacher episodes that executed and validated, so its data is skewed towards tasks the teacher already solves.
  • Intervals are about tasks and seeds, not the world. The 95% intervals resample the evaluation tasks (and training seeds where there are several). They do not cover other prompts, other base models or other tool ecosystems.
  • Some slices are small. Breakdown cells (e.g. deep workflows or rare families) can contain only a handful of tasks — always check the n shown next to each number.
  • One base model. Every student is Qwen2.5-0.5B-Instruct; conclusions may not transfer to other sizes or families.