cognitive-core-lab
Trace explorer

Watch each configuration work through the same task

Every configuration is evaluated on identical held-out tasks. Pick a task, then read the turn-by-turn trace: the model's raw output, the calls the runtime parsed from it, what each call returned, and how the validator scored the episode. Replay shows recorded evaluation traces; Live runs a configuration on this machine through the local inference server.

50 shared tasks · same list for every configuration
In-distributionfamily: selectsel_weatherrole: userdepth 1graph seen in trainingexpected status: done
Forecast for Seville on 2026-12-19, please.

eval-sel_weather-10000000-d7f00c · simulated date 2026-11-18 · select: pick the one right tool among distractors

Tool catalogue shown to the model (4)

  • weather_backup(city: string, date: string YYYY-MM-DD)
    Secondary weather provider. Use only if get_weather is unavailable.
  • find_contact(name: string)
    Look up a person in the address book by name.
  • get_weather(city: string, date: string YYYY-MM-DD)
    Weather forecast for a city on a date.
  • lookup_order(order_id: string)
    Get an order by order ID.

Gold dependency graph

g1get_weather

Columns are dependency depth; arrows point from prerequisite to dependent call. Nodes in the same column are independent and can run in parallel.

Facts the answer must contain

5snowy
This task across configurations (recorded seed-0 runs) — select to view

A · Baseline (zero-shot)

REPLAY— recorded trace from evaluation run A_base
failureno status4 turns · 1 callended: parse errors89 generated tokensfailure mode: malformed output
  1. Turn 1tool callsnon-strict output32 tok
    Model output (raw)
    {"tool": "get_weather", "args": {"city": "Seville", "date": "2026-12-19"}}
    Parsed call → runtime observation
    get_weatherok
    args
    {"city":"Seville","date":"2026-12-19"}
    result
    {"city":"Seville","date":"2026-12-19","condition":"snowy","temp_c":5}
  2. Turn 2unparseablenon-strict output27 tok
    Model output (raw)
    The forecast for Seville on 2026-12-19 is snowy with a temperature of 5°C.
    Parse error: no JSON object found
  3. Turn 3unparseablenon-strict output15 tok
    Model output (raw)
    I'm sorry, but I'm unable to proceed without a valid request.
    Parse error: no JSON object found
  4. Turn 4unparseablenon-strict output15 tok
    Model output (raw)
    I'm sorry, but I'm unable to proceed without a valid request.
    Parse error: no JSON object found

Final answer

No final action — the episode ended by parse_errors.

Validator checks

  • failed:terminated with a final action
  • failed:correct final status
  • failed:answer contains required facts
  • passed:required calls made
  • passed:side effects match
  • passed:no permission violation
  • passed:no retry after denial