JARVIS Bench

Two changes on 11 September: the acknowledgement stopped blocking the provider call, then only task-relevant tools were offered to the model. Median turn time fell from 3.08 s to 1.27 s.

Test: Latency Lab (tools/latency/lab.py): 21 scripted journeys (status, time, recall, project, hud, latency, chat; three passes) through the normal launcher; end of input to end of turn.

The count fell from 256 to 9 because the definition of LIVE became stricter (a real booted process, a real port, no test doubles, evidence on disk, expiring after 24 hours), not because features were removed.

Test: Count of registered capabilities whose LIVE state is backed by a current (unexpired) probe receipt obtained by asking the running process; 304 capabilities registered.

Passes rose and then fell as training continued: evidence of interference rather than steady learning.

Test: Frozen 40-case TRAIN-derived transfer diagnostic on the update-24 checkpoint (8 FAIL).

Test: Complete pytest run via the project's suite runner in a fresh worktree (timeout 300 s per test); 10162 tests collected.

Failures are recorded, not skipped. The count rises when new tests land faster than fixes, and part of each reading was later traced to worktree-only artefacts.

Test: Same run: failed tests plus collection/setup errors (107 failed, 39 errors) of 10162 collected.

The gate set grew from 22 to 28. The dip on 17 September is a new mutation-testing gate that had not yet been run on that revision.

Test: Registered automated gates (secret scans, egress, orphan/binding checks, proof re-runs, mutation ratchet, etc.); 22 gates registered at the time.

Test: tools/mutation/mutate.py over 38 tier-1 modules (up to 14 mutants each): killed 273 of 527.

Test: tools/hud/shell_performance.py: control input to changed visible geometry, viewport width 1600px.

Readings from different voice paths are not directly comparable; each point names its path.

Test: tools/probes/probe_reflex_first_audio.py: wall clock from return of the injected transcribe callable to the real speaker's first audio write.

Test: tools/campaign/live_drive.py P14 engineering drive: spoken-style request to compare pressures on three plates; time to complete the turn.

Test: Tensile-bar reference case (1 m steel cube, one C3D8 hex, hand-derivable answer) solved by CalculiX; median of the day's wall times from the engineering run ledger.

An honest self-assessment against a demanding reference. It has not met its gate.

Test: Mean of twelve design/fidelity categories (cinematic hierarchy, density, originality, data truth, 3D depth, motion, etc.), scored 0–10; release gate requires every category strictly above 9.0.

Test: Latency Lab derived span speech_end_to_first_audio across all 21 journeys (reflex and cognition lanes).

No campaign came close to the gate. This is the core evidence behind the 21 September negative architecture decision.

Test: Fresh-process reload, 26 frozen validation cases (20 release-blocking) under the CP-7 contract; gate requires all blocking cases to pass.

Test: 25 adversarial development cases (19 release-blocking), fresh-process generation.

Test: Strict passes on the 166 paired DEV items.

Test: Frozen 40-case TRAIN-derived transfer diagnostic with actual output-bound independent semantic review; viability gate is at most 4 FAIL and zero critical FAIL.

Lower loss on the training targets (exact fit) did not produce passing transfer results. Fitting is not generalising.

Test: Mean negative log-likelihood of the 12 exact TRAIN targets (6 mechanism families × grounded/counterfactual).