On 21 September the project closed its first local-model effort with a formal negative decision: no full training run, and no architecture selected from the probes so far. What followed was not a new campaign. The owner authorised a bounded study, strictly limited in scope, to find out why the model was not learning cleanly, and whether any small adaptation could pass the frozen gate.
The gate
Every experiment is graded on the same 60 frozen cases: 12 exact tasks, 40 transfer cases that test whether a principle generalises to new wording, and 8 anchors that check nothing the base model already did well was broken. Gate C requires at least 10 exact passes with no fails, no critical transfer fails and at most 4 transfer fails, and at least 7 anchor passes with no fails. Two stricter rules were added on 22 September: no transfer case may be worse than the untrained base, and answers must arrive within latency limits. No threshold has ever been relaxed.
Every run froze its plan, took an exclusive GPU measurement window, reloaded the adapter in a fresh process, and was graded by independent reviewers.
Three base models in two days
Qwen3.5-9B came closest. After 48 updates of a small adapter it reached 11 of 12 exact, 32 of 40 transfer and 8 of 8 anchors. It still failed on critical state-reasoning errors.
Gemma-4-12B-it became the study subject on the evening of 21 September. Quantised to 4 bits it fits a single 12 GB GPU with a small adapter: about 5.5 million trainable parameters, 0.046% of the model. A one-update training run saved and reloaded in a fresh process with exactly matching loss. That proved the training runtime, not the behaviour.
Twelve single variables
Then came about twelve experiments, each changing one thing:
- the curriculum;
- the weighting between exact and supporting examples;
- a native reasoning mode;
- an adapter-off control;
- a gradient census;
- longer exposure;
- more diverse data;
- targeted coverage.
None passed. Three results mattered most.
Longer training made behaviour worse. Continuing from 16 to 48 updates fitted the exact targets much better, with loss falling from 2.35 to 0.86. But anchors dropped from 8 passes to 4. The result was 9 pass and 3 fail on exact, 27 pass, 5 partial and 8 fail on transfer, and 4 pass and 4 fail on anchors.
The base model was hard to beat on transfer. With the adapter switched off, the base scored better on transfer than every adapter so far. The best Gemma run, plan010 on 23 September, reached 8 passes on exact, 28 on transfer and 6 on anchors. Those were the best transfer and anchor results on Gemma, and still only equal to the untrained base on transfer.
The obvious suspects were cleared. Parity audits found no difference between how the model was trained and how it was evaluated. A cache check agreed on 192 of 192 token choices. A gradient census over 152 backward passes found every training group pulling the same way, so the objectives were not fighting each other inside training.
What the adapter mostly learned was a style: shorter, more committal answers that stop hedging. Those gained on the narrow targets and lost everywhere else.
The bug was in the prompt
The breakthrough came from a failure-matrix audit, not a training run. The system prompt’s own example of how to say “no record” described an inspection that had never happened: exactly the fabrication the same prompt forbids. That example was present in every one of the 1,997 training rows and every one of the 60 test cases. The model had been copying its instructions.
The owner chose a contract v2 response. A corrected persona policy lets JARVIS say “no record” only after a search it actually ran in that turn; it is tested and landed, but not yet active. The deterministic body will compute simple facts itself, such as whether a receipt is still valid or whether a version is current, and show its working without ever handing the model a verdict. A new versioned gate is frozen alongside. The old gate stays exactly as it was, and its results stay published.
Where it stands
No model is qualified, and 0 of 5 production training campaigns have been used. Production reasoning still uses a hosted model. The training stack as run is preserved byte for byte in custody snapshots, so every result above can be reproduced. The next decisive experiment is to run the base model on the old and new contracts side by side, to isolate the contract’s effect before any more training.
The training page explains the method, and the Bench charts every run.