When a fine-tuning run goes well, the obvious evidence is a falling loss curve. The model assigns higher probability to the answers it was shown, and the number goes down. JARVIS’s local-model work produced plenty of falling curves. It did not produce a model that behaves correctly on situations it was not shown. This entry sets the two kinds of numbers side by side, because the gap between them decided the project’s direction in September.
Two different questions
The project asks each candidate two separate questions.
Exact fit. Replay twelve prompts taken from the training set and check whether the model produces the trained behaviour. This asks whether the model learned what it was taught.
Transfer. A 40-case battery derived from the training material changes the situation in controlled ways: a different referent, an expired receipt, withheld authority, a paraphrase. This asks whether the model learned the mechanism rather than the wording.
A third, smaller set of anchor cases checks that broad behaviour from before training has survived.
Each case is judged PASS, PARTIAL or FAIL. The viability gate for any bounded probe is fixed in advance: exact replay at least 10 PASS with 0 FAIL, transfer with no critical failure and at most 4 failures in total, and anchors at least 7 PASS with 0 FAIL.
Early warning: validation loss fell, behaviour did not
The first sign appeared in the small foundation runs on 10 September. With 231, and then 271, training conversations, validation loss fell. Behaviour did not improve. The untuned base passed 22 of 51 validation targets. Adapter checkpoints ranged from 16 to 23, and several of them invented a completion or a tool that did not exist. The project wrote the conclusion down: “Reduced validation loss is not claimed as demonstrated general behavioral improvement.”
Ministral: the likelihood kept dropping
The clearest series comes from the Ministral-3-14B probes on 21 September, all measured on the same twelve exact-target cases:
| Configuration | Mean exact-target NLL | Exact (P/Pa/F) | Transfer (P/Pa/F) |
|---|---|---|---|
| Architecture001, update 96 | 1.560626 | 5 / 5 / 2 | n/a |
| Architecture001, update 125 | 1.507634 | n/a | n/a |
| +16 updates of exposure | 0.964849 | 10 / 1 / 1 | 31 / 3 / 6 |
| + MLP down-projection capacity | 0.805738 | 9 / 1 / 2 | 32 / 5 / 3 |
The loss nearly halved between the parent and the capacity probe. The capacity probe even met the transfer failure count of at most 4. It still failed viability, because an independent reviewer found critical errors among its answers: it accepted a receipt it should have rejected and invented a sequence of steps. The ledger’s verdict was “lower supervised likelihood did not remove critical behavioral errors.”
Learned, then lost
A second finding came from scoring Recovery004 at several points along one training run. Transfer PASS counts out of 40 were 29 at update 24, 36 at update 48, then 32 at update 72 and 32 at update 93. The model learned something by the midpoint that it had lost by the end. The project recorded this as “evidence of interference/forgetting, not a monotonic need for more epochs.”
The later Architecture001 analysis found the same pattern at the level of individual behaviours. Later checkpoints learned which receipt intervals applied but started proposing self-approval, and the final checkpoint accepted an expired receipt. More exposure traded one mechanism for another.
Part of the cause was in the data. A generalisation map found that 704 targeted replacement rows had pushed broad-behaviour rows down to 82 of 786, which is a measured distribution defect. Architecture001 restored those anchors and added counterfactual pairs, trained so that each positive and negative example lands in the same optimizer step. That improved the development battery (101 of 166 conditioned passes, against 89 for Recovery004). It did not pass the gates.
Other families, same shape
The stronger-local study repeated the experiment on different bases:
- Qwen3.5-9B. Untuned: 4/0/8 exact, 20/3/17 transfer. A 16-update adapter reached 11/1/0 exact and 8/0/0 anchors, clearing both of those gates, but scored 24/7/9 on transfer. Continuing to 48 updates gave 11/0/1 exact and 32/1/7 transfer. Exact fit was essentially solved. Transfer still failed.
- Gemma 4 12B. Untuned, it failed 8 of 40 transfer cases. Four bounded specialisation contrasts evaluated on 22 September all failed the gates, and the best of them reached 24/7/9 on transfer. Mean exact-target loss went from 3.95 untuned to 2.35 after 16 updates and 0.86 after 48. The behavioural evaluation of the 48-update run was pending when this was written.
The Gemma loss figures are arithmetic means computed from the training receipt. They measure how probable the exact training answers have become, and they say nothing yet about transfer.
Why the gate is behaviour, not loss
Negative log-likelihood on a target answer measures agreement with one specific wording. JARVIS’s hard cases are rarely about wording. They are about state: whether a receipt is still valid, whether authority was granted, whether “it” still refers to the same window. A model can come to assign high probability to “I cannot approve that myself” in the trained context and still propose an approval when the same situation is described differently.
The project draws two practical rules from this.
- Loss selects nothing. Checkpoints are chosen by a frozen behavioural diagnostic under a rule declared before training, never by loss and never after looking at canonical results.
- Wording failures still count. Some automatic failures are exact-wording or length checks, not substantive errors. The project reports them separately and did not weaken them. A gate that bends after the results arrive stops measuring anything.
For Gemma, the pre-registered decision rule states the lesson directly: if fit improves but transfer still fails, “do not automatically extend again”.
What this does and does not show
The series shows that on these bases, with these adapters and this data, more fitting did not produce the behaviour the gates require. It does not show that no local model can. Longer reasoning modes, other ranks and new context representations were not measured. Until a candidate clears transfer, production reasoning stays on the hosted model, and the local model stays experimental.