JARVISLabs

Gemma 4 12B: mean exact-target loss while training

Lower loss on the training targets (exact fit) did not produce passing transfer results. Fitting is not generalising.

Gemma 4 12B: mean exact-target loss while training

Gemma 4 12B untuned 3.95 Gemma 4 12B after 16 updates (ter… 2.35 Gemma 4 12B after 48 updates (exp… 0.859
Scale: nats per token.

Lower loss on the training targets (exact fit) did not produce passing transfer results. Fitting is not generalising.

Test: Mean negative log-likelihood of the 12 exact TRAIN targets (6 mechanism families × grounded/counterfactual).

Protocol

Test definition

Mean negative log-likelihood of the 12 exact TRAIN targets (6 mechanism families × grounded/counterfactual).

Limits

Limitations

  • Arithmetic mean computed from the per-target values in the receipt. Lower likelihood loss on training targets has repeatedly NOT translated into passing behaviour; the 48-update behavioural evaluation was pending at compilation time.

Log

Recorded values

Variant Value Evidence Samples Source
Gemma 4 12B untuned 22 Sept 2026 3.95 nats per token CALCULATION 12 lm-runtime/reports/production-architecture-study-001-gemma4-bounded-adapter-results-00{5,6}/training/exact-target-nll-*.json
Gemma 4 12B after 16 updates (terminal 005) 22 Sept 2026 2.35 nats per token CALCULATION 12 lm-runtime/reports/production-architecture-study-001-gemma4-bounded-adapter-results-00{5,6}/training/exact-target-nll-*.json
Gemma 4 12B after 48 updates (exposure continuation 006) 22 Sept 2026 0.859 nats per token CALCULATION 12 lm-runtime/reports/production-architecture-study-001-gemma4-bounded-adapter-results-00{5,6}/training/exact-target-nll-*.json

Source paths refer to the private JARVIS repository and runtime records. They are listed for audit and are not published.