Does our fine-tuning improve on the same base queried in the same format? This LoRA fails the specified success criterion and shows worse calibration, with no significant cell-level differences after Holm. Missing manipulation coverage in training is a post-hoc explanation that remains untested.
55. Own LoRA on Qwen3.5-9B versus its base JEV-89
- Question
- Does a Qwen3.5-9B LoRA trained on our own dataset (no evaluation case) decide better than the same untrained base, asked in the same format (Tev1 contract), and where does it land against Jev and Nimble-9B?
- Criterion
- H1 (primary, versus the base): confirmed if any of the 53 cells survives Holm in favour of the trained model and none against; refuted if any survives against or its adjusted score is below the control's. Practical success: adjusted score ≥ 44 and no cell in favour of jev_v3. A single run; the checkpoint is chosen by synthetic-validation log-loss, never by looking at the battery.
- Dates
- Pre-registration: 2026-10-17 · Results: 2026-10-18 · Amendment: 2026-10-17 documentary declaration
- Models
jev89_qwen35_9b_lora · subjectqwen35_9b_base_tev1fmt · controljev_v3 · Holm referencenimble_9b · Holm reference
- Result
- Measured. The untrained base, asked in Tev1 format, obtains these results against the comparators (including a dedicated ~9B decider): adjusted score 48 [38–57] versus Jev 49 [39–59] and Nimble-9B 47 [37–56] (majority 0), with no significant differences after Holm. This contrast does not isolate the effect of format. The LoRA lands at 38 [25–51]: H1 REFUTED by the pre-registered criterion (adjusted score below the control's), with 0/53 cells after Holm in either direction (b = 90, c = 86). It makes almost the same errors (200 → 204 of 964), but those at confidence ≥ 0.99 go from 7 to 50 and the noul Brier worsens from 0.083 to 0.108: fine-tuning makes it overconfident. It improves triage (triage_en 51.1 → 74.5) and adv4/adv5 (31.9 → 38.3; 36.5 → 54.1); its scores are descriptively lower on manipulation (adv1 26.7 → −86.7; adv3 66.3 → 42.7) and on papers (accuracy 73.5 → 68.4 %).
- Verdict
- H1 refuted because the adjusted score is below the base; no practical success. No cell-level difference is significant after Holm.
- GT
- v5 (current); single run on Oct 18
Cited figures (5)
- Adjusted score [CI95]: jev_v3 49 [39–59] · control 48 [38–57] · nimble_9b 47 [37–56] · LoRA 38 [25–51] · majority 0 (Battery (194 cases, 0 errors) · GT v5)
- Adjusted score by phase, control → LoRA (majority %): triage_es 61.7 → 66.0 (66.4) · triage_en 51.1 → 74.5 (66.4) · triage_ext_es 73.6 → 74.5 (57.7) · triage_ext_en 73.6 → 76.4 (57.7) · papers32 45.0 → 34.2 (51.9) · adv1 26.7 → −86.7 (85.0) · adv2 16.0 → 4.0 (75.0) · adv3 66.3 → 42.7 (55.5) · adv4 31.9 → 38.3 (76.5) · adv5 36.5 → 54.1 (63.0) (Battery (194 cases, 0 errors) · GT v5)
- McNemar + Holm (53 cells), minimum Holm p: LoRA versus the control 1.0 (b = 90, c = 86); versus jev_v3 1.0 (b = 94, c = 67); versus nimble_9b 0.41 (papers32 domain, 8–0 in favour of Nimble, nominal p 0.008). Control versus jev_v3 and versus nimble_9b: 1.0 (Battery (194 cases, 0 errors) · GT v5)
- Errors out of 964 (at confidence ≥ 0.99): control 200 (7) · LoRA 204 (50) · jev_v3 177 (7) · nimble_9b 178 (2); on noul, 14 for the LoRA versus 1 for the control (High-confidence errors · GT v5)
- Synthetic validation, epoch 2 chosen: log-loss 0.069 and accuracy 0.976 (untrained base: 0.482 and 0.855) (Training)
Caveat: Post-hoc interpretation, untested (not part of the verdict). The LoRA learns what is in the data and unlearns what is not: the training set had no verified manipulation example, and acceptance by exact agreement keeps only unambiguous cases, which rewards trusting the apparent clinical content; the drop on papers fits weak MeSH labels. adv1 has 10 cases and an 85 % majority, so it weighs heavily on the adjusted score. A high synthetic validation (97.6 %) did not predict the battery: it measured resemblance to the generator. The pre-registration decided the verdict: without the criterion fixed beforehand, it would have been easy to read it as a tie by removing adv1 or by picking the epoch while looking at the battery.
Notes: Measured lesson: any fine-tuned decider must be compared with its own base asked in the same format, not with zero. The two runs are documented only as an experiment and do not enter the leaderboard; the weights are not published (the dataset includes CC BY-NC data) and the model is research-only.
Documents: jev89_preregistro.md jev89_resultado.md jev89_qwen35_9b.md
system-one-bench