How the bench was built and how to read it

A "System One" model does not write an answer: it receives a text (the state) and several questions with a fixed type, and returns a probability distribution per question in a single pass. Every model receives exactly the same questions, with the same options and descriptions; each result stores a hash of the questions to catch any change. The scoreboard measures each configuration with a single run; the variation between two runs has been measured by replica on 11 configurations (see Experiments).

On triage and adversarial there are 5 questions per message: department, urgency, clinical, hostile and answer today. On papers, another 5: relevance, domain, design, reading depth and whether it changes clinical practice.

The adapter is a translator between our questions and a generalist model. There are three separate checks: that it finishes on time, that it produces a useful answer and that its confidence matches its hits. Improving one does not guarantee the others.

A schema is like a form: it controls the boxes, not the truth of what is written. An all-zero probability vector is invalid; turning it into equiprobable options can hide it. On the other hand, equal probabilities can also express legitimate doubt. That is why we need to keep the original answer.

Local Qwen improved without a forced schema; Cerebras works with both routes. The mechanism of the zeros is already identified: with a forced schema, several local servers do not copy the questions into the prompt (JEV-66). What is not isolated is how much of the gap versus Cerebras is due to the prompt, the server or the precision. As a reviewer of Jev, Cerebras is promising, not a demonstrated winner, and it offers no better calibration on these tests.

The isolation test was already run (JEV-63): two prompts × with/without schema × 12 cases × three repetitions, on NVFP4, FP8, Cerebras and GGUF on Intel Arc. The null vectors only appear with the TypeSafe prompt and the schema together; removing either one eliminates them. The external review found the mechanism and a confirmation experiment (JEV-66) verified it: in structured mode the local servers do not inject the output schema into the prompt, so the model answers blind; with the schema added to the prompt and the same grammar, zero null vectors on the three backends. The zeros are a measurement without visible questions, not a quality of the model. It is not a full battery nor a clinical validation.

The later ablation (JEV-65) ruled out that the prompt's anti-injection sentences are necessary or sufficient for the degeneration: removing or rewriting them still produces the null vectors under schema, and adding them to the simple prompt does not provoke them (the count variations are compatible with noise over a blind model; whether the sentences modulate with the questions visible was not measured). In discrete-answer mode the degeneration shows up as systematic output failure (one observed limit exhaustion in one block, only timeouts in the other), not as zeros.

The messages are invented; none comes from real patients. Each case's correct answer was drafted by language models and reviewed by people to varying degrees:

The GT is versioned: v1 is the original, v2 (Sep 26) corrects a paper and v3 (Sep 27) incorporates the second-annotation adjudication. The previous v4 (Oct 6) removes P02, whose fetch duplicated P03's paper (PMID 37130440), and keeps P03 with its GT. It adds no cases: 31 papers and 194 cases remain in the full battery (191 without OOD). The earlier data is kept and every model is re-scored; figures across GT versions are not directly comparable. The previous eleven field changes:

• Paper P04 (cryobiopsy in ILD): domain "ild", which was not a valid option, becomes ip; design cohort becomes meta (the title is "Systematic Review").
• T03 resting dyspnoea with orthopnoea and oedema: outpatient clinic becomes emergency.
• T10 complaint about the wait: answer today becomes no (the sender gives a deadline of "this week").
• T12 insurer asking for the indication report: clinical becomes yes.
• T26, T32, T36 and T38 (hostile or threatening messages): answer today becomes yes.
• T37 certificate to hand in today: medium urgency becomes critical (explicit deadline).
• A10 polite lawyer: hostile becomes no.

To measure the GT's noise, a blind form was prepared (showing neither the GT nor the models' answers) with the disputed fields: those where most models contradicted the GT or were split. Of the 167 fields, a person answered 78; the rest were completed by another model and are not used as evidence.

These are fields selected by disagreement; these figures describe that subset and do not estimate the GT's overall reliability. The relevant part: urgency and "answer today" are highly interpretable. Re-scoring every model with the second annotation's answers did not change the ordering between models; they all went up a few points.

For each experiment the success criterion was written before running it, and committed to version control. Negative results are documented the same way as positive ones. Closure figures with GT v4 (or v3 where noted); the current scoreboard uses GT v5.

The whole bench is measured with jev-1.13, which OpenRouter publishes as typesafe/jev-1.13-20260917 and the TypeSafe API as jev-1.13.0. Both give the same answers (406 of 409 equal; the 3 different ones sit on the rounding boundary). As of Sep 27, 2026, jev-latest and jev-preview point to that same version.

Jev has some call-to-call noise: re-running the battery changes the probabilities by about 0.01 on average and a borderline case here and there changes its answer. If the alias moves to a new version, the measurements have to be repeated before comparing.

The code (the jevbench harness), the cases, the versioned GT and every model's full answers are in the repository's public mirror (v2.10.0). The paper abstracts are not distributed: the repo carries metadata and hashes, and python3 -m jevbench.fetch_abstracts rebuilds them from PubMed. The README describes three levels of reproduction: redoing the maths with no keys and no GPU, re-running Jev (under $0.02) and re-running the open models on GPU.