system-one-bench ES·EN v2.10.0
Experiments

Limits and caveats

What was tested, what was found and with what certainty — schemas, thinking, modes and option order. Measured on Qwen3.8-27B via the LLM adapter (except DiffusionGemma in §8). Summary on the home page.

Subject of this page. Almost every figure comes from Qwen3.8-27B, an open-weight generalist LLM scored via the llm adapter (system-one-adapter, TypeSafe contract)—not a dedicated decider like Jev/Clef. The pre-registered confirmatory session (hypotheses §2, cells §5) is local FP8 (SGLang). The bars in §6–§7 add the same questions under NVFP4, GGUF, and, where noted, the Cerebras API—same model, different quant or host. Schema non-injection (§3–§4) was also probed on llama.cpp, vLLM 0.29.0 and Ollama 0.32.14. The only other-model row is DiffusionGemma in §8 (rotation), shown separately on purpose.

1. Map of the experiment

What is changed, what is measured and where each finding lives. The adjusted subtracts the majority baseline: 0 = doing the same as always answering the majority class; high values = real improvement over that trivial floor.

From variables to findings

Four variables: schema visibility, probability or discrete mode, thinking and option order. The measured metric is the adjusted score with reference 0 equal to the majority. Findings: blindness, hypothesis forest, tokens, modes and rotation.

Map: variables, metric and findings Left-to-right flow: four variables are changed; the adjusted score is measured (0 = majority); each finding points to its section. What changes 1. Does it see the questions? (blind · visible / inject) 2. Output mode (probabilities · discrete) 3. Thinking on/off 4. Option order (department d0–d3) What is measured Adjusted accuracy − majority 0 = majority Where to see it → Hypotheses (forest) → Who injects → Tokens / canary → Blindness → adjusted → Discrete vs prob. → Order / rotation Qwen3.8-27B FP8 local session · GT v4 · thresholds fixed before seeing results
Short glossary (bench terms)
Adjusted
Accuracy points above (or below) the majority class. 0 = no better than the trivial floor.
Visible / inject
The typed questions are copied into the system prompt. Without that, on several local servers Qwen3.8-27B answers blind.
Blind (struct without inject)
Schema forced only as output grammar; the questions do not appear in the text the model reads.
probabilities / discrete
The adapter's two output modes (distributions vs a discrete choice). They change prompt and format at once.
Thinking
The model's reasoning chain turned on or off.
F0–F3 / T0–T1 / D0–D1 / DT
Session cells: F = probabilities without thinking (orders d0–d3); T = with thinking; D = discrete without thinking; DT = discrete with thinking (JEV-76 session, which repeats F/T/D as fresh controls; ′ marks that session's control).
pp
Percentage points (a difference of percentages, not adjusted points).
Canary
Behavioural test: it only hits the correct label if it reads its description. Observational; visibility is accredited by tokens + the blind control.
CONFIRMED / INCONCLUSIVE / REFUTED
Verdict against the pre-registered threshold and its confidence interval — not "there is/is no effect" in the abstract.

2. Pre-registered hypotheses: estimate, CI and verdict

Each row is a hypothesis with a threshold fixed before running. The dot is the estimate; the horizontal bar, the confidence interval; the dashed line, the threshold. The badge is the verdict (colour and text). INCONCLUSIVE ≠ "no effect": the CI does not cross the threshold the required way.

Forest plot — Qwen3.8-27B FP8 session (GT v4)

Thinking original order T0 minus F0: estimate +11.69, 97.5 % CI [−0.28, +28.04], threshold ≥ +5, INCONCLUSIVE. Thinking rotated order T1 minus F1: +6.51, 97.5 % CI [−1.97, +16.58], threshold ≥ +5, INCONCLUSIVE. Stability range F0 to F3: 9.62, 95 % CI [4.58, 17.55], threshold ≤ 5, INCONCLUSIVE. Primacy bronchoscopia: +35.14 pp, 95 % CI [16.67, 52.78], threshold ≥ +5 pp, n=37, CONFIRMED. Discrete minus F0: +12.85, 95 % CI [3.11, 24.94], threshold ≥ +10, INCONCLUSIVE. Replica S202: 253 of 255 (99.22 %), 95 % CI [97.2 %, 99.9 %], threshold ≥ 97 %, CONFIRMED. Anti-injection V3: −1.11 pp, 97.5 % CI [−3.33, 0], threshold ≥ +10 pp, n=12, REFUTED. Anti-injection V5: −2.22 pp, 97.5 % CI [−8.89, 0], threshold ≥ +10 pp, n=12, REFUTED.

Forest plot of hypotheses with CI and threshold Thinking T0−F0: +11.69 [−0.28, +28.04], threshold ≥ +5, INCONCLUSIVE. Thinking T1−F1: +6.51 [−1.97, +16.58], threshold ≥ +5, INCONCLUSIVE. Range F0–F3: 9.62 [4.58, 17.55], threshold ≤ 5, INCONCLUSIVE. Primacy: +35.14 pp [16.67, 52.78], threshold ≥ +5 pp, CONFIRMED. Discrete D0−F0: +12.85 [3.11, 24.94], threshold ≥ +10, INCONCLUSIVE. S202: 253/255 [97.2 %, 99.9 %], threshold ≥ 97 %, CONFIRMED. V3 n=12: −1.11 pp [−3.33, 0], threshold ≥ +10 pp, REFUTED. V5 n=12: −2.22 pp [−8.89, 0], threshold ≥ +10 pp, REFUTED. Thinking · original order (T0−F0) Δ adjusted · 97.5 % CI · threshold ≥ +5 +5 +11.69 [−0.28, +28.04] INCONCLUSIVE Thinking · rotated order (T1−F1) Δ adjusted · 97.5 % CI · threshold ≥ +5 +5 +6.51 [−1.97, +16.58] INCONCLUSIVE Stability across orders (F0–F3) adjusted range · 95 % CI · threshold ≤ 5 ≤5 9.62 [4.58, 17.55] INCONCLUSIVE Primacy of "bronchoscopia" Δ dept. errors (first−last) · 95 % CI +5 pp +35.14 pp [16.67, 52.78] CONFIRMED Discrete vs probabilities (D0−F0) Δ adjusted · 95 % CI · threshold ≥ +10 +10 +12.85 [3.11, 24.94] INCONCLUSIVE Replica S202 (same order, another seed) decision agreement · 95 % Clopper-Pearson CI · threshold ≥ 97 % 97% 253/255 [97.2, 99.9]% CONFIRMED Anti-injection sentences V3 (visible, n=12) Δ diagnostic accuracy · 97.5 % CI · threshold ≥ +10 pp +10 −1.11 pp [−3.33, 0] REFUTED Anti-injection sentences V5 (visible, n=12) Δ diagnostic accuracy · 97.5 % CI · threshold ≥ +10 pp +10 −2.22 pp [−8.89, 0] REFUTED Primacy: n=37 records (35 cases), Holm p=0.009155. Each row uses its own horizontal scale (see the formula in the source).

Exact figures from the «Pre-registered classification» table of the session's results manifest (2026-10-07). F0–F3 range mean: 52.19. Primacy errors d0/d1/d2/d3 = 22/9/15/24 out of 37.

The JEV-76 session (Oct 7) completes the factorial with the missing cell — discrete + thinking (DT) — and repeats F/T/D as fresh controls in both orders (8×194 cases, same GT v4 battery). Four primary contrasts, all with the same ≥ +5 threshold and 98.75 % CI (Bonferroni for a 95 % family coverage).

Forest plot — JEV-76 session: discrete × thinking (GT v4)

Thinking over discrete, original order, DT0 minus D0′: estimate −3.48, 98.75 % CI [−11.39, +3.62], threshold ≥ +5, REFUTED. Thinking over discrete, rotated order, DT1 minus D1: −5.58, 98.75 % CI [−14.82, +2.59], threshold ≥ +5, REFUTED. Discrete vs prob with thinking, original order, DT0 minus T0′: +0.85, 98.75 % CI [−5.19, +6.75], threshold ≥ +5, INCONCLUSIVE. Discrete vs prob with thinking, rotated order, DT1 minus T1′: −8.65, 98.75 % CI [−18.09, −1.94], threshold ≥ +5, REFUTED.

Forest plot of the discrete × thinking factorial with CI and threshold DT0−D0′: −3.48 [−11.39, +3.62], threshold ≥ +5, REFUTED. DT1−D1: −5.58 [−14.82, +2.59], threshold ≥ +5, REFUTED. DT0−T0′: +0.85 [−5.19, +6.75], threshold ≥ +5, INCONCLUSIVE. DT1−T1′: −8.65 [−18.09, −1.94], threshold ≥ +5, REFUTED. Thinking over discrete · original order (DT0−D0′) Δ adjusted · 98.75 % CI · threshold ≥ +5 +5 −3.48 [−11.39, +3.62] REFUTED Thinking over discrete · rotated order (DT1−D1) Δ adjusted · 98.75 % CI · threshold ≥ +5 +5 −5.58 [−14.82, +2.59] REFUTED Discrete vs prob with thinking · original order (DT0−T0′) Δ adjusted · 98.75 % CI · threshold ≥ +5 +5 +0.85 [−5.19, +6.75] INCONCLUSIVE Discrete vs prob with thinking · rotated order (DT1−T1′) Δ adjusted · 98.75 % CI · threshold ≥ +5 +5 −8.65 [−18.09, −1.94] REFUTED

The four rows share the scale (−20 to +10; formula in the chart's source), so the +5 threshold falls on the same x. H1's CIs include zero: neither degradation nor equivalence demonstrated. Exact figures from «Pre-registered classification» in docs/infra_runs/qwen38_jev76.md §RESULTADOS (session s81f-20261007-0800; 1,552/1,552 cases, 0 errors). Descriptive interaction (outside the confirmatory family): ≈ −12.1 points in both orders.

3. How the questions reach the model

With structured output, the adapter sends the system prompt and the document in the chat. The typed questions travel in response_format (JSON Schema). If the server does not copy that schema into the prompt, the model answers blind — it only uses the schema as the grammar of what it can write. On this page the measured subject is Qwen3.8-27B; the list names servers where non-injection was observed.

Diptych: on the left a closed, veiled document (the model does not see the typed questions); on the right the same document open with legible typed marks (the questions are in the prompt)
Without the questions in the prompt the model does not see them; with inject (or Cerebras) they are part of what it reads.
Before / after: do the questions appear in the text the model reads?

Two columns. Before: TypeSafe probabilities prompt without questions in the system, schema as grammar only; Cerebras does inject, several local servers do not. After: inject copies the schema into the system prompt; the model sees the questions.

Blind prompt versus prompt with visible questions Left column: TypeSafe probabilities system («Evaluate every question…») and a document without questions; response_format outside the prompt. Right column: schema added to the system; grammar still active. BEFORE — blind on local servers system (TypeSafe probabilities): Evaluate every question using only the supplied document.… user: [case document] questions only in response_format (outside the text being read) SGLang · llama.cpp vLLM 0.29.0 · Ollama 0.32.14 schema = grammar, not reading AFTER — inject / Cerebras system: Evaluate every question using only the supplied document.… + JSON Schema (department…) user: [case document] The model reads the questions Cerebras injects on its own; locally: inject_schema_in_prompt grammar still active

Measured local mitigation: copying the schema into the system prompt while keeping the grammar. Upstream proposal: system-one-adapter#50.

4. Token probe (visibility)

Same request, same case: if the questions are in the prompt, there are many more input tokens. Operational gate: visible − blind ≥ 200 tokens, plus a blind control. The behavioural canary (choosing a label only if its description is read) remains an observation: under structured grammar the label's name can give it away by chance.

Input tokens · same case · FP8 / SGLang

Horizontal bars: blind around 222 tokens; visible with inject around 806. Difference 584, above the 200 threshold.

Input tokens blind versus visible Blind bar approximately 222 tokens; visible bar with inject approximately 806. Difference threshold 200 tokens. input tokens → Blind (schema as grammar only) ~222 Visible (questions in the prompt) ~806 Δ ≈ 584 · gate ≥ 200 · canary = observation

On Cerebras the measured contrast was ~222 local versus ~958 with provider injection (same idea: visible questions).

5. Adjusted score per cell (FP8 session)

Seven cells from the same local session (struct+inject, GT v4, 194 cases). Axis from majority = 0: further right = better than the trivial floor. No per-cell CIs on this chart (the forest above carries the hypothesis CIs).

Adjusted · Qwen3.8-27B FP8 · GT v4 · majority = 0

Bars: no thinking original order 49.57; rotated order 58.41; d2 51.97; d3 48.80; with thinking d0 61.26; with thinking d1 64.92; discrete 62.42.

Adjusted per session cell Seven horizontal adjusted-score bars. Reference majority equal to zero. No confidence intervals in this figure. majority = 0 → better No thinking, original order (F0) 49.57 No thinking, rotated order (F1) 58.41 No thinking, order d2 (F2) 51.97 No thinking, order d3 (F3) 48.80 With thinking, original order (T0) 61.26 With thinking, rotated order (T1) 64.92 Discrete, original order (D0) 62.42 No CIs here · see the forest (§2) for hypotheses

6. Effect of blindness on the adjusted score

When Qwen3.8-27B does not see the questions, the adjusted score falls below the majority (negative). With visible questions it returns to the ~48–52 band. That explains the trivial floor; it does not close the gap versus the same model on Cerebras (59–66). Descriptive comparison (not a single-variable ablation); no CIs on these scoreboard bars.

Adjusted (descriptive) · majority = 0 · no CIs on the scoreboard

Historical blind minus 15 asterisk GT v4; inject FP8 51; NVFP4 50; GGUF 48; without forced schema NVFP4 52.

Adjusted blind versus visible Horizontal bars. Negative below majority when blind; positive with visible questions. Reference 0 = majority. ← worse · majority = 0 · better → Historical blind (GT v4) −15* Visible inject · FP8 51 Visible inject · NVFP4 50 Visible inject · GGUF 48 No forced schema · NVFP4 52 * −15 with GT v4 (−16 was GT v3). Partial coverage on the historical blind.

7. Discrete versus probabilities

With visible questions, discrete mode raises the adjusted score versus probabilities on the local backends measured for Qwen3.8-27B (NVFP4/GGUF). The mode changes prompt and format at once; on Cerebras the sign flips. Here: scoreboard figures, no CIs (the forest §2 carries the CI of discrete−F0 on FP8: +12.85 [3.11, 24.94], threshold +10 → INCONCLUSIVE).

Local adjusted with questions in the prompt · majority = 0

Pairs: NVFP4 probabilities 50 versus discrete 63; GGUF 48 versus 63.

Discrete versus probabilities on the adjusted score Two bar pairs from majority equal to zero. NVFP4 50 vs 63; GGUF 48 vs 63. No confidence intervals. majority = 0 NVFP4 prob 50 disc 63 (+13) GGUF prob 48 disc 63 (+15)

8. Sensitivity to option order

Rotating the order of the choice options moves the accuracy even though the content is the same. They are not the same experiment or the same model: they are shown separately. Only DiffusionGemma publishes a CI95 on this figure; Qwen A3 is a point Δ with no CI here (primacy with a CI is in the forest §2).

Δ accuracy when rotating options · 0 = no change

Qwen A3 plus 6.2 percentage points with no confidence interval on this figure. DiffusionGemma Rot1 minus 7.4 with a CI95 from minus 11.9 to minus 3.1 (GT v4).

Effect of rotating the choice order per model Two independent rows. Qwen plus 6.2 pp without a CI. DiffusionGemma minus 7.4 pp with CI95 whiskers [−11.9, −3.1] (GT v4). 0 = same accuracy · pp Qwen FP8 inject (cell A3) +6.2 pp · no CI here DiffusionGemma Rot1 −7.4 CI95 [−11.9, −3.1] · GT v4

9. Fix expected in the adapter

With structured=true, the questions should be injected into the prompt when the server does not do it on its own. Evidence packaged in the upstream issue:

github.com/typesafe-ai/system-one-adapter-python/issues/50

← Back to the recommended circuit · Methodology: LLM adapter