Views 2, 3 and 4 · per question

What each model gets right, question by question

A set's total mixes very different questions. Here they are split: accuracy per question, which department each model sends cases that belonged elsewhere, and whether its probabilities match what happens.

Accuracy per question

Percentage of points per question (on urgency and relevance, being off by one level is worth half a point). The last row is the baseline of always answering the most frequent: a cell below it adds nothing.

Set

Loading…

Department confusion matrix

Rows: correct department. Columns: what the model answered. The diagonal (outlined) is the hits. Shading is the row percentage.

Model

Set

Calibration of the yes/no questions

The answers are grouped by the probability the model gave (0–10 %, 10–20 %…) and we look at how often the correct answer was "yes". A well-calibrated model falls on the diagonal: when it says 80 %, it is right 8 out of 10 times. Hollow points have fewer than 5 answers.

Set