TypeSafe shipped Jev this week, a model that returns typed decisions and cannot write a sentence. We use an LLM as the judge that decides whether a cheaper model held the business outcome, so we pointed Jev at that job and ran it against Claude Haiku 4.5, Sonnet 5 and Opus 5 on the same 212 cases, through the same code, at the same cutoff.
intervals overlap
opus-5 $4,589
10.7× tighter
What is Jev?
Jev is TypeSafe AI's first System One model. It takes a block of state plus a map of typed questions and returns typed answers in a single parallel pass. It cannot generate text. A noul question returns the probability that a yes/no statement is true, a choice returns one of your options with a probability distribution, and a score returns a position on ordered levels. Input costs $0.042 per million tokens and output is free.
Is Jev as accurate as Claude for eval judging?
On our benchmark, yes. Jev graded 209 of 212 cases correctly, 98.6%. The best Claude configuration we could build, Sonnet 5 with thinking enabled, graded 207, or 97.6%. The 95% Wilson intervals were [0.96, 1.00] and [0.95, 0.99], which overlap almost entirely, so the honest reading is a tie rather than a win.
How much cheaper is Jev than an LLM judge?
Between 16 and 164 times, depending on which model you were using. Per thousand judgements Jev cost $0.028, Haiku 4.5 cost $0.451, Sonnet 5 with thinking cost $1.601 and Opus 5 at default effort cost $4.589. For a team grading a million agent runs a month that is $28 against $4,589.
How much faster is Jev?
2.3 to 8.3 times at the median, and far more in the tail. Over pooled connections Jev ran 339 ms at p50 and 390 ms at p95, a spread of 51 ms. Sonnet 5 with thinking ran 1,590 ms to 4,179 ms. Jev's p95 is 1.15 times its median, where every Claude configuration measured between 1.33 and 3.00 times.
Why is a System One model faster than an LLM?
An LLM prefills the prompt, then decodes its answer one token at a time, and the number of tokens varies per call. That variation is the latency tail. Jev ingests the state once and emits every answer in one parallel pass, so there is no decode length to vary. A flat tail is the architectural signature of removing the decode loop.
Where does Jev fail?
On comparisons that span two places in the input. Its weakest result was catching em dashes, 14 of 17, a literal character check. It also missed a fieldKey mismatch 11 times out of 20 until the rubric named that field explicitly, after which it scored 20 of 20. The fix was a narrower question rather than a larger model. Notably, all three of its errors scored 0.510, 0.500 and 0.500 against a cutoff of 0.500, so it never made a confident mistake.
Can Jev decide which model to ship?
It can, but it should not. We gave it 120 switch decisions whose correct answer was computed by our own gates, and it agreed 98.3% of the time on clear margins and 91.7% on margins under 3%, with zero wrong switches. The reason to keep that decision in code is not capability, it is that the arithmetic is exact and free. Spend the model where there is a judgement to make.
Does swapping the judge change which model you ship?
Not on our workload. We replayed a real agent workload through two candidate models 144 times and pushed each judge's labels through our real parity, cost and latency gates. All three judges reached the same recommendation, and scored within noise of each other against objective truth: Jev 76.7%, Haiku 75.9%, Sonnet 73.3%. What the cheap judge changes is how many cases you can afford to put behind the decision.
How was this benchmarked?
212 cases were built by injecting eight named defects and four harmless changes into real production turns from our pilot, so the label is a fact rather than an opinion. A further 144 pairs were produced by replaying real prompts through two models with the request settings production uses. Every judge ran through the same valuemaxx eval code at the same 0.5 cutoff, over pooled connections.
The full comparison
| Judge | Accuracy | p50 | p95 | p95/p50 | Cost /1k |
|---|---|---|---|---|---|
| jev | 98.6% | 339 ms | 390 ms | 1.15 | $0.028 |
| sonnet-5, thinking | 97.6% | 1,590 ms | 4,179 ms | 2.63 | $1.601 |
| opus-5, default effort | 97.2% | 2,816 ms | 4,269 ms | 1.52 | $4.589 |
| sonnet-5, matched | 94.3% | 1,242 ms | 1,934 ms | 1.56 | $1.078 |
| opus-5, low effort | 93.9% | 2,136 ms | 3,185 ms | 1.49 | $3.037 |
| haiku-4.5 | 92.0% | 771 ms | 2,314 ms | 3.00 | $0.451 |
| opus-5, no thinking | 92.0% | 2,005 ms | 2,658 ms | 1.33 | $2.687 |
Why the architecture produces this
The judge asks one bounded question with a fixed answer set. There is no planning, no multi-step reasoning and nothing to write, which is the whole System One envelope, so a frontier model's extra capability sits idle. We were paying an autoregressive decoder to emit four characters. Price follows the same fact: output is free on Jev because there is barely any, and a model that never has to generate can be much smaller.
What this does not show
The 212 defects were injected rather than observed, because our pilot's own outcome labels would not support the claim: the outcome fired on 39 of 49 turns and the ones where it did not are no worse than the ones where it did. So we constructed truth instead of borrowing it. This measures whether a judge catches known breakage, on one workload from one agent, with no threshold tuning anywhere. Run it on your own traffic before believing it about yours.