Jev Benchmarks Explained: What 193x Faster Measures

Jev launched with big numbers: up to 200 times faster, up to 444.6 times cheaper, “can’t hallucinate”. This page separates what TypeSafe measured, how it measured it, and what it says about its own method. It does not report our own accuracy tests.

The claims

Claim	Where it comes from
70 to 500 ms per call	TypeSafe’s end-to-end timings, run from its West Coast offices close to its servers
Up to 193.6x faster, 444.6x cheaper	Four “workflow evaluations” published with the launch
Similar intelligence to LLMs on System One tasks	The same workflow evaluations, against averaged answers from top LLMs
No type errors	A property of the interface: answers can only be one of the declared options
Output tokens free, input $0.042 per million	Published pricing

How the workflow evaluations work

TypeSafe built a new kind of test rather than using public benchmarks:

1. Each workflow is a fixed program (“compute graph”) that breaks a business task into many small questions, the way you would use Jev in production.
2. Every model receives the same workflow and questions. The harness is not tuned per model.
3. There is no human-labeled ground truth. Instead, the reference answer is the average of two frontier LLMs (GPT-6 Astra and Fable 5.1), and each model is scored on how closely it matches.
4. LLMs answer through TypeSafe’s own wrapper that constrains them to structured decisions, which TypeSafe says is the most accurate way to get decisions from an LLM but also slower and more expensive than asking for a bare answer.

On those workflows, TypeSafe reports that Jev sits on the cost-quality frontier by almost two orders of magnitude.

The caveats TypeSafe lists itself

To its credit, the launch post spells out the weaknesses of its own evaluation:

• The multiples “are on the higher end” of what to expect in real-world use.
• The workflows were written by TypeSafe’s model capabilities team, so some bias could exist, although TypeSafe says they were not chosen to flatter Jev and are outside its training data.
• Using OpenAI and Anthropic models as the reference biases scores toward those models’ answers.
• Timings were taken from the West Coast of the US, where the service runs; your latency will include your network distance.
• TypeSafe cannot prove its pricing is not subsidized.
• The hallucination comparison uses LLM figures gathered through OpenRouter, while Jev’s 0% for type errors is not measured but guaranteed by the schema.

What critics pointed out

On Hacker News, several objections came up repeatedly:

• “Can’t hallucinate” is a narrow claim. Jev cannot return an invalid option, but it can still return the wrong option with high confidence. See Can Jev hallucinate?
• Comparing against the average of LLMs is circular. If the smartest models are wrong, matching them is not a win. Why not compare against a correct answer?
• No public benchmark scores. Several people asked how Jev does on standard classification benchmarks. As of September 19, 2026, we have not found a published table of Jev’s scores on standard public benchmarks.

Independent data points so far

There is no independent, controlled benchmark of Jev yet. The public data points are anecdotal but consistent in direction:

• Vercel reported five to 18 times faster results after replacing an OpenAI model with Jev in a safety classifier, with better accuracy (via TechCrunch).
• Bryo AI found Gemini slightly more accurate on business-email classification but 10 to 20 times more expensive (via TechCrunch).
• The typesafe-computer-use project measured about 155 times lower cost and 14 to 40 times lower model latency per agent decision against Claude Opus 5.

In our own timing tests on TypeSafe’s API from East Asia, small requests took 253 to 378 ms (median 284 ms), within TypeSafe’s published range despite the longer network path.

How to benchmark Jev on your own task

TypeSafe’s published multiples will not tell you whether Jev works for your data. A small test will:

1. Collect 50 to 200 real inputs and label the correct answer yourself.
2. Write each decision as a Noul, Choice or Score, and fix the questions and threshold before looking at results.
3. Run the same questions through Jev and through the LLM you use today. TypeSafe’s open-source system-one-adapter makes the LLM side a one-line change.
4. Compare accuracy, the share of cases each flags as uncertain, cost per 1,000 decisions and latency.
5. Look at the errors, not just the score. Jev’s failures cluster in the patterns TypeSafe documents (math, dates, indirection, noisy states), which you can often move into code.

Pin jev-1.13.0 while testing so a model update does not change results midway. See Jev limitations and Jev vs LLMs.