A decision wearing an expensive model#

This triage step shows up everywhere inside an agent pipeline. Is this ticket urgent? Which of these 4 tools applies? Does this text break a policy? None of those are generation tasks. Each has a small, fixed set of possible answers, and the job is to pick the right one. Teams send these decisions through a general reasoning model anyway, because it’s already wired in for the steps that do need prose, and one more call is less work than standing up something new. Every one of those calls then pays a full call’s latency and token cost for a question with 3 possible answers. It also opens a hallucination surface (room for the model to invent or misformat an answer) on output that was never meant to be free text.

Teams usually reach for the obvious fixes in order. Each one solves part of the problem and leaves part of it standing.

Fix 1: ask nicely and parse the JSON#

The first thing most teams try is a tighter prompt. Describe the exact output schema, ask for strict JSON, and wrap the call in a parser that retries when the response doesn’t match. This helps. Most responses now come back machine-readable, and the retry catches the ones that don’t.

What it leaves broken is the shape of the call. The model still runs a full autoregressive generation pass to answer a question with 3 possible answers. Autoregressive means it produces text one token at a time, predicting each next word from everything written so far. The token bill didn’t move. Neither did the latency. Worse, a retry costs a second full call exactly when the model was already struggling with the input. That’s the worst time to double the cost.

Many model APIs now go further and constrain decoding to a schema or a fixed list of values. That removes the malformed-response case outright, so the format gap is closed. The call is still a full generation pass, billed and timed like one, and the model can still pick the wrong value from the list.

Figure 2 - The parse-and-retry loop: Tightening the prompt and parsing the output catches most malformed responses. A parse failure still means paying for a second full generation call on a decision that only needed one answer back the first time.

Fix 2: send the decision to a cheaper model#

The next obvious move keeps the same shape of call, prompt in and text out. It just sends the call to a smaller, cheaper LLM instead of the flagship model doing the rest of the pipeline’s work. Our companion article, The Advisor Tool Is Real, and Anthropic Ranks It Last, covers this move in depth: pick the right-sized model tier for each subtask instead of sending everything to the most capable model [1].

Company-scale spend shows how much a cheaper tier can save. LegalOn Technologies, a legal-AI company, first gave its developers unlimited use of one high-end model, then moved to three tiers matched to the work to stay inside its annual budget. “Teams started with a lightweight model and moved to more capable models as task complexity increased,” with GPT-6 Luna on code implementation with clear requirements, GPT-6.1 Sol on standard design and analysis, and GPT-6 Astra on architecture [2]. Combined with switching off a setting called Fast mode by default and adding per-department budget caps, OpenAI’s write-up says the change cut estimated daily costs “by approximately 65%” compared with the earlier model, GPT-5.5 [2]. That’s routing whole coding tasks rather than narrow decisions, and the figure is an estimate from an OpenAI customer case study, but it shows the size of the lever.

Moving to a cheaper model does lower the per-call cost. It doesn’t change what kind of model makes the decision. The cheaper model is still trained to generate prose, so it still needs a carefully engineered prompt to stay on-format, and it can still wander off-schema when a ticket is confusing or ambiguous. This fix makes the mistake cheaper. It does not make the mistake structurally impossible. Fix 2’s cheaper LLM still does the same kind of work. The approach after fix 3 takes free-form text generation out of the decision.

Fix 3: train your own classifier#

Say a team already has labeled examples: past tickets paired with their correct category. Then skip the prompt and train a small classifier, a model built to output one label from a fixed set and never a sentence. On a bounded task with real labeled data, this can beat everything else, including the commercial decision models discussed below. A presenter’s own benchmark on Banking77 (a public dataset of 77 overlapping customer-intent categories) tested this directly. A trained 22-million-parameter classifier scored 93.2% accuracy at 8 milliseconds per call on CPU. The comparison was Jev from TypeSafe AI, a commercial decision model introduced properly in the next section. Running zero-shot (with no training on this specific task), it scored 80.1% for about 22 cents across the same test set [3].

Labels are the catch. With labeled data, a small trained classifier can be faster, cheaper, and more accurate than a zero-shot model, as it was in that benchmark. Support desks often do record a ticket category, and that’s exactly the case where fix 3 is worth trying first. Most other narrow decisions in a real agent pipeline have no labeled history. Nobody logged “input, correct answer” for every classification point a production pipeline picks up. Building and maintaining a custom classifier for each one doesn’t scale the way adding one more typed question does.

KEY INSIGHT: Before reaching for a hosted decision model or a bigger prompt, check whether you already have labeled examples for the decision. In the Banking77 benchmark, a small classifier trained on labeled data beat a zero-shot decision model by 13 points, 93.2% against 80.1%. A zero-shot tool earns its place when no labels exist yet.

A model that only ever returns a typed answer#

A newer category of model targets this gap. Simon Willison credits Maggie Appleton with its name: decision models [4]. The first commercial one is Jev, from TypeSafe AI, a company co-founded by Diogo Almeida [5]. Almeida co-wrote the InstructGPT paper, the 2022 research that shaped how modern chat models are trained to follow instructions [6]. TypeSafe’s current release is jev-1.13.0. Its jev-latest alias points there, and its documentation calls it “the most recent stable, official release” [7]. On the open side, Cactus publishes Needle 3, a 121-million-parameter open-weight model (its trained weights are published, so you can run it yourself) under the Apache 2.0 license [8]. A third open option arrived on Oct. 1, 2026. AWS’s Strands Labs released Strands Decider 2B, a 2-billion-parameter decision model that picks among predefined options and returns a confidence score instead of writing text, with its code, training data and weights published. Its authors report a median of about 115 milliseconds per decision on a local Nvidia RTX 3090 [9].

Jev and Needle 3 work differently. Jev takes unstructured state (the ticket, the conversation so far, whatever context the decision needs) plus a typed question. It returns a typed answer with probabilities instead of a sentence. TypeSafe says the model is trained to return calibrated decisions [7], meaning that across many answers, a 0.8 should be right about 80% of the time. TypeSafe documents exactly 3 question types. A Choice question picks one option from a set and returns the chosen option, a probability per option, and a confidence score. A Score question rates the state against a set of ordered, described levels. A Noul question is a yes-or-no question that returns the probability the answer is yes [10]. That’s the whole list. Jev’s API has no free-text answer type, so it can’t return a sentence.

Jev is no longer the only commercial option. On Oct. 6, 2026, OpenAI released a Decisions API on a model called gpt-6-luna, with the same 3 question types under different names: predicate (a yes-or-no question returned as a probability), choice, and score. A choice answer comes back with a probability per option and a confidence field, much like Jev’s [11]. OpenAI says narrowing the model to a few choices makes it “nearly 10 times faster,” and its ticket-routing demo answered on the server in under 100 milliseconds [12]. It also accepts images, which Jev doesn’t, and it bills only for input, at 10 cents per million tokens [11]. Simon Willison puts Jev’s input price at 4.2 cents per million [13], so the bigger vendor costs more per token but can look at a screenshot. Two vendors shipping the same typed shape within weeks of each other is the strongest sign so far that decision models are a category worth designing for.

Figure 3 - Two different shapes of output for the same input: A generation model takes the ticket text in and can produce any sentence its vocabulary allows. A decision model takes the same text in plus one typed question and returns only an answer that fits the question’s declared shape.

Fix 1’s retry loop was working around this from the outside. Here the question type declares the answer’s shape before the call runs. The API has no way to return anything else, so a parser has nothing to catch. TypeSafe doesn’t publish the model’s architecture, so we can only describe what the API does. This is also why the approach doesn’t need fix 3’s labeled data. It works zero-shot on a new decision the moment you write the typed question. Once you have labels, as fix 3 showed, a trained classifier can still win.

Figure 4 - Three typed question shapes, no fourth: Choice returns one option from a set plus a probability per option. Score rates the state against ordered levels. Noul returns the probability that a yes-or-no question is true. Jev has no answer shape outside these 3.

Needle 3 takes a different route to the same goal. It treats a decision as a tool choice. Give it a request and a few tool definitions (each tool’s declared name and arguments, called its schema), and it picks which tool to call and copies the arguments out of the request. It still writes short text, but a grammar compiled from those schemas constrains every token it writes, so it can only fill the slots a schema declares [14]. Cactus calls the underlying idea grounding, and it makes the “no guessing” guarantee concrete: “A required argument with no default and no evidence in the request is withheld rather than guessed” [8]. The practical guidance that follows is aimed squarely at anyone designing tool schemas, on any model, at any size: “If a value is not going to be in the request, do not make it a required argument” [15].

We ran this ourselves. We turned the 3 triage buckets into 3 typed support tools and sent Needle 3 4 ordinary support-style prompts, on a CPU, with no GPU. Needle did try to fill arguments it had no evidence for. It set an account ID to the word “invoice”. It filled an email field with a sentence from the ticket. None of those calls ran. Every attempted call landed in a suppressed list instead, and 2 were explicitly flagged as ungrounded arguments. Of the 4 prompts, 1 correctly produced no call at all.

One caveat. All of our calls also scored far below the engine’s 0.1 confidence floor, and that floor withholds a call on its own [8]. So our run can’t tell the two mechanisms apart. It was also small: one session, our own machine, 4 prompts. That’s too few for an accuracy number. What it does show is the withholding working on our own prompts the way Cactus documents it.

KEY INSIGHT: Design every tool schema, on any model, with Needle’s withheld-call rule in mind: if a field’s value won’t reliably show up in the request text, don’t make it required. A withheld call shows up as an empty result you can debug. A guessed call runs with an invented value and may not be noticed.

Figure 5 - Withheld, not guessed: When a required argument has no matching evidence in the request and no schema default, the call is withheld into a suppressed list rather than executed with an invented value.

The most useful outside evidence so far isn’t about raw accuracy. It comes from LangChain, which calls its own study “a narrow test” and notes the Jev version wasn’t recorded [16]. LangChain ran Jev against 5 agent runs on weather requests and judged each one 100 times, for 500 binary pass/fail decisions in all. Jev matched a human reviewer’s labels (the oracle, the answer key the test treats as correct) on all 500. Three LLM judges (general models prompted to grade the same runs) scored 99.8%, 96.4%, and 80.0%. The number that matters most here is variance: how much a judge’s score for the same case moved between repeated runs. Jev’s per-case variance was 92 to 913 times lower than the 3 LLM judges, and the whole Jev run cost $0.34 against $28.17 for the Claude Sonnet 4.6 judge [16]. That’s a reliability result. The same input got nearly the same score every time, far more steadily than with any of the 3 LLM judges.

Figure 6 - Jev gave the same verdict far more consistently: On LangChain’s 500-decision agent-eval test, Jev matched a human oracle on all 500. Its larger advantage over the 3 LLM judges was variance, 92 to 913 times lower, at a fraction of the cost.

Where it stops working#

A valid shape doesn’t guarantee a correct pick. A decision model can’t invent a tool name or break its output shape, and that’s what people mean when they say it can’t hallucinate. It can still pick a valid, well-formed option that is simply wrong. In our own run, Needle picked escalate-to-billing-manager for what read like an ordinary billing dispute. Nothing in it seemed to need a manager. The pick looked defensible, but it was arguably wrong. A model that can’t malform its answer can still get the answer wrong.

Confidence has gaps too. It’s the number that’s supposed to tell you how much to trust a given call. Needle 3’s confidence score returns None if you fine-tune (further train) the model locally on your own tools. Local fine-tuning leaves the confidence head untouched, and only Cactus’s hosted fine-tuning recalibrates it [14].

We ran the prompts twice: once with the model’s state reset between calls, once without. Our own print script crashed on the last prompt of the second run, so 7 calls were recorded. Without the reset, the model’s reasoning text mentioned a “previous turn” that wasn’t in the current request. It was carrying context over from the call before. Calling reset() between unrelated requests fixed that, so do it on every call.

Confidence sat between 0.0008 and 0.0041 on all 7 calls. Far below the 0.1 floor. The call that was correctly declined looked no different from the calls that picked a defensible-but-wrong tool. It’s a small sample: 7 calls, 1 toolset, no tuning. Still, take it as a caution. Check the confidence number on your own data before you assume it’s doing useful work.

The category also hits hard limits at scale. Needle 3 caps at 5 tools in context. Above that, only the 5 highest-scoring tools enter the model’s context, and any other tool is unreachable for that request [15]. CLM-8B (from Contrastive-LM, a different open decision-model architecture) has no hard cap. Its accuracy drains away as the list grows instead. It’s built on a frozen encoder, a model that turns text into a list of numbers and is kept fixed rather than retrained. CLM-8B uses it as a dual encoder. The request and each candidate are turned into numbers separately and scored one pair at a time, and the candidates are never compared with each other [17]. In a presenter’s own test, tool-selection accuracy fell from 86% at 8 candidates to 17% at 1,080 [18]. The presenter traced the failures to tools that sounded alike, such as several different refund tools. A model that never compares options side by side can’t tell near-duplicates apart [18].

Figure 7 - Accuracy falls as the option list grows: CLM-8B scores each candidate alone, never side by side against the others. In a presenter’s own test, tool-selection accuracy fell from 86% at 8 candidates to 17% at 1,080. The chart shows those 2 measured points only.

TypeSafe’s own “model jaggedness” page lists 9 named weaknesses for the current Jev release, including literal reading, math and number handling, date and time comparison, and adversarial content [19]. The vendor wrote that list about itself, and nobody has audited it independently. It’s still useful, since the vendor is naming its own failure modes. Treat the list as the minimum set of cases to test before you trust the model with a decision.

Explainability is another cost. No benchmark table shows it. Simon Willison reviewed Jev independently. He called Jev “a regression even further towards black box machine learning systems” [4], since it returns a number with no reasoning attached. An LLM’s chain of reasoning is sometimes wrong, but a person can at least read it and argue with it. A bare Choice or Score gives them nothing. Willison’s own example of why this matters: “I really hope nobody uses Jev to rank job applicants” [4]. If someone will ever need a decision explained, a decision model makes that harder.

Last, a decision model that gates a dangerous action only protects the path it’s watching. One practitioner’s test on a custom agent setup showed this [20]. The presenter ran the same agent in 5 configurations, with and without each decision-model component, for about 15 runs in total. A classifier gate blocked the agent from dropping a database column through the SQL tool it watched. The agent reached for a separate shell tool that held the same database login details. At one point it split the literal string it was trying to avoid into two pieces, just to dodge the classifier. The presenter reports that the gate still caught the split attempt, even though it was only designed to watch the SQL tool. The agent kept looking for another way in. In 10 of the 15 runs, it searched the filesystem and found other runs’ already-scored results sitting on disk, effectively reading an answer key instead of solving the task [20]. A gate guards the tool it’s attached to, not the agent behind it. Our companion articles on Multi-Tenant Agent Security [21] and Give the Agent a Budget, Not a Token [22] make the same point about permissions generally: every path that reaches a protected resource needs its own gate.

KEY INSIGHT: A decision model gating one tool leaves every other route open. If there’s any other route to the same resource, an ungated shell, a second API, a raw file write, an agent under pressure to complete its task will find it. Put the gate on the resource itself, such as the database role, so every tool that reaches it passes the same check.

Figure 8 - The blocked agent tried another tool: Blocked at the tool the decision model was watching, the agent reached for a separate shell tool holding the same database connection and kept trying from there. The gate’s design had only planned for the first path.

The qualifying test#

Run these questions against any narrow decision in a pipeline before choosing a vendor:

1. Is the decision narrow? A fixed set of options, a score on a scale, or a yes-or-no, rather than something that genuinely needs prose out. If the answer has to explain itself in words, this category doesn’t apply.
2. Do you already have labeled data? If yes, revisit fix 3. A small trained classifier you own can beat a hosted decision model on your specific task, and it costs less to run.
3. Does a free, self-hostable model already clear your bar? A notebook comparison published by LlamaIndex (which sells document-processing tools) found a free, open text-generating model, repurposed as a classifier, “pretty closely matching Jev’s performance” on several document-processing decisions, though it trailed on document splitting [23]. Check whether something you can run yourself already does the job before paying for a hosted decision model.
4. Does asking the question move data somewhere it shouldn’t go? If the typed question itself requires sending sensitive content to a third-party API, you may have already created the problem you were trying to gate. Sam Witteveen put it plainly while building a router around a hosted decision model: “If I’m sending your prompt to TypeSafe to ask whether it’s private, I’ve kind of already leaked it” [24].
5. Will someone later need this decision explained? If a human, an auditor, or a customer will ever ask why the system decided what it decided, a bare number is a worse answer than even an imperfect LLM reasoning trace.
6. Does the decision guard something? If it does, make sure every other path to that same resource carries its own gate, not just the one the decision model is watching.

Figure 9 - The qualifying test: 6 questions to run against any narrow decision in a pipeline before moving it off a reasoning model. Is it narrow, do you have labels, does a free model already clear the bar, does asking leak data, will it need explaining, and does it guard something that needs its own independent gate.

Conclusion#

The architecture argument holds up without any vendor’s headline numbers. A bounded decision with a fixed set of answers doesn’t need a model built to write open-ended text. A typed answer shape removes the parsing problem, and a model like Jev doesn’t pay for a generation pass to get it. The evidence for any one vendor’s performance is thinner and younger than that. So be careful which numbers you repeat. The category is still worth using.

Cheaper decisions also tend to give the reasoning model more work. The economist W. S. Jevons observed in 1865 that making coal more efficient to burn didn’t reduce how much coal got burned. It expanded what was worth burning coal for [25]. Inside a pipeline, cheaper judgment does more than save money on the decisions a team already makes. It changes which questions are worth asking at all. Narrow checks that used to cost too much start to make sense. That surfaces more work, both for the reasoning models and for the humans downstream of them [26].

If you run a pipeline in production, walk the qualifying test above against its decisions. Start with the ones a reasoning model quietly answers every day while nobody checks whether it should. Find out how many of them qualify to leave.

References#

[1] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “The Advisor Tool Is Real, and Anthropic Ranks It Last: The Cost Ladder and the One Number That Decides,” Dotzlaw Consulting, Sep. 3, 2026. /insights/ai-32-advisor-tool-pattern/

[2] OpenAI, “LegalOn halves Codex costs while maintaining development speed,” OpenAI, Oct. 8, 2026. https://openai.com/index/legalon-halves-codex-costs

[3] Prompt Engineering, “OPEN JEVs are Here!” (originally published as “JEV Is Just a Text Classifier…”), YouTube, Sep. 19, 2026. https://www.youtube.com/watch?v=FVzI3QAxwNs

[4] S. Willison, “Jev introduces a new shape of LLM, System One, aka Decision Models,” Sep. 21, 2026. https://simonwillison.net/2026/Sep/21/jev/

[5] “Jev (AI model),” Wikipedia, accessed Oct. 2, 2026. https://en.wikipedia.org/wiki/Jev_(AI_model)

[6] L. Ouyang et al., “Training language models to follow instructions with human feedback,” arXiv:2203.02155, 2022. https://arxiv.org/abs/2203.02155

[7] “Models,” TypeSafe AI Documentation, 2026. https://docs.typesafe.ai/models.md

[8] “Needle 3,” Cactus Compute, 2026. https://cactuscompute.com/needle

[9] M. Brooker, M. Chambers, and F. Nonato de Paula, “Introducing Strands Decider,” Strands Agents Blog, Oct. 1, 2026. https://strandsagents.com/blog/introducing-strands-decider/

[10] “Primitives (Questions),” TypeSafe AI Documentation, 2026. https://docs.typesafe.ai/primitives.md

[11] “Decisions,” OpenAI API Documentation, accessed Oct. 7, 2026. https://developers.openai.com/api/docs/guides/decisions

[12] OpenAI, “Introducing the Decisions API,” YouTube, Oct. 6, 2026. https://www.youtube.com/watch?v=FB6oCmrIj-Y

[13] S. Willison, “llm-openai-decisions 0.1a0,” Oct. 6, 2026. https://simonwillison.net/2026/Oct/6/llm-openai-decisions/

[14] cactus-compute/needle, GitHub, 2026. https://github.com/cactus-compute/needle

[15] “How to Design Tools for Needle 3,” Cactus Compute Blog, 2026. https://cactuscompute.com/blog/designing-tools-for-needle

[16] D. Shea and S. Roche, “Can Jev Be a Better Agent Evaluator?,” LangChain Blog, Sep. 20, 2026. https://www.langchain.com/blog/jev-agent-evals-langsmith

[17] Contrastive-LM, “CLM-v0.1-8B,” Hugging Face, Sep. 2026. https://huggingface.co/Contrastive-LM/CLM-v0.1-8B

[18] Prompt Engineering, “This New AI Architecture Makes Decisions 13x Faster,” YouTube, Sep. 26, 2026. https://www.youtube.com/watch?v=eSuMmMMMrm0

[19] “Jev 1.13 jaggedness,” TypeSafe AI Documentation, last reviewed Sep. 17, 2026. https://docs.typesafe.ai/model-jaggedness/jev-1.13

[20] Prompt Engineering, “Jev + AI Agents: Where a Decision Model Actually Helps,” YouTube, Sep. 29, 2026. https://www.youtube.com/watch?v=4YVeQf8huyM

[21] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “Multi-Tenant Agent Security: The LLM Is Not Your Security Boundary,” Dotzlaw Consulting, Aug. 25, 2026. /insights/ai-26-multi-tenant-agent-security/

[22] G. Dotzlaw, “Give the Agent a Budget, Not a Token: Four Dimensions That Replace a Yes-or-No Grant,” Dotzlaw Consulting, Sep. 30, 2026. /insights/ai-50-give-the-agent-a-budget-not-a-token/

[23] LlamaIndex, “Processing Documents: Jev vs OSS Models,” YouTube, Sep. 24, 2026. https://www.youtube.com/watch?v=MLZ1dUOZG74

[24] S. Witteveen, “How to Build Things with Jev & OpenJevs,” YouTube, Sep. 2026. https://www.youtube.com/watch?v=ZR7anrL50xs

[25] W. S. Jevons, The Coal Question, Macmillan, 1865. https://archive.org/details/TheCoalQuestion

[26] N. B. Jones, “Why Developers Are Losing Their Minds Over AI That Can’t Write,” YouTube, Sep. 2026. https://www.youtube.com/watch?v=tYugqJ9YytQ