When to Use Jev (and Other Decision Models)

You might've heard of Jev when it came out a few weeks ago. It advertised a different kind of model that provides decision and insight instead of chatbot text output. It sounded intriguing, so I signed up for early access to play around with it and see what kind of things a decision model might be useful for. After a preliminary exploration, we started kicking around ideas for how decision models could be used at Future Shock. Things that sprung to mind were evaluating relevancy of stories in our news pipeline, providing coding agents a second opinion when they wanted one, and some exploratory work on some forthcoming research. The news job was fairly straightforward since our pipeline already sorts incoming stories and keeps records we could compare a model against. The other two use cases were closer to curiosity, things requiring some exploration before deciding whether they were worth building into our system.

A decision model, at least the kind we were looking at, doesn't send you a written message as a response. You hand it a situation and a set of options, and it returns one of those options or a probability for each option, in a form a program can use without parsing a paragraph of prose. That kind of compact answer is appealing when you want to use a judgment to order a queue or choose what a program does next.

We compared TypeSafe’s Jev with four newer models (Microsoft Decision 1, Clef, Clef Flash and Perplexity Decider), using tasks with checkable answers and a historical sample of our news queue. Jev’s results combine saved responses, a partial fresh run of Jev 1.14 on the rule tests, and a separate comparison with our current news-scoring code. The full results are on our research page.

Jobs that need a ranking

We can only review so many stories each day, so a useful news ranker would put the important ones near the top of the queue. For that job, we care more about which stories it puts ahead of others than whether an individual story’s score is high enough to label it important.

We tested that on 220 stories from our pipeline’s history. Our earlier editorial and automated process had marked 79 as important, and we counted how many of those each ranking placed in the top 110. Perplexity put 56 there, compared with 49 for the ranking from saved Jev responses. Two importance scores already stored in our pipeline put 57 and 64 of those stories in the top 110.

Perplexity ranked these stories better than historical Jev, but the comparison gave us no reason to replace our existing ranking. The one-story difference between Perplexity and the stored score that found 57 important stories is too little to establish an advantage for either approach, or to show that they perform equally well. The test also had limits: our own process supplied the importance labels, and the stored score that found 64 may have included information the models weren’t given. This was an initial exploration of whether decision models might be useful for sorting our news queue.

In a follow-up comparison on October 17, we ran our current JavaScript scoring rules and Jev 1.14 on recovered inputs for 169 historical stories. With 85 review slots each, the rules found 50 of 59 stories labeled important, compared with 39 for Jev. We excluded 51 stories with multiple source receipts, which may have changed the mix of stories in the comparison. The labels came from earlier editorial decisions, not fresh independent judgments. These results support keeping our existing rules for news ranking, without establishing whether either approach would reduce review effort.

A model’s scores can help sort items even when they don’t work well for automatic yes-or-no decisions. In a separate test, we gave the models a rule and incomplete information, then asked whether the information was sufficient to determine an answer. Clef Flash was best in this group at ranking answerable cases above cases with insufficient information. But its scores stayed below the 0.5 cutoff we’d chosen in advance for all 120 answerable cases. Using that cutoff, it would classify every one of them as having insufficient information, despite ranking them relatively well.

Jev identified only 3 of the 120 answerable cases as having enough information, but never said the information was sufficient when it wasn’t. Its scores also ranked answerable cases above insufficient-information cases less reliably than any of the newer models’ scores did. Both Jev and Clef Flash rarely judged the information sufficient, but Clef Flash’s scores were more useful for sorting the cases. We didn’t test other cutoffs in this exploratory comparison, so these results don’t establish which cutoff would work best for either model.

Choosing what to do

For coding, we wanted to explore whether an agent could recognize when to ask a decision model for a second opinion, and whether that advice would help it avoid mistakes or choose a better approach. Testing that would mean giving the agent tasks with checkable outcomes, letting it decide when to consult the adviser, and comparing its results with and without that option. Our preliminary tests examined how models chose among supplied options, rather than whether their advice improved a coding agent’s results.

If an agent proposes several actions and none is appropriate, its adviser needs a way to say so. Our first menu test checked whether the four newer models could choose “none of these” when we left the correct answer out. All four did so every time, including when we reversed the options or gave them unfamiliar labels. These were simple answer-selection tasks, but they tested a useful prerequisite for an adviser: recognizing when none of the proposed options is correct.

There are also different reasons to reject the options. The adviser might have enough information to identify the right action, but find that the agent hasn’t proposed it. Or it might lack the information needed to choose an action at all. The first situation calls for revising the options; the second may require gathering more information before deciding.

Our second menu test gave the models separate options for “the answer is missing” and “there isn’t enough information to answer.” All four correctly identified insufficient information in all 12 cases. When the information was sufficient but the correct answer was missing from the list, Microsoft and Clef Flash chose the right option all 12 times, Perplexity 9 times and Clef 4. We also changed the examples and the other answer choices, so the errors can’t be attributed to adding a second way to decline. The first test’s perfect results didn’t carry over to every model when we tested these two reasons separately.

Another idea was to use decision models to choose actions for characters in a game we’re building. We’d want their choices to reflect their backgrounds, temperaments and past experiences while responding to the current board and other characters. The game’s code could determine which moves are legal, and the model could choose among them using that character’s history and circumstances. Evaluating that would mean testing whether characters behave consistently without repeating the same choices regardless of what happens around them.

Rules you could write down

Keeping the game’s rules in code would let us calculate legal moves exactly. Our larger rule test showed why we’d avoid relying on a model for that calculation. We supplied a row of 64 on-or-off cells and a table specifying each cell’s next value from its current value and those of its two neighbors. Every model correctly repeated values supplied in the prompt, but none consistently calculated the next values correctly. Across 529 cases, Perplexity answered 87 percent correctly, historical Jev 1.13 answered 71 percent, and Microsoft, Clef and Clef Flash answered between 54 and 58 percent. A fresh, partial run of Jev 1.14 scored 67 percent. Differences in the test setup mean that comparison doesn’t establish that the newer Jev version performed worse because of the update.

The four newer models all answered simple rule lookups correctly in our smaller tests, so the larger test’s failures don’t mean they can’t follow rules at all. But when a rule can be implemented exactly in code, we’d use that implementation rather than depend on a model to return the correct result. For the game, that would keep legal moves under the game code’s control while letting us explore whether a decision model can make suitable choices for each character.

What we'd measure next

These initial tests helped us identify what to explore next. For news, the comparison with our current scoring rules supports keeping them. A separate workflow test would be needed to determine whether a decision model could reduce review effort. For coding, we’d test whether an agent chooses useful moments to ask for advice and makes fewer mistakes with that advice available. For the game, we’d evaluate whether characters’ choices reflect their histories and temperaments while adapting to what happens around them.

The full tests, results and limitations are on our decision-model research page.