Skip to content

Models

DecisionBench compares model behavior through explicit readout contracts:

Surface Probability source
Native decision-token model Masked softmax over valid decision tokens
Structured chat model Schema-constrained generated probability vector
Top-logprobs chat model Conditional next-token probabilities when all candidates are returned
Jev-compatible API Provider Noul, Choice, or Score distribution
Public candidate scorer Model-native logits mapped to benchmark candidates

The leaderboard never treats these as interchangeable metadata. Each result records its adapter, prompt, probability source, coverage, eligibility, and truncation contract.

Supported adapters and models

Check this table before adding an adapter. It lists the model-native contracts the runner already knows how to evaluate. A listed model should use its existing runner; an unlisted native contract is the reason to add a model adapter.

Runner Supported model or surface Native readout contract
run-hf Hanno-Labs/bosun-v3.1-0.6b, Hanno-Labs/bosun-v3.1-1.7b Masked softmax over valid decision tokens
run-nimble-hf bespokelabs/Bespoke-Nimble-9B Published candidate-token logits; up to 26 choices
run-public-hf --model-type cua-s1 cua-ai/cua-s1-4b-0.2 text adapter Published final-position option-letter logits; up to 26 candidates
run-public-hf --model-type gliner25 fastino/GLiNER2.5-Decide Exclusive classification logits over candidate labels, softmaxed as conditional option preference; rows over 512 encoded tokens or with ( in a candidate label are unsupported
run-public-hf --model-type mojev MoLeMo-Lab/mojev Published packed candidate logits; up to 255 candidates
run-public-hf --model-type nanojev C-Tianyu/NanoJev Published parallel candidate-path head
run-public-hf --model-type openjev com-kotobalabs/open-jev-deberta-v3-large Published grouped-span head
run-public-hf --model-type system-one pngwn/system-one-qwen3.5-4b-scorer Published candidate scorer
run-public-hf --model-type tev1 togethercomputer/Tev1-4B-experimental Softmax over A–X next-token logits; up to 24 choices
run-jev-openrouter typesafe/jev-1.13 OpenRouter Decisions API distributions
run-system-one-http juspay/xor Released Jev-compatible SystemOne API; forward/reverse option-letter logprobs with published calibration; up to 26 candidates
run-system-one-http (jobs/run_imajev_eval.sh) mohit67890/imajev-4b Open-weight Jev-compatible SystemOne server (Qwen3.5-4B + LoRA + trained 256-code decision readout); direct candidate scoring averaged over 4 option rotations, temperature-calibrated; up to 255 candidates, 65,536 input tokens
run-openrouter An OpenRouter chat model that can produce the required JSON-schema probability vector Structured probability vector
run-openrouter-top-logprobs An OpenRouter chat model that returns every required candidate in top-logprobs Conditional next-token probabilities

The local Hugging Face rows run pinned weights. MoJev's adapter pins model revision 0c8695b6252f4205907433d4e196a94f032e60c3 and packing code revision a74d58cd19ec573e83e8e27f9fecd837b8d830fb. The OpenRouter rows are hosted service surfaces: record the provider model name, request settings, and dated service snapshot when you submit results. decision-bench --help documents all runners; use the runner's --help for its required paths and model-specific options.

The registered Cua-S1 release is adapter revision 16818868b0cc7813808aae4e87b417657046ab79 on base revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a; its text adapter SHA-256 is 9b59c5aed96171a50b26526613766bbf44347a5c7af70f81efe6bcc6e9dbfb0e. DecisionBench's frozen rows contain textual state rather than screenshots, so this runner uses the release's text adapter.

The XOR runner pins model revision 679decd4c669e5c37f4ac29dbd9957997424c876 and serving bundle SHA-256 0a63473caaa3c6bfc8bc15fbab62f0a9a84c7ebf4ab6e06d0699891b7be6159b. The released bundle pins SGLang image digest sha256:6bcaa47db52f78ce0d67863b8b2431221b79bc23204a80cad757fa819d00e921. The runner sends the benchmark's Noul, Choice, and Score requests to /v1/systemone. Rows with more than 26 candidates, rendered state over 4 MiB, or serialized requests over 8 MiB remain explicit unsupported rows; the adapter never truncates them.

The imajev runner (jobs/run_imajev_eval.sh) pins adapter revision c9e5f132465da85d31735ec502d5557982671a7d of mohit67890/imajev-4b (LoRA weights SHA-256 88c2c44361e0c469352495abcfee789ff73a4deae0811168d9402cfc2b6e749c; every file is checked against the repository's SHA256SUMS) on base weights Qwen/Qwen3.5-4B at 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, and serves them with the model's own server, scripts/playground/server.py from github.com/mohit67890/imajev at commit a0134749e0900189c129cd6bb5000969f3b64bb5 (--backend torch --rotations 4 --calibration calibration-rot4.json --max-input-tokens 65536). The server scores every offered candidate in one forward pass through a trained readout (no generation), averages the four cyclic option orders and applies the shipped temperature, which leaves the argmax unchanged. Rows with more than 255 candidates remain explicit unsupported rows; at 65,536 input tokens every DecisionBench row fits. The run records adapter=imajev-serving-systemone-v1 and probability_source=trained_readout_4_rotation_mean_temperature_calibrated_v1 through the --adapter-name / --probability-source options.