拓展阅读 / JevBench：独立评测基准

知识库收录材料 · 原始文件 ↗ · 来源、版本与许可

v1.2.8 pre-registration (add-requests run 4): mappings, endpoint conditions and cost bases

Written and pushed before any of the runs below started (21 Sep 2026). Same frozen v1.2 set (534 decisions: easy, standard, judge, hard), same scorer (jevbench/composite_v12.py), one request at a time. Nothing here changes an earlier row. The decider-35b-a3b and reflex-27b rows measured earlier the same night are described in docs/v1.2-additions-run4.md.

Mappings

All seven serve TypeSafe's POST /v1/systemone (or, for GLiNER2 large, use the documented GLiNER2 mapping), so the existing typesafe adapter runs unchanged: noul → {"yes": p, "no": 1 - p}, choice/score → the returned probabilities. Nothing is re-normalised or invented on our side. A system that returns no distribution for a question is scored label-only for that question, the same way as for everyone.

Endpoint conditions and latency

• decision-machine-1: a production API → measured latency from Sandy (Germany), network included, no adjustment. The test key has the same inference infrastructure as a paid key, per the provider; only rate limits differ.

• jqv, decider-2b, reflex-4b, opendecision: our GPU, reached over the internet from Sandy → the standing self-hosted adjustment ×2 + 0.15 s, raw p50/p95 kept in the row. The pod contains no keys or tokens.

• gliner2-large, werr: our CPU (4 threads, Ryzen 5 3600) → ×2 + 0.15 s.

Cost (never an automatic 100)

• decision-machine-1: the public tariff, $0.04 per million input tokens, output free (https://docs.milliseconds.ai/reference/pricing, read 2026-09-21), × the input tokens the API reports per request. The free test key was used; the price is the paid one.

• jqv: OpenRouter qwen/qwen3-32b input tariff, $0.08/M, as in v1.2.7 (the exact base model; nothing is generated).

• reflex-4b: DeepInfra Qwen3.5-4B, $0.03/M input (the exact base weights; one pass, no generated output), as kev 4B.

• decider-2b (1.9B): the same $0.03/M 4B-class price; no hosted ~2B Qwen3.5 is listed, so this errs high.

• opendecision, gliner2-large: as pre-registered (large-encoder class, $0.01/M).

• werr: it has no weights; the smallest size class in our table, base-size encoders at $0.005/M input, is used so the row is not free by construction.

Held-out items

Held-out items go only to our own machines (Sandy, our RunPod pod) or to a production API (milliseconds.ai). None goes to a submitter-operated endpoint. jqv's earlier partial row is replaced by the full run on our GPU if that run completes; otherwise the partial row stays as it is.

Contributor code

Contributor code (jqv, decider, reflex, OpenDecision, werr) runs only inside a disposable RunPod container or a local container with no network, no host home, no Docker socket and an empty environment. It is never installed on the host.

Addendum (pushed before these runs started, 21 Sep 2026, 04:07 UTC)

• jev-local scores each option by its mean log-probability under a frozen Qwen3.5-9B (one forward pass per option, no generation). Cost: OpenRouter qwen/qwen3.5-9b input tariff, $0.10/M, the exact base weights, × the input tokens the server reports (it re-reads the state once per option, which a token count may not include; stated in the row).

• nimble-9b: same cost basis as its existing row. If the re-run completes on all 534 decisions it replaces the existing Nimble row (the earlier run is kept in the repo history and named in the revision log); otherwise the existing row stays.

Not measurable this round

• Werr (issue #10): the documented server (python -m werr.server) imports scratch.jevbench_eval.optimize_werr_jevbench, which is not in the public repository, so the submitted configuration cannot be started. The engine also sends telemetry about each request to an outside server by default. Not run.

• DIY Jev (issue #7): github.com/VakeDomen/DIY-Jev answers 404 (checked 21 Sep 2026, 04:06 UTC). Not run.

Addendum 2 (pushed before the LitJev run started)

• Mapping: native POST /v1/systemone, the unchanged typesafe adapter, as above.

• Cost: the exact base model's hosted input tariff, $0.214/M input (the price the reflex-27b row uses for Qwen3.8-27B), × the input tokens the server reports; nothing is generated.

• The only change to the author's code path: its CLI binds to 127.0.0.1, so it is reached through an SSH tunnel from Sandy to the pod (the request still crosses the internet; the same ×2 + 0.15 s rule applies).

decider-2b ran with DECIDER_COMPILE=0, the author's documented fast-start mode: the default compiled start had not finished after ten minutes (during which the pod's disk briefly filled up). The same mode was used for decider-35b-a3b.

Post-run notes (written after the runs, 21 Sep 2026 ~06:15 UTC)

• Where the GPUs were. Both RunPod pods of this round (the H100 NVL and the A40 for Nimble) were in Canada; the pod the earlier decider-35b-a3b / reflex-27b runs used was in the same datacenter. Latency is measured from our server in Germany, so every GPU row of v1.2.8 includes a transatlantic network path in its raw latency (the easiest items take about 0.33 s round trip). Rows from earlier rounds used pods in Europe. The adjustment rule is unchanged.

• reflex 4B — the first attempt was discarded before completion: decider-2b's startup compile was still running on the same pod and slowed it. Kept in the job's runs/_failed/ with the reason; the published run had the GPU to itself.

• jev-local — the first attempt failed to load (its [hf] extra does not install accelerate, which its own loader needs); accelerate was added to the server's environment and the run repeated. Its API reports far fewer input tokens than the prompt has, so it is priced on the token counts of the reference run with the same prompts.

• decision-machine-1 — class decision-api ("Closed decision model (API only, not Jev)"): closed weights, so it is not shown among the open Jev rebuilds.

• Bespoke Nimble 9B — the re-run completed (534/534) and replaces the v1.1.3 row. Hard tier 43.6 % → 65.5 %, but the score fell (63.7 → 61.8): the long items that used to fail are now answered and priced, and the pod was in Canada instead of Sweden. The old row's score and tiers are kept in the artifact under superseded_rows.

• Timing — the decider-2b start-mode line in addendum 2 (f8ce713, 04:43:20 UTC) was pushed 45 seconds after decider-2b's run began (04:42:35). It records a server start mode, not a mapping or a price. Everything else in the addenda was pushed before the runs it covers.