拓展阅读 / JevBench：独立评测基准

知识库收录材料 · 原始文件 ↗ · 来源、版本与许可

v1.2.7 additions — mappings, endpoint conditions and cost bases, written before the runs

Same rules as docs/v1.2-additions.md (v1.2.2). The frozen v1.2 items and jevbench/composite_v12.py are unchanged. Each system answers all 534 decisions (72 easy, 242 standard/judge, 220 hard) one request at a time. This file was committed before any row was aggregated and before the published GLiNER2.5 runs started, so no mapping or price was chosen after seeing a score. jqv's run had begun about ten minutes earlier (its endpoint was a temporary tunnel); it needs no mapping, and its price is the exact base model's public tariff, fixed here.

The systems

Not in v1.2.7: gliner2-large and opendecision are registered here but were not run in this revision. GLiNER2 large was left for the next round for time; OpenDecision needs over two minutes per hard-tier decision on our CPU and so needs a GPU round. Their mapping and price stay as written here for when they run.

Mappings

• jqv and OpenDecision serve TypeSafe's POST /v1/systemone natively, so the existing typesafe adapter runs unchanged: noul → {"yes": p, "no": 1 - p}, choice/score → the returned probabilities. Nothing is re-normalised or invented on our side.

• GLiNER2.5 small and multi and GLiNER2 large use the same mapping as the GLiNER2 row, written down in docs/v1.2-additions.md before that run: the question goes in front of the state, options become labels with their rubric text, and the distribution is GLiNER2's own single-label softmax read out in full (class_act="softmax", multi_label=True, cls_threshold=0.0). Checked on both checkpoints before these runs: the top softmax probability equals the default single-label confidence to every digit (small: 0.7771874070167542; multi: 0.656845211982727), so the distribution is the model's own, not a normalisation we chose.

Endpoint conditions and the speed adjustment

• gliner2.5-small, gliner2.5-multi, gliner2-large, opendecision: our CPU (4 threads, Ryzen 5 3600, Sandy), weights loaded before the clock → the standing self-hosted adjustment ×2 + 0.15 s. Raw p50/p95 stay in the row and the repo.

• jqv: the submitter's own machine (Apple M5 Max, Japan) behind a public Cloudflare tunnel, offered for this run. It is not a production service and it is not our server, so it is treated like the other author-hosted endpoints: ×2, no +0.15 s. The measured latency includes the network path from Germany to Japan; that is stated in the row.

Cost — never an automatic 100

• jqv: the public per-token tariff of the exact base model, OpenRouter qwen/qwen3-32b, $0.08 per million input tokens, $0.28 per million output tokens (read 20 Sep 2026). Nothing is generated, so only input tokens are charged, against the input-token count the endpoint itself reports. The endpoint costs us nothing; the free run is not the price.

• gliner2.5-small, gliner2.5-multi: the base-size encoder class used for the GLiNER2 row, $0.005 per million input tokens (deepinfra base-size encoders). The small checkpoint is smaller than that class, so its estimate errs high.

• gliner2-large: the large-encoder class, $0.01 per million input tokens (deepinfra large encoders), because it is a large-encoder-sized checkpoint.

• opendecision: the large-encoder class used for the Laya row, $0.01 per million input tokens. Its choice path runs several NLI passes over the same state, which the reported token count does not include, so a per-token hosted price would be higher than this estimate. Stated in its footnote.

Held-out items

The checkpoints we run ourselves — gliner2.5-small, gliner2.5-multi, gliner2-large — ran the full set including the held-out items, on our own machine, as every earlier row did.

jqv is different and is therefore a partial row, shown but not ranked. Its endpoint is the submitter's own machine, and this round stopped sending held-out items to an endpoint that a submitter operates. The easy and standard/judge tiers had already been sent in full when that was decided and do contain their held-out items; the run was then stopped before the held-out block of the hard tier and re-run against the 111 public hard items alone. The 109 held-out hard items were never sent, count as unanswered, and the row covers 425 of 534 decisions. The stopped run is kept with its reason in the job folder.

This is stricter than the README's general statement that held-out items are sent to the services being evaluated ("not public is not the same as not seen"). Rows measured before this round are unchanged.