拓展阅读 / JevBench：独立评测基准

知识库收录材料 · 原始文件 ↗ · 来源、版本与许可

JevBench v1.2 additions: how the requested systems are run

Written on 19 Sep 2026, before the full runs. The runs happened after this file was committed. The systems were requested in replies to the JevBench launch posts. Every system gets the same frozen 534 decisions (easy, standard, judge, hard, held-out items included) and the unchanged v1.2 scoring (jevbench/composite_v12.py). No other row changes.

v1.2.6 — Verdict 1.4 and SimpleJev

openJev Verdict 1.4 uses the same public heman10x/rlcd-modernbert-151m weights as the earlier Verdict row, but the author's fixed engine at commit 30f1556: automatic calibrator loading for every option count, NLI-sentence candidate framing and a 512-token context budget. It ran locally on four Sandy CPU threads. The earlier Verdict row stays untouched because the engine, not the weights, changed.

The SimpleJev rows use Featherless AI's public no-login classifier demo at commit 7cba7d1, one request at a time and below its published two-request-per-second limit. The exact public model IDs are recorded. SimpleJev reads the model's answer-token logits and returns the complete distribution without generating an answer. Its shared demo is not a production service, so the normal public-demo latency adjustment (raw latency ×2) applies. Cost is an estimated hosted input price for the same weights or size class; it is not free/100.

v1.2.5 — kev family

Four public Apache-2.0 checkpoints from jaredpalmer/kev were measured through the author's native POST /v1/systemone server at commit 20fa626: the released Qwen2.5-based kev 0.5B, plus the Qwen3-based 0.6B, 4B and 8B research previews. All use LoRA plus a learned pointer head and return their own complete probability distributions.

Each checkpoint ran the same frozen 534-decision v1.2 set, serially in BF16 on one RunPod RTX 3090, reached over the internet from Sandy. All four completed 534/534 requests with no failures. Speed uses the normal self-hosted adjustment (raw latency ×2 + 0.15 seconds). Cost is an estimate from hosted prices for the corresponding size class and measured input tokens; output is zero because the pointer head does not generate tokens. The preview label comes from the author's model cards and is preserved in the display name.

Common rules

• One request at a time, serial, through the same runner as every other entrant. Each run folder is immutable.

• Held-out items stay on our machines or go to a production API. The local systems run on Sandy's CPU (AMD Ryzen 5 3600, 4 threads, MemoryMax=4G). classifier.dev is a production API. For ProgramAsWeights, only the rubric text reaches its hosted compiler; the item states never leave our machine.

• Weights load before the clock starts (JEVBENCH_WARM_LOAD=1), like a server that is already up.

• Latency. Our CPU gets the same adjustment as the other self-hosted rows: ×2 + 0.15 s (an assumption, not a measurement). A production API gets no adjustment. Raw p50/p95 are kept.

• Cost is never an automatic 100. A public tariff × tokens where one exists. Otherwise the "est." hosted-provider list price of the same size class × tokens, with the same method and table as the other open rows:

  • Large encoders (~400M): DeepInfra bge-large / e5-large / Qwen3-Embedding-0.6B, $0.01 per M input tokens.

  • Base encoders (~150–200M): DeepInfra bge-base / e5-base / gte-base / all-mpnet-base, $0.005 per M input tokens.

  • Generative models under 1B: no such model is listed, so the smallest listed one applies: OpenRouter meta-llama/llama-3.2-1b-instruct, $0.027/M in and $0.201/M out, as for Needle 3. This errs high.

  • Tokens are the system's own count where it reports one. Otherwise we use the per-decision counts of the other local rows (452 per decision, 1,235 per hard decision).

• A missing fit is a miss, the same way for everyone. A failed, refused or unparseable answer counts as wrong. So does an input over a system's documented context limit.

• Label-only systems get accuracy and a calibration of 0, like the other label-only rows.

Per system

GLiNER2 mapping (fixed before the run)

GLiNER2 classifies a text into one of a set of labels, optionally with label descriptions. It has no instruction field and no yes/no or ordinal primitive.

• Text: "Question: <instructions>\n\n<state>".

• Labels:

  • choice: {option: description}.

  • noul: {"yes": criteria.true or "Yes", "no": criteria.false or "No"}.

  • score: {"0": level 0 text, "1": level 1 text, ...}.

• Distribution: GLiNER2's default single-label decoding ("auto" activation) is a softmax over the label logits, and it returns only the argmax. We read the whole softmax through its documented parameters class_act="softmax", multi_label=True and cls_threshold=0, which list every label with its softmax probability.

  • On the smoke item, the top probability equals the default single-label confidence to every digit.

  • So the distribution is GLiNER2's own. We did not choose a normalization, and the independent sigmoid (multi-label) scores are not used.

  • This answers the concern in AVAILABILITY.md: the per-label sigmoid scores were never a categorical posterior, but the single-label head already is one.

Verdict: which weights

• The Hugging Face repo heman10x/openJev-verdict-2.0 has config.json and the tokenizer, but no weight file (checked 19 Sep 2026).

• Its config.json is byte-identical (sha256 303f8eef…) to the one in heman10x/rlcd-modernbert-151m.

• That repo's model.safetensors (sha256 d2528239…, verified after download) is the checkpoint named in the author's GitHub bundle manifest (artifacts/v2/bundle_manifest.json in openJev-verdict-2.0). The GitHub webgpu-demo also loads it.

• So this row is those published weights. It is labelled "openJev Verdict (heman10x, …)", not "2.0".

• The separate verdict2-base checkpoint, which the README's headline numbers describe, is a Git LFS object the server does not have (HTTP 404). We will run it when it is published.

ProgramAsWeights compile step

• What gets compiled: each distinct JevBench question (instructions + rubric) is compiled once, before its items run. That is 223 programs for 534 decisions.

• The spec is built by paw_spec() in jevbench/adapters/paw_local.py: the instructions, then Return ONLY one of: <labels>., then one line per label with its description. This follows the author's spec guide. No examples are added, because no entrant gets examples.

• Compiling is not decision latency. It runs on PAW's hosted API, paced under its published anonymous quota of 20 compiles per hour; each compile took about 4 s. Loading a compiled program also takes time. Both happen before the runner's clock starts, through a prepare() hook, and are kept separately as prepare_s. Decision latency is the fn(state) call only.

• Scoring the output: the output text is matched to the label set after trimming whitespace, quotes and a trailing full stop, ignoring case. Anything else counts wrong. An input over the 2,048-token context is the system's documented limit and counts wrong.

classifier.dev mapping

• One primitive: classifier.dev has only single-label choice, so every item is sent as one.

• Labels:

  • choice: the option keys.

  • noul: yes / no.

  • score: the level texts, mapped back to level indices. The API's own advice is that semantic names classify better than codes.

• Instructions: the question plus the full rubric (every option or level with its description), as for every other adapter.

• Distribution: its scores field. If the service withholds scores for an input, that item is scored on its label alone.

• Rate limits: we stay far inside them (534 of 20,000 per day). A 429 is answered by waiting and retrying at most 3 times; after that the run stops.

• Cost: the service is free for us. We use its published paid price: Pro, $20 per month for 200,000 fast classifications a day, which is $0.0033 per 1,000 at full use. This is labelled a plan price, not a per-token tariff.