Skip to content

PT-SYNTHETIC-DATA. Instruction Generation and Verifier Admission | 合成指令生成与验证准入 ​

难度: Medium | 环境: CPU 主线 | 标签: 后训练, 合成数据, 验证器, 数据准入 | 目标人群: 理解指令数据、需要筛选合成样本的读者

🚀 云端运行环境

本章节的实战代码可以点击以下链接在免费 GPU 算力平台上直接运行:

Open In ColabOpen In Studio (国内推荐:魔搭社区免费实例)


本节导读 ​

已有退款示范很多,生成模型却继续给出“怎样退款”的轻微改写;另一些回答措辞流畅,却漏掉了业务条件。增加生成数量,未必能补足训练数据中缺少的任务,还可能把评测问题带进训练。

本节先用任务蓝图写明要补足的能力、输入范围和成功条件,再记录生成来源,比较指令相似度并读取验证结果。候选据此进入接纳、复核或拒绝三类;接纳后仍保留来源与检查信息,继续进行字段、重复和切分检查。

关键词: 任务蓝图, 生成来源, 字符片段, 验证器, 候选筛选


运行环境 ​


前置阅读 ​

先理解合成数据为什么需要验证,再区分候选筛查与训练集切分检查。

Step 1:保存任务要求与生成来源 ​

任务蓝图在生成前写明要补足的能力、输入范围和成功条件。例如已有普通退款问答,但缺少“订单未发货时修改地址”的示范,就应针对订单状态和可执行步骤设计任务,而不只是改写退款问题。蓝图随后配合生成模型和提示模板产生候选。

每条候选保存蓝图版本、生成模型版本、模板版本和样本标识,这四项合称生成身份。例如 refund-v1 / teacher-v2 / self-instruct-v1 / candidate-1,更换模板后就能区分新旧来源。把验证结果也关联到该标识,便于回查错误来自哪次生成、哪项检查;若还要重新生成同一输出,应继续保存采样参数与随机种子。

来源信息记录内容用途
任务蓝图能力目标、输入范围、难度与成功条件检查候选是否补足预定缺口
生成模型版本模型权重提交或服务版本定位哪一版本产生了候选
提示模板版本角色格式、种子示例与生成要求比较模板改变后的候选差异
候选标识当前集合中稳定、可区分的样本编号关联内容、验证结果与修改记录

合成候选的来源与筛查流程

Step 2:分别比较训练任务、同批候选与评测任务 ​

字符 n-gram 是连续 n 个字符组成的片段;“三步退款”在 n=2 时得到“三步”“步退”“退款”。非空文本短于 n 时,本页把整段作为一个片段。Jaccard 相似度用共有片段数除以全部不同片段数;记文本片段集合为 Nn(x),候选指令为 qz,参考指令集合为 D,则最大相似度及这里使用的新颖度分数为

S(z,D)=maxd∈D|Nn(qz)∩Nn(d)||Nn(qz)∪Nn(d)|,novelty(z)=1−S(z,D).

比较前统一大小写与空白;空参考集取最大相似度 0,两份空片段集取相似度 1。训练参考用于识别近重复,同批参考应排除候选自身;两者达到相似度阈值时进入复核。评测侧另查规范化后的任务完全重合,重合时直接拒绝。本页将任务条件写在指令里;有独立上下文时,应把上下文一起纳入任务比较。字符重合是筛查信号,改写后的语义重合还需语义检索或人工检查。

比较对象比较范围结果用途
已有训练指令候选与训练参考任务高相似时检查是否只作了轻微改写
同批其他指令候选与同次生成的其他候选高相似时检查是否重复占用数据预算
固定评测任务规范化后的候选任务与评测任务完全重合时拒绝,保留原因

Step 3:根据验证结果决定接纳、复核或拒绝 ​

验证器(verifier)按任务规则检查候选,例如格式检查确认字段齐全,执行验证运行代码或核对可计算答案。每项结果保存名称、版本、是否通过和失败原因;开放式回答还可记录质量评分。任务先声明至少一个必需验证器,准入时读取它们的已有结果。格式通过说明结构符合规则,答案是否正确则取决于任务选用的检查方法。

先收集全部问题,再决定去向:空内容、评测任务重合、必需验证缺失或失败、质量低于下限,均进入拒绝;其余近重复或中间分数进入复核。设质量为 Q,复核线和接纳线为 τr、 τa,满足 0≤τr≤τa≤1。例如取 0.60 和 0.80:低于 0.60 拒绝,等于 0.60 复核,达到 0.80 且其他检查通过时接纳。同一候选若同时重现评测任务且接近训练模板,最终拒绝,但两项原因都保存。

决定触发条件后续动作
接纳内容完整,无完全重合评测任务,必需验证通过,相似度低于复核线且质量达到接纳线保留来源和验证结果,继续检查训练数据字段与切分
复核必需验证通过,但近重复或质量处于中间区间检查任务价值与回答,再重新筛查
拒绝空内容、评测任务重合、必需验证缺失或失败,或质量低于复核线保留全部原因,修改或移除候选

合成候选的拒绝、复核与接纳条件

Step 4:实现合成候选的验证准入 ​

题目使用内存中构造的任务蓝图、生成来源、参考指令和已给出的验证结果。骨架校验阈值、收集问题并保存结果;你将检查来源信息是否齐全、计算指令相似度、选择接纳/复核/拒绝,并筛出可以交给32节继续检查的候选。这里不调用生成模型或运行验证器,测试包含已有任务复写、同批近重复、评测任务重现、验证器缺失和阈值等号等情况。

TODO / 函数学习者完成的机制必须满足的约束测试证据
TODO 1:检查生成来源是否齐全
build_generation_identity
检查蓝图、生成器、模板与候选标识是否完整任一版本变化都必须改变身份;缺失字段不得形成有效身份相同候选更换模板版本后身份变化,缺失生成器时报错
TODO 2:计算集合相似度
max_reference_similarity
用字符片段 Jaccard 相似度识别模板复写n 必须为正整数;空参考集返回零;精确重复返回一精确重复、不同任务、空参考集和非法 n 得到预期结果
TODO 3:选择三态准入结果
decide_synthetic_admission
在全部原因已收集后,按风险优先级选择准入结果阈值顺序有效;必需验证器保留版本和失败原因;拒绝优先于复核干净候选接纳;两类近重复进入复核;污染、缺失或失败验证器被拒绝
TODO 4:筛选可继续审计的记录
build_sft_candidate_records
判断决定与原因是否同时允许交接样本身份不得重复;来源、验证器版本和决定可回查拒绝与复核行排除;决定与原因矛盾时排除;生成、评分和验证证据完整保留
python
from copy import deepcopy
import json
from typing import Dict, Iterable, List, Sequence
python
def normalize_text(text: object) -> str:
    """统一大小写与空白,生成相似度及精确任务比较形式。"""
    return " ".join(str(text or "").casefold().split())


def build_generation_identity(candidate: Dict[str, object]) -> tuple[str, str, str, str]:
    """读取蓝图、生成模型、模板版本与候选标识,拒绝缺项或空值。"""
    fields = ("blueprint_id", "generator_id", "template_id", "candidate_id")
    identity = tuple(str(candidate.get(field) or "").strip() for field in fields)
    # ==========================================
    # TODO 1:检查生成来源是否齐全
    # ==========================================
    # missing = ???  # 字段名列表;只保留生成身份中为空的分量所对应的字段。
    raise NotImplementedError("TODO 1:检查生成来源是否齐全")
    if missing:
        raise ValueError(f"生成身份字段缺失:{missing}")
    return identity


def character_ngrams(text: object, n: int = 2) -> set[str]:
    """提取字符片段集合,短文本保留为单个片段。"""
    normalized = normalize_text(text)
    if not normalized:
        return set()
    if len(normalized) < n:
        return {normalized}
    return {normalized[index:index + n] for index in range(len(normalized) - n + 1)}


def max_reference_similarity(
    instruction: object, references: Iterable[object], n: int = 2
) -> float:
    """比较字符集合,返回候选与参考指令的最大重合率。"""
    if type(n) is not int or n <= 0:
        raise ValueError("n 必须为正整数")
    candidate_ngrams = character_ngrams(instruction, n=n)
    similarities = []
    for reference in references:
        reference_ngrams = character_ngrams(reference, n=n)
        union = candidate_ngrams | reference_ngrams
        # ==========================================
        # TODO 2:计算集合相似度
        # ==========================================
        # similarity = ???  # [0, 1];两组字符片段的重合率,两个空集合视为完全相同。
        raise NotImplementedError("TODO 2:计算集合相似度")
        similarities.append(similarity)
    return max(similarities, default=0.0)


def decide_synthetic_admission(
    candidate: Dict[str, object],
    training_references: Sequence[str],
    batch_references: Sequence[str],
    evaluation_tasks: set[str],
    required_verifiers: Sequence[str],
    similarity_review_threshold: float = 0.72,
    quality_admit_threshold: float = 0.80,
    quality_review_threshold: float = 0.60,
) -> Dict[str, object]:
    """保留各项风险,按拒绝、复核、接纳的优先级选择结果。"""
    if not required_verifiers or any(not isinstance(name, str) or not name.strip() for name in required_verifiers):
        raise ValueError("必须声明至少一个具名验证器")
    if not 0.0 <= similarity_review_threshold <= 1.0:
        raise ValueError("similarity_review_threshold 必须位于 [0, 1]")
    if not 0.0 <= quality_review_threshold <= quality_admit_threshold <= 1.0:
        raise ValueError("质量阈值必须满足 0 <= review <= admit <= 1")
    generation_identity = build_generation_identity(candidate)
    instruction_key = normalize_text(candidate.get("instruction", ""))
    training_similarity = max_reference_similarity(
        candidate.get("instruction", ""), training_references
    )
    batch_similarity = max_reference_similarity(
        candidate.get("instruction", ""), batch_references
    )
    quality_score = float(candidate.get("quality_score", 0.0))
    if not 0.0 <= quality_score <= 1.0:
        raise ValueError("quality_score 必须位于 [0, 1]")
    verifier_evidence = deepcopy(candidate.get("verifier_results") or {})
    if not isinstance(verifier_evidence, dict):
        raise ValueError("verifier_results 必须为字典")
    reasons = []
    if not instruction_key:
        reasons.append("missing_instruction")
    if not normalize_text(candidate.get("response")):
        reasons.append("missing_response")
    if instruction_key in evaluation_tasks:
        reasons.append("evaluation_task_leak")
    for verifier in required_verifiers:
        evidence = verifier_evidence.get(verifier)
        if not isinstance(evidence, dict) or not evidence.get("version"):
            reasons.append(f"required_verifier_missing:{verifier}")
        elif evidence.get("passed") is not True:
            reasons.append(f"required_verifier_failed:{verifier}")
            if not str(evidence.get("failure_reason") or "").strip():
                reasons.append(f"required_verifier_reason_missing:{verifier}")
    if training_similarity >= similarity_review_threshold:
        reasons.append("high_training_similarity")
    if batch_similarity >= similarity_review_threshold:
        reasons.append("high_batch_similarity")
    if quality_review_threshold <= quality_score < quality_admit_threshold:
        reasons.append("quality_needs_review")
    elif quality_score < quality_review_threshold:
        reasons.append("quality_below_minimum")

    reject_reasons = [
        reason for reason in reasons
        if reason in {"evaluation_task_leak", "missing_instruction", "missing_response"}
        or reason.startswith("required_verifier_missing:")
        or reason.startswith("required_verifier_failed:")
        or reason.startswith("required_verifier_reason_missing:")
        or reason == "quality_below_minimum"
    ]
    # ==========================================
    # TODO 3:选择三态准入结果
    # ==========================================
    # decision = ???  # 字符串;有拒绝原因时拒绝,否则有复核原因时复核,其余接纳。
    raise NotImplementedError("TODO 3:选择三态准入结果")
    return {
        "candidate_id": generation_identity[3],
        "blueprint_id": generation_identity[0],
        "generator_id": generation_identity[1],
        "template_id": generation_identity[2],
        "instruction": str(candidate.get("instruction") or "").strip(),
        "response": str(candidate.get("response") or "").strip(),
        "generation_identity": generation_identity,
        "training_similarity": training_similarity,
        "batch_similarity": batch_similarity,
        "quality_score": quality_score,
        "verifier_evidence": verifier_evidence,
        "decision": decision,
        "reasons": reasons,
    }


def build_sft_candidate_records(rows: Sequence[Dict[str, object]]) -> List[Dict[str, object]]:
    """将通过筛查的候选转为SFT记录,保留来源版本和验证结果。"""
    records = []
    seen_ids = set()
    for row in rows:
        # ==========================================
        # TODO 4:筛选可继续审计的记录
        # ==========================================
        # handoff_allowed = ???  # bool;同时满足准入决定与原因清单的一致性。
        raise NotImplementedError("TODO 4:筛选可继续审计的记录")
        if not handoff_allowed:
            continue
        sample_id = str(row["candidate_id"])
        if sample_id in seen_ids:
            raise ValueError(f"接纳样本身份重复:{sample_id}")
        seen_ids.add(sample_id)
        verifier_versions = {
            name: evidence["version"]
            for name, evidence in row["verifier_evidence"].items()
            if isinstance(evidence, dict) and evidence.get("version")
        }
        records.append({
            "sample_id": sample_id,
            "source_id": json.dumps(list(row["generation_identity"]), ensure_ascii=False, separators=(",", ":")),
            "split": "train",
            "instruction": row["instruction"],
            "input": "",
            "response": row["response"],
            "admission_decision": row["decision"],
            "verifier_versions": verifier_versions,
            "provenance": {
                "generation_identity": list(row["generation_identity"]),
                "admission_decision": row["decision"],
                "verifier_evidence": deepcopy(row["verifier_evidence"]),
                "quality_score": row["quality_score"],
                "training_similarity": row["training_similarity"],
                "batch_similarity": row["batch_similarity"],
                "reasons": list(row["reasons"]),
            },
        })
    return records

测试 ​

运行下方CPU机制测试,检查来源版本是否齐全、相似度如何影响候选去向、验证失败是否优先拒绝,以及交给32节的记录是否仍保留生成与验证信息。

python
def _candidate(**overrides) -> Dict[str, object]:
    """构造可覆盖来源、质量与验证结果的候选。"""
    base = {
        "candidate_id": "candidate-1",
        "blueprint_id": "refund-v1",
        "generator_id": "teacher-v2",
        "template_id": "self-instruct-v1",
        "instruction": "说明未发货订单如何修改收货地址",
        "response": "先核对订单状态,再更新地址并确认结果。",
        "quality_score": 0.92,
        "verifier_results": {
            "format": {"passed": True, "version": "format-v1", "failure_reason": None},
            "answer": {"passed": True, "version": "answer-v2", "failure_reason": None},
        },
    }
    return {**base, **overrides}


def _assert_raises(error_type, fn) -> None:
    """确认非法输入抛出指定异常。"""
    try:
        fn()
    except error_type:
        return
    raise AssertionError(f"预期抛出 {error_type.__name__}")


def test_generation_identity():
    """核对生成版本变化与身份缺失。"""
    original = _candidate()
    changed = _candidate(template_id="self-instruct-v2")
    assert build_generation_identity(original) != build_generation_identity(changed)
    _assert_raises(ValueError, lambda: build_generation_identity(_candidate(generator_id="")))


def test_similarity_boundaries():
    """核对空参考、完全重复与不同任务的相似度。"""
    instruction = "说明未发货订单如何修改收货地址"
    assert max_reference_similarity(instruction, []) == 0.0
    assert max_reference_similarity(instruction, [instruction]) == 1.0
    different = max_reference_similarity(instruction, ["计算两个矩阵的乘积"])
    assert 0.0 <= different < 0.5
    _assert_raises(ValueError, lambda: max_reference_similarity(instruction, [], n=0))


def test_admission_decisions():
    """分别覆盖接纳、复核与拒绝的来源。"""
    required = ["format", "answer"]
    clean = decide_synthetic_admission(
        _candidate(), ["解释退款处理时限"], [], set(), required
    )
    assert clean["decision"] == "admit" and clean["reasons"] == []

    repeated = decide_synthetic_admission(
        _candidate(), [_candidate()["instruction"]], [], set(), required
    )
    assert repeated["decision"] == "review"
    assert "high_training_similarity" in repeated["reasons"]

    batch_duplicate = decide_synthetic_admission(
        _candidate(), [], [_candidate()["instruction"]], set(), required
    )
    assert batch_duplicate["decision"] == "review"
    assert "high_batch_similarity" in batch_duplicate["reasons"]

    contaminated = decide_synthetic_admission(
        _candidate(), [], [], {normalize_text(_candidate()["instruction"])}, required
    )
    assert contaminated["decision"] == "reject"
    assert "evaluation_task_leak" in contaminated["reasons"]

    failed = decide_synthetic_admission(
        _candidate(verifier_results={
            "format": {"passed": True, "version": "format-v1", "failure_reason": None},
            "answer": {"passed": False, "version": "answer-v2", "failure_reason": "wrong_answer"},
        }),
        [], [], set(), required,
    )
    assert failed["decision"] == "reject"
    assert failed["reasons"] == ["required_verifier_failed:answer"]

    uncertain = decide_synthetic_admission(
        _candidate(quality_score=0.70), [], [], set(), required
    )
    assert uncertain["decision"] == "review"
    assert uncertain["reasons"] == ["quality_needs_review"]
    _assert_raises(
        ValueError,
        lambda: decide_synthetic_admission(
            _candidate(), [], [], set(), required,
            quality_admit_threshold=0.5, quality_review_threshold=0.8,
        ),
    )


def test_reject_precedes_review():
    """同时记录多项原因,并保证拒绝优先。"""
    candidate = _candidate(
        quality_score=0.70,
        verifier_results={
            "format": {"passed": False, "version": "format-v1", "failure_reason": "bad_format"},
            "answer": {"passed": True, "version": "answer-v2", "failure_reason": None},
        },
    )
    row = decide_synthetic_admission(
        candidate,
        [candidate["instruction"]],
        [],
        {normalize_text(candidate["instruction"])},
        ["format", "answer"],
    )
    assert row["decision"] == "reject"
    assert set(row["reasons"]) == {
        "evaluation_task_leak",
        "required_verifier_failed:format",
        "high_training_similarity",
        "quality_needs_review",
    }


def test_sft_candidate_handoff():
    """核对交接过滤、重复身份及验证器版本。"""
    admitted = decide_synthetic_admission(
        _candidate(), ["解释退款处理时限"], [], set(), ["format", "answer"]
    )
    rejected = decide_synthetic_admission(
        _candidate(candidate_id="candidate-2"),
        [], [], {normalize_text(_candidate()["instruction"])}, ["format", "answer"],
    )
    records = build_sft_candidate_records([admitted, rejected])
    assert build_sft_candidate_records([rejected]) == []
    assert build_sft_candidate_records([{**admitted, "decision": "review"}]) == []
    assert [record["sample_id"] for record in records] == ["candidate-1"]
    assert json.loads(records[0]["source_id"]) == ["refund-v1", "teacher-v2", "self-instruct-v1", "candidate-1"]
    assert records[0]["split"] == "train" and records[0]["response"]
    assert records[0]["verifier_versions"] == {"format": "format-v1", "answer": "answer-v2"}
    _assert_raises(ValueError, lambda: build_sft_candidate_records([admitted, admitted]))



def test_empty_content_and_verifier_policy():
    """空监督、空验证策略及缺失版本均不能获得接纳。"""
    required = ["format", "answer"]
    for field in ("instruction", "response"):
        row = decide_synthetic_admission(_candidate(**{field: "   "}), [], [], set(), required)
        assert row["decision"] == "reject" and f"missing_{field}" in row["reasons"]
    _assert_raises(ValueError, lambda: build_generation_identity(_candidate(template_id=None)))
    _assert_raises(ValueError, lambda: decide_synthetic_admission(_candidate(), [], [], set(), []))
    row = decide_synthetic_admission(_candidate(verifier_results={"format": {"passed": True}}), [], [], set(), required)
    assert row["decision"] == "reject"
    assert set(row["reasons"]) == {"required_verifier_missing:format", "required_verifier_missing:answer"}
    _assert_raises(ValueError, lambda: max_reference_similarity("文本", [], n=True))


def test_quality_threshold_boundaries():
    """质量恰好等于复核线和接纳线时,进入相应区间。"""
    for score, expected in ((0.59, "reject"), (0.60, "review"), (0.799, "review"), (0.80, "admit")):
        row = decide_synthetic_admission(_candidate(quality_score=score), [], [], set(), ["format", "answer"])
        assert row["decision"] == expected
    for score in (float("nan"), -0.1, 1.1):
        _assert_raises(ValueError, lambda: decide_synthetic_admission(_candidate(quality_score=score), [], [], set(), ["format", "answer"]))


def test_handoff_evidence_and_consistency():
    """证据完整交接且独立复制,决定与原因矛盾时不得交接。"""
    candidate = _candidate()
    row = decide_synthetic_admission(candidate, [], [], set(), ["format", "answer"])
    record = build_sft_candidate_records([row])[0]
    assert record["provenance"]["generation_identity"] == list(row["generation_identity"])
    assert record["provenance"]["verifier_evidence"] == candidate["verifier_results"]
    row["verifier_evidence"]["answer"]["version"] = "changed"
    assert record["provenance"]["verifier_evidence"]["answer"]["version"] == "answer-v2"
    inconsistent = {**row, "reasons": ["evaluation_task_leak"]}
    assert build_sft_candidate_records([inconsistent]) == []
    # JSON编码保留身份分量边界,不因分量包含分隔符而改变来源。
    special = decide_synthetic_admission(_candidate(blueprint_id="refund|v1"), [], [], set(), ["format", "answer"])
    assert json.loads(build_sft_candidate_records([special])[0]["source_id"])[0] == "refund|v1"


test_generation_identity()
test_similarity_boundaries()
test_admission_decisions()
test_reject_precedes_review()
test_sft_candidate_handoff()
print("✅ 合成候选生成身份、相似度、验证器与三态准入测试通过。")
test_empty_content_and_verifier_policy()
test_quality_threshold_boundaries()
test_handoff_evidence_and_consistency()

🛑 STOP HERE 🛑









请先尝试自己完成代码并跑通测试。
如果遇到困难没有思路,可以向下滚动查看参考答案。










参考代码与解析 ​

代码 ​

python
def normalize_text(text: object) -> str:
    """统一大小写与空白,生成相似度及精确任务比较形式。"""
    return " ".join(str(text or "").casefold().split())


def build_generation_identity(candidate: Dict[str, object]) -> tuple[str, str, str, str]:
    """读取蓝图、生成模型、模板版本与候选标识,拒绝缺项或空值。"""
    fields = ("blueprint_id", "generator_id", "template_id", "candidate_id")
    identity = tuple(str(candidate.get(field) or "").strip() for field in fields)
    # TODO 1:检查生成来源是否齐全
    missing = [field for field, value in zip(fields, identity) if not value]
    if missing:
        raise ValueError(f"生成身份字段缺失:{missing}")
    return identity


def character_ngrams(text: object, n: int = 2) -> set[str]:
    """提取字符片段集合,短文本保留为单个片段。"""
    normalized = normalize_text(text)
    if not normalized:
        return set()
    if len(normalized) < n:
        return {normalized}
    return {normalized[index:index + n] for index in range(len(normalized) - n + 1)}


def max_reference_similarity(
    instruction: object, references: Iterable[object], n: int = 2
) -> float:
    """比较字符集合,返回候选与参考指令的最大重合率。"""
    if type(n) is not int or n <= 0:
        raise ValueError("n 必须为正整数")
    candidate_ngrams = character_ngrams(instruction, n=n)
    similarities = []
    for reference in references:
        reference_ngrams = character_ngrams(reference, n=n)
        union = candidate_ngrams | reference_ngrams
        # TODO 2:计算集合相似度
        similarity = len(candidate_ngrams & reference_ngrams) / len(union) if union else 1.0
        similarities.append(similarity)
    return max(similarities, default=0.0)


def decide_synthetic_admission(
    candidate: Dict[str, object],
    training_references: Sequence[str],
    batch_references: Sequence[str],
    evaluation_tasks: set[str],
    required_verifiers: Sequence[str],
    similarity_review_threshold: float = 0.72,
    quality_admit_threshold: float = 0.80,
    quality_review_threshold: float = 0.60,
) -> Dict[str, object]:
    """保留各项风险,按拒绝、复核、接纳的优先级选择结果。"""
    if not required_verifiers or any(not isinstance(name, str) or not name.strip() for name in required_verifiers):
        raise ValueError("必须声明至少一个具名验证器")
    if not 0.0 <= similarity_review_threshold <= 1.0:
        raise ValueError("similarity_review_threshold 必须位于 [0, 1]")
    if not 0.0 <= quality_review_threshold <= quality_admit_threshold <= 1.0:
        raise ValueError("质量阈值必须满足 0 <= review <= admit <= 1")
    generation_identity = build_generation_identity(candidate)
    instruction_key = normalize_text(candidate.get("instruction", ""))
    training_similarity = max_reference_similarity(
        candidate.get("instruction", ""), training_references
    )
    batch_similarity = max_reference_similarity(
        candidate.get("instruction", ""), batch_references
    )
    quality_score = float(candidate.get("quality_score", 0.0))
    if not 0.0 <= quality_score <= 1.0:
        raise ValueError("quality_score 必须位于 [0, 1]")
    verifier_evidence = deepcopy(candidate.get("verifier_results") or {})
    if not isinstance(verifier_evidence, dict):
        raise ValueError("verifier_results 必须为字典")
    reasons = []
    if not instruction_key:
        reasons.append("missing_instruction")
    if not normalize_text(candidate.get("response")):
        reasons.append("missing_response")
    if instruction_key in evaluation_tasks:
        reasons.append("evaluation_task_leak")
    for verifier in required_verifiers:
        evidence = verifier_evidence.get(verifier)
        if not isinstance(evidence, dict) or not evidence.get("version"):
            reasons.append(f"required_verifier_missing:{verifier}")
        elif evidence.get("passed") is not True:
            reasons.append(f"required_verifier_failed:{verifier}")
            if not str(evidence.get("failure_reason") or "").strip():
                reasons.append(f"required_verifier_reason_missing:{verifier}")
    if training_similarity >= similarity_review_threshold:
        reasons.append("high_training_similarity")
    if batch_similarity >= similarity_review_threshold:
        reasons.append("high_batch_similarity")
    if quality_review_threshold <= quality_score < quality_admit_threshold:
        reasons.append("quality_needs_review")
    elif quality_score < quality_review_threshold:
        reasons.append("quality_below_minimum")

    reject_reasons = [
        reason for reason in reasons
        if reason in {"evaluation_task_leak", "missing_instruction", "missing_response"}
        or reason.startswith("required_verifier_missing:")
        or reason.startswith("required_verifier_failed:")
        or reason.startswith("required_verifier_reason_missing:")
        or reason == "quality_below_minimum"
    ]
    # TODO 3:选择三态准入结果
    decision = "reject" if reject_reasons else "review" if reasons else "admit"
    return {
        "candidate_id": generation_identity[3],
        "blueprint_id": generation_identity[0],
        "generator_id": generation_identity[1],
        "template_id": generation_identity[2],
        "instruction": str(candidate.get("instruction") or "").strip(),
        "response": str(candidate.get("response") or "").strip(),
        "generation_identity": generation_identity,
        "training_similarity": training_similarity,
        "batch_similarity": batch_similarity,
        "quality_score": quality_score,
        "verifier_evidence": verifier_evidence,
        "decision": decision,
        "reasons": reasons,
    }


def build_sft_candidate_records(rows: Sequence[Dict[str, object]]) -> List[Dict[str, object]]:
    """将通过筛查的候选转为SFT记录,保留来源版本和验证结果。"""
    records = []
    seen_ids = set()
    for row in rows:
        # TODO 4:筛选可继续审计的记录
        handoff_allowed = row["decision"] == "admit" and not row["reasons"]
        if not handoff_allowed:
            continue
        sample_id = str(row["candidate_id"])
        if sample_id in seen_ids:
            raise ValueError(f"接纳样本身份重复:{sample_id}")
        seen_ids.add(sample_id)
        verifier_versions = {
            name: evidence["version"]
            for name, evidence in row["verifier_evidence"].items()
            if isinstance(evidence, dict) and evidence.get("version")
        }
        records.append({
            "sample_id": sample_id,
            "source_id": json.dumps(list(row["generation_identity"]), ensure_ascii=False, separators=(",", ":")),
            "split": "train",
            "instruction": row["instruction"],
            "input": "",
            "response": row["response"],
            "admission_decision": row["decision"],
            "verifier_versions": verifier_versions,
            "provenance": {
                "generation_identity": list(row["generation_identity"]),
                "admission_decision": row["decision"],
                "verifier_evidence": deepcopy(row["verifier_evidence"]),
                "quality_score": row["quality_score"],
                "training_similarity": row["training_similarity"],
                "batch_similarity": row["batch_similarity"],
                "reasons": list(row["reasons"]),
            },
        })
    return records

解析 ​

TODO 1:检查生成来源 ​

  • 蓝图、生成模型、模板与候选标识共同定位来源;缺少任一项时,后续验证结果就难以对应到完整生成记录。
  • 根据骨架给出的四个分量找出缺项。测试更换模板版本、清空生成器或传入空值,分别检查身份变化和缺项报错。

TODO 2:计算片段集合相似度 ​

  • 共有片段占全部不同片段的比例越高,文本表面越接近;没有参考对象时最大相似度取零。
  • 用交集大小除以并集大小,并在两者均空时取一,由骨架汇总最大值。测试包括完全相同、不同任务、空参考集合及非法片段长度。

TODO 3:按全部原因选择去向 ​

  • 评测重合或必需验证失败时直接拒绝;通过这些检查后,近重复或中间质量分数才进入复核。
  • 先检查拒绝原因,再检查是否还有复核原因。组合用例保留污染、验证失败和近重复等所有原因;分数恰好为 0.60 与 0.80 时分别进入复核与接纳。

TODO 4:保留接纳记录及其来源 ​

  • 决定为接纳且问题列表为空,记录才可继续用于训练数据审计。生成与验证信息随样本一起复制,便于后续回查。
  • 过滤复核、拒绝和“写着接纳却仍有问题”的记录。测试验证重复标识报错,并在修改原始验证信息后,确认已复制记录仍保留原值。

框架一致性核对(可选) ​

用同一批三态记录检查 Hugging Face Datasets 的转换与过滤:核对接纳集合、行顺序和嵌套来源信息是否保持。准入决定仍由 Step 4 的实现产生,这里核对数据接口。

核对对象Step 4 教学实现成熟框架 CPU 路径共同输入与通过条件
接纳集合decide_synthetic_admission 生成三态结果Dataset.from_list 与 Dataset.filter同一批结果按接纳状态过滤后,样本标识及顺序相同
来源和验证记录generation_identity 与验证结果Dataset.to_list转换前后的完整行、嵌套字段值与顺序相同
SFT 候选字段build_sft_candidate_recordsDataset.from_list 与 Dataset.to_list同一批接纳行转换后,样本、来源、指令、回答及附带信息保持
python
from importlib.metadata import version

try:
    from datasets import Dataset
except ImportError:
    framework_check = {
        "status": "skipped",
        "reason": "请使用 PT-CPU 环境运行本单元",
    }
else:
    def check_synthetic_data_interface():
        """核对本地准入决定与 Datasets 过滤、血缘和SFT交接字段。"""
        def framework_candidate(**overrides):
            """构造用于接口往返的三态候选。"""
            base = {
                "candidate_id": "candidate-1",
                "blueprint_id": "refund-v1",
                "generator_id": "teacher-v2",
                "template_id": "self-instruct-v1",
                "instruction": "说明未发货订单如何修改收货地址",
                "response": "先核对订单状态,再更新地址并确认结果。",
                "quality_score": 0.92,
                "verifier_results": {
                    "format": {"passed": True, "version": "format-v1", "failure_reason": None},
                    "answer": {"passed": True, "version": "answer-v2", "failure_reason": None},
                },
            }
            return {**base, **overrides}

        candidates = [
            framework_candidate(),
            framework_candidate(candidate_id="candidate-2", instruction="解释退款处理时限"),
            framework_candidate(
                candidate_id="candidate-3",
                instruction="列出账户注销步骤",
                verifier_results={
                    "format": {"passed": True, "version": "format-v1", "failure_reason": None},
                    "answer": {"passed": False, "version": "answer-v2", "failure_reason": "wrong_answer"},
                },
            ),
        ]
        references = ["解释退款处理时限"]
        rows = [
            decide_synthetic_admission(
                candidate, references, [], set(), ["format", "answer"]
            )
            for candidate in candidates
        ]
        serializable_rows = [
            {
                **row,
                "generation_identity": list(row["generation_identity"]),
                "reasons": list(row["reasons"]),
            }
            for row in rows
        ]
        dataset = Dataset.from_list(serializable_rows)
        admitted_dataset = dataset.filter(lambda row: row["decision"] == "admit")
        local_admitted_ids = [row["candidate_id"] for row in rows if row["decision"] == "admit"]
        framework_admitted_ids = admitted_dataset["candidate_id"]
        sft_records = build_sft_candidate_records(rows)
        sft_dataset = Dataset.from_list(sft_records)
        checks = {
            "admitted_ids_match": framework_admitted_ids == local_admitted_ids,
            "rows_values_and_order_preserved": dataset.to_list() == serializable_rows,
            "handoff_values_preserved": sft_dataset.to_list() == sft_records,
            "lineage_fields_preserved": all(
                row["blueprint_id"] and row["generator_id"] and row["template_id"]
                for row in admitted_dataset
            ),
            "generation_identity_preserved": all(
                len(row["generation_identity"]) == 4
                for row in admitted_dataset
            ),
            "sft_handoff_fields_readable": all(
                row["sample_id"] and row["source_id"] and row["split"] == "train"
                and row["instruction"] and row["response"]
                for row in sft_dataset
            ),
        }
        failed = [name for name, passed in checks.items() if not passed]
        if failed:
            raise AssertionError(f"合成数据框架核对失败:{failed}")
        return {
            "status": "passed",
            "check_kind": "data_interface",
            "versions": {"datasets": version("datasets")},
            "checks": checks,
            "candidate_rows": len(dataset),
            "admitted_rows": len(admitted_dataset),
        }

    framework_check = check_synthetic_data_interface()

framework_check

相关阅读 ​

继续比较合成方法,再核对候选怎样进入实际数据处理。

Released under the MIT License.