PT-SYNTHETIC-DATA. Instruction Generation and Verifier Admission | 合成指令生成与验证准入
难度: Medium | 环境: CPU 主线 | 标签: 后训练, 合成数据, 验证器, 数据准入 | 目标人群: 理解指令数据、需要筛选合成样本的读者
🚀 云端运行环境
本章节的实战代码可以点击以下链接在免费 GPU 算力平台上直接运行:
本节导读
已有退款示范很多,生成模型却继续给出“怎样退款”的轻微改写;另一些回答措辞流畅,却漏掉了业务条件。增加生成数量,未必能补足训练数据中缺少的任务,还可能把评测问题带进训练。
本节先用任务蓝图写明要补足的能力、输入范围和成功条件,再记录生成来源,比较指令相似度并读取验证结果。候选据此进入接纳、复核或拒绝三类;接纳后仍保留来源与检查信息,继续进行字段、重复和切分检查。
关键词: 任务蓝图, 生成来源, 字符片段, 验证器, 候选筛选
运行环境
- 基础练习: 默认使用
BASE-CPU,运行 Step 4 的题目与测试。 - 框架核对: 如需检查准入记录能否无损进入 Hugging Face Datasets,使用
PT-CPU,运行框架一致性核对。
前置阅读
先理解合成数据为什么需要验证,再区分候选筛查与训练集切分检查。
- 合成数据与轨迹验证:明确生成目标、验证方法与数据版本。
- 32 SFT 数据契约与泄漏审计:认识指令、回答、来源和切分字段,以及完整样本与任务输入的区别。
Step 1:保存任务要求与生成来源
任务蓝图在生成前写明要补足的能力、输入范围和成功条件。例如已有普通退款问答,但缺少“订单未发货时修改地址”的示范,就应针对订单状态和可执行步骤设计任务,而不只是改写退款问题。蓝图随后配合生成模型和提示模板产生候选。
每条候选保存蓝图版本、生成模型版本、模板版本和样本标识,这四项合称生成身份。例如 refund-v1 / teacher-v2 / self-instruct-v1 / candidate-1,更换模板后就能区分新旧来源。把验证结果也关联到该标识,便于回查错误来自哪次生成、哪项检查;若还要重新生成同一输出,应继续保存采样参数与随机种子。
| 来源信息 | 记录内容 | 用途 |
|---|---|---|
| 任务蓝图 | 能力目标、输入范围、难度与成功条件 | 检查候选是否补足预定缺口 |
| 生成模型版本 | 模型权重提交或服务版本 | 定位哪一版本产生了候选 |
| 提示模板版本 | 角色格式、种子示例与生成要求 | 比较模板改变后的候选差异 |
| 候选标识 | 当前集合中稳定、可区分的样本编号 | 关联内容、验证结果与修改记录 |
Step 2:分别比较训练任务、同批候选与评测任务
字符
比较前统一大小写与空白;空参考集取最大相似度 0,两份空片段集取相似度 1。训练参考用于识别近重复,同批参考应排除候选自身;两者达到相似度阈值时进入复核。评测侧另查规范化后的任务完全重合,重合时直接拒绝。本页将任务条件写在指令里;有独立上下文时,应把上下文一起纳入任务比较。字符重合是筛查信号,改写后的语义重合还需语义检索或人工检查。
| 比较对象 | 比较范围 | 结果用途 |
|---|---|---|
| 已有训练指令 | 候选与训练参考任务 | 高相似时检查是否只作了轻微改写 |
| 同批其他指令 | 候选与同次生成的其他候选 | 高相似时检查是否重复占用数据预算 |
| 固定评测任务 | 规范化后的候选任务与评测任务 | 完全重合时拒绝,保留原因 |
Step 3:根据验证结果决定接纳、复核或拒绝
验证器(verifier)按任务规则检查候选,例如格式检查确认字段齐全,执行验证运行代码或核对可计算答案。每项结果保存名称、版本、是否通过和失败原因;开放式回答还可记录质量评分。任务先声明至少一个必需验证器,准入时读取它们的已有结果。格式通过说明结构符合规则,答案是否正确则取决于任务选用的检查方法。
先收集全部问题,再决定去向:空内容、评测任务重合、必需验证缺失或失败、质量低于下限,均进入拒绝;其余近重复或中间分数进入复核。设质量为
| 决定 | 触发条件 | 后续动作 |
|---|---|---|
| 接纳 | 内容完整,无完全重合评测任务,必需验证通过,相似度低于复核线且质量达到接纳线 | 保留来源和验证结果,继续检查训练数据字段与切分 |
| 复核 | 必需验证通过,但近重复或质量处于中间区间 | 检查任务价值与回答,再重新筛查 |
| 拒绝 | 空内容、评测任务重合、必需验证缺失或失败,或质量低于复核线 | 保留全部原因,修改或移除候选 |
Step 4:实现合成候选的验证准入
题目使用内存中构造的任务蓝图、生成来源、参考指令和已给出的验证结果。骨架校验阈值、收集问题并保存结果;你将检查来源信息是否齐全、计算指令相似度、选择接纳/复核/拒绝,并筛出可以交给32节继续检查的候选。这里不调用生成模型或运行验证器,测试包含已有任务复写、同批近重复、评测任务重现、验证器缺失和阈值等号等情况。
| TODO / 函数 | 学习者完成的机制 | 必须满足的约束 | 测试证据 |
|---|---|---|---|
TODO 1:检查生成来源是否齐全build_generation_identity | 检查蓝图、生成器、模板与候选标识是否完整 | 任一版本变化都必须改变身份;缺失字段不得形成有效身份 | 相同候选更换模板版本后身份变化,缺失生成器时报错 |
TODO 2:计算集合相似度max_reference_similarity | 用字符片段 Jaccard 相似度识别模板复写 | 精确重复、不同任务、空参考集和非法 | |
TODO 3:选择三态准入结果decide_synthetic_admission | 在全部原因已收集后,按风险优先级选择准入结果 | 阈值顺序有效;必需验证器保留版本和失败原因;拒绝优先于复核 | 干净候选接纳;两类近重复进入复核;污染、缺失或失败验证器被拒绝 |
TODO 4:筛选可继续审计的记录build_sft_candidate_records | 判断决定与原因是否同时允许交接 | 样本身份不得重复;来源、验证器版本和决定可回查 | 拒绝与复核行排除;决定与原因矛盾时排除;生成、评分和验证证据完整保留 |
from copy import deepcopy
import json
from typing import Dict, Iterable, List, Sequencedef normalize_text(text: object) -> str:
"""统一大小写与空白,生成相似度及精确任务比较形式。"""
return " ".join(str(text or "").casefold().split())
def build_generation_identity(candidate: Dict[str, object]) -> tuple[str, str, str, str]:
"""读取蓝图、生成模型、模板版本与候选标识,拒绝缺项或空值。"""
fields = ("blueprint_id", "generator_id", "template_id", "candidate_id")
identity = tuple(str(candidate.get(field) or "").strip() for field in fields)
# ==========================================
# TODO 1:检查生成来源是否齐全
# ==========================================
# missing = ??? # 字段名列表;只保留生成身份中为空的分量所对应的字段。
raise NotImplementedError("TODO 1:检查生成来源是否齐全")
if missing:
raise ValueError(f"生成身份字段缺失:{missing}")
return identity
def character_ngrams(text: object, n: int = 2) -> set[str]:
"""提取字符片段集合,短文本保留为单个片段。"""
normalized = normalize_text(text)
if not normalized:
return set()
if len(normalized) < n:
return {normalized}
return {normalized[index:index + n] for index in range(len(normalized) - n + 1)}
def max_reference_similarity(
instruction: object, references: Iterable[object], n: int = 2
) -> float:
"""比较字符集合,返回候选与参考指令的最大重合率。"""
if type(n) is not int or n <= 0:
raise ValueError("n 必须为正整数")
candidate_ngrams = character_ngrams(instruction, n=n)
similarities = []
for reference in references:
reference_ngrams = character_ngrams(reference, n=n)
union = candidate_ngrams | reference_ngrams
# ==========================================
# TODO 2:计算集合相似度
# ==========================================
# similarity = ??? # [0, 1];两组字符片段的重合率,两个空集合视为完全相同。
raise NotImplementedError("TODO 2:计算集合相似度")
similarities.append(similarity)
return max(similarities, default=0.0)
def decide_synthetic_admission(
candidate: Dict[str, object],
training_references: Sequence[str],
batch_references: Sequence[str],
evaluation_tasks: set[str],
required_verifiers: Sequence[str],
similarity_review_threshold: float = 0.72,
quality_admit_threshold: float = 0.80,
quality_review_threshold: float = 0.60,
) -> Dict[str, object]:
"""保留各项风险,按拒绝、复核、接纳的优先级选择结果。"""
if not required_verifiers or any(not isinstance(name, str) or not name.strip() for name in required_verifiers):
raise ValueError("必须声明至少一个具名验证器")
if not 0.0 <= similarity_review_threshold <= 1.0:
raise ValueError("similarity_review_threshold 必须位于 [0, 1]")
if not 0.0 <= quality_review_threshold <= quality_admit_threshold <= 1.0:
raise ValueError("质量阈值必须满足 0 <= review <= admit <= 1")
generation_identity = build_generation_identity(candidate)
instruction_key = normalize_text(candidate.get("instruction", ""))
training_similarity = max_reference_similarity(
candidate.get("instruction", ""), training_references
)
batch_similarity = max_reference_similarity(
candidate.get("instruction", ""), batch_references
)
quality_score = float(candidate.get("quality_score", 0.0))
if not 0.0 <= quality_score <= 1.0:
raise ValueError("quality_score 必须位于 [0, 1]")
verifier_evidence = deepcopy(candidate.get("verifier_results") or {})
if not isinstance(verifier_evidence, dict):
raise ValueError("verifier_results 必须为字典")
reasons = []
if not instruction_key:
reasons.append("missing_instruction")
if not normalize_text(candidate.get("response")):
reasons.append("missing_response")
if instruction_key in evaluation_tasks:
reasons.append("evaluation_task_leak")
for verifier in required_verifiers:
evidence = verifier_evidence.get(verifier)
if not isinstance(evidence, dict) or not evidence.get("version"):
reasons.append(f"required_verifier_missing:{verifier}")
elif evidence.get("passed") is not True:
reasons.append(f"required_verifier_failed:{verifier}")
if not str(evidence.get("failure_reason") or "").strip():
reasons.append(f"required_verifier_reason_missing:{verifier}")
if training_similarity >= similarity_review_threshold:
reasons.append("high_training_similarity")
if batch_similarity >= similarity_review_threshold:
reasons.append("high_batch_similarity")
if quality_review_threshold <= quality_score < quality_admit_threshold:
reasons.append("quality_needs_review")
elif quality_score < quality_review_threshold:
reasons.append("quality_below_minimum")
reject_reasons = [
reason for reason in reasons
if reason in {"evaluation_task_leak", "missing_instruction", "missing_response"}
or reason.startswith("required_verifier_missing:")
or reason.startswith("required_verifier_failed:")
or reason.startswith("required_verifier_reason_missing:")
or reason == "quality_below_minimum"
]
# ==========================================
# TODO 3:选择三态准入结果
# ==========================================
# decision = ??? # 字符串;有拒绝原因时拒绝,否则有复核原因时复核,其余接纳。
raise NotImplementedError("TODO 3:选择三态准入结果")
return {
"candidate_id": generation_identity[3],
"blueprint_id": generation_identity[0],
"generator_id": generation_identity[1],
"template_id": generation_identity[2],
"instruction": str(candidate.get("instruction") or "").strip(),
"response": str(candidate.get("response") or "").strip(),
"generation_identity": generation_identity,
"training_similarity": training_similarity,
"batch_similarity": batch_similarity,
"quality_score": quality_score,
"verifier_evidence": verifier_evidence,
"decision": decision,
"reasons": reasons,
}
def build_sft_candidate_records(rows: Sequence[Dict[str, object]]) -> List[Dict[str, object]]:
"""将通过筛查的候选转为SFT记录,保留来源版本和验证结果。"""
records = []
seen_ids = set()
for row in rows:
# ==========================================
# TODO 4:筛选可继续审计的记录
# ==========================================
# handoff_allowed = ??? # bool;同时满足准入决定与原因清单的一致性。
raise NotImplementedError("TODO 4:筛选可继续审计的记录")
if not handoff_allowed:
continue
sample_id = str(row["candidate_id"])
if sample_id in seen_ids:
raise ValueError(f"接纳样本身份重复:{sample_id}")
seen_ids.add(sample_id)
verifier_versions = {
name: evidence["version"]
for name, evidence in row["verifier_evidence"].items()
if isinstance(evidence, dict) and evidence.get("version")
}
records.append({
"sample_id": sample_id,
"source_id": json.dumps(list(row["generation_identity"]), ensure_ascii=False, separators=(",", ":")),
"split": "train",
"instruction": row["instruction"],
"input": "",
"response": row["response"],
"admission_decision": row["decision"],
"verifier_versions": verifier_versions,
"provenance": {
"generation_identity": list(row["generation_identity"]),
"admission_decision": row["decision"],
"verifier_evidence": deepcopy(row["verifier_evidence"]),
"quality_score": row["quality_score"],
"training_similarity": row["training_similarity"],
"batch_similarity": row["batch_similarity"],
"reasons": list(row["reasons"]),
},
})
return records测试
运行下方CPU机制测试,检查来源版本是否齐全、相似度如何影响候选去向、验证失败是否优先拒绝,以及交给32节的记录是否仍保留生成与验证信息。
def _candidate(**overrides) -> Dict[str, object]:
"""构造可覆盖来源、质量与验证结果的候选。"""
base = {
"candidate_id": "candidate-1",
"blueprint_id": "refund-v1",
"generator_id": "teacher-v2",
"template_id": "self-instruct-v1",
"instruction": "说明未发货订单如何修改收货地址",
"response": "先核对订单状态,再更新地址并确认结果。",
"quality_score": 0.92,
"verifier_results": {
"format": {"passed": True, "version": "format-v1", "failure_reason": None},
"answer": {"passed": True, "version": "answer-v2", "failure_reason": None},
},
}
return {**base, **overrides}
def _assert_raises(error_type, fn) -> None:
"""确认非法输入抛出指定异常。"""
try:
fn()
except error_type:
return
raise AssertionError(f"预期抛出 {error_type.__name__}")
def test_generation_identity():
"""核对生成版本变化与身份缺失。"""
original = _candidate()
changed = _candidate(template_id="self-instruct-v2")
assert build_generation_identity(original) != build_generation_identity(changed)
_assert_raises(ValueError, lambda: build_generation_identity(_candidate(generator_id="")))
def test_similarity_boundaries():
"""核对空参考、完全重复与不同任务的相似度。"""
instruction = "说明未发货订单如何修改收货地址"
assert max_reference_similarity(instruction, []) == 0.0
assert max_reference_similarity(instruction, [instruction]) == 1.0
different = max_reference_similarity(instruction, ["计算两个矩阵的乘积"])
assert 0.0 <= different < 0.5
_assert_raises(ValueError, lambda: max_reference_similarity(instruction, [], n=0))
def test_admission_decisions():
"""分别覆盖接纳、复核与拒绝的来源。"""
required = ["format", "answer"]
clean = decide_synthetic_admission(
_candidate(), ["解释退款处理时限"], [], set(), required
)
assert clean["decision"] == "admit" and clean["reasons"] == []
repeated = decide_synthetic_admission(
_candidate(), [_candidate()["instruction"]], [], set(), required
)
assert repeated["decision"] == "review"
assert "high_training_similarity" in repeated["reasons"]
batch_duplicate = decide_synthetic_admission(
_candidate(), [], [_candidate()["instruction"]], set(), required
)
assert batch_duplicate["decision"] == "review"
assert "high_batch_similarity" in batch_duplicate["reasons"]
contaminated = decide_synthetic_admission(
_candidate(), [], [], {normalize_text(_candidate()["instruction"])}, required
)
assert contaminated["decision"] == "reject"
assert "evaluation_task_leak" in contaminated["reasons"]
failed = decide_synthetic_admission(
_candidate(verifier_results={
"format": {"passed": True, "version": "format-v1", "failure_reason": None},
"answer": {"passed": False, "version": "answer-v2", "failure_reason": "wrong_answer"},
}),
[], [], set(), required,
)
assert failed["decision"] == "reject"
assert failed["reasons"] == ["required_verifier_failed:answer"]
uncertain = decide_synthetic_admission(
_candidate(quality_score=0.70), [], [], set(), required
)
assert uncertain["decision"] == "review"
assert uncertain["reasons"] == ["quality_needs_review"]
_assert_raises(
ValueError,
lambda: decide_synthetic_admission(
_candidate(), [], [], set(), required,
quality_admit_threshold=0.5, quality_review_threshold=0.8,
),
)
def test_reject_precedes_review():
"""同时记录多项原因,并保证拒绝优先。"""
candidate = _candidate(
quality_score=0.70,
verifier_results={
"format": {"passed": False, "version": "format-v1", "failure_reason": "bad_format"},
"answer": {"passed": True, "version": "answer-v2", "failure_reason": None},
},
)
row = decide_synthetic_admission(
candidate,
[candidate["instruction"]],
[],
{normalize_text(candidate["instruction"])},
["format", "answer"],
)
assert row["decision"] == "reject"
assert set(row["reasons"]) == {
"evaluation_task_leak",
"required_verifier_failed:format",
"high_training_similarity",
"quality_needs_review",
}
def test_sft_candidate_handoff():
"""核对交接过滤、重复身份及验证器版本。"""
admitted = decide_synthetic_admission(
_candidate(), ["解释退款处理时限"], [], set(), ["format", "answer"]
)
rejected = decide_synthetic_admission(
_candidate(candidate_id="candidate-2"),
[], [], {normalize_text(_candidate()["instruction"])}, ["format", "answer"],
)
records = build_sft_candidate_records([admitted, rejected])
assert build_sft_candidate_records([rejected]) == []
assert build_sft_candidate_records([{**admitted, "decision": "review"}]) == []
assert [record["sample_id"] for record in records] == ["candidate-1"]
assert json.loads(records[0]["source_id"]) == ["refund-v1", "teacher-v2", "self-instruct-v1", "candidate-1"]
assert records[0]["split"] == "train" and records[0]["response"]
assert records[0]["verifier_versions"] == {"format": "format-v1", "answer": "answer-v2"}
_assert_raises(ValueError, lambda: build_sft_candidate_records([admitted, admitted]))
def test_empty_content_and_verifier_policy():
"""空监督、空验证策略及缺失版本均不能获得接纳。"""
required = ["format", "answer"]
for field in ("instruction", "response"):
row = decide_synthetic_admission(_candidate(**{field: " "}), [], [], set(), required)
assert row["decision"] == "reject" and f"missing_{field}" in row["reasons"]
_assert_raises(ValueError, lambda: build_generation_identity(_candidate(template_id=None)))
_assert_raises(ValueError, lambda: decide_synthetic_admission(_candidate(), [], [], set(), []))
row = decide_synthetic_admission(_candidate(verifier_results={"format": {"passed": True}}), [], [], set(), required)
assert row["decision"] == "reject"
assert set(row["reasons"]) == {"required_verifier_missing:format", "required_verifier_missing:answer"}
_assert_raises(ValueError, lambda: max_reference_similarity("文本", [], n=True))
def test_quality_threshold_boundaries():
"""质量恰好等于复核线和接纳线时,进入相应区间。"""
for score, expected in ((0.59, "reject"), (0.60, "review"), (0.799, "review"), (0.80, "admit")):
row = decide_synthetic_admission(_candidate(quality_score=score), [], [], set(), ["format", "answer"])
assert row["decision"] == expected
for score in (float("nan"), -0.1, 1.1):
_assert_raises(ValueError, lambda: decide_synthetic_admission(_candidate(quality_score=score), [], [], set(), ["format", "answer"]))
def test_handoff_evidence_and_consistency():
"""证据完整交接且独立复制,决定与原因矛盾时不得交接。"""
candidate = _candidate()
row = decide_synthetic_admission(candidate, [], [], set(), ["format", "answer"])
record = build_sft_candidate_records([row])[0]
assert record["provenance"]["generation_identity"] == list(row["generation_identity"])
assert record["provenance"]["verifier_evidence"] == candidate["verifier_results"]
row["verifier_evidence"]["answer"]["version"] = "changed"
assert record["provenance"]["verifier_evidence"]["answer"]["version"] == "answer-v2"
inconsistent = {**row, "reasons": ["evaluation_task_leak"]}
assert build_sft_candidate_records([inconsistent]) == []
# JSON编码保留身份分量边界,不因分量包含分隔符而改变来源。
special = decide_synthetic_admission(_candidate(blueprint_id="refund|v1"), [], [], set(), ["format", "answer"])
assert json.loads(build_sft_candidate_records([special])[0]["source_id"])[0] == "refund|v1"
test_generation_identity()
test_similarity_boundaries()
test_admission_decisions()
test_reject_precedes_review()
test_sft_candidate_handoff()
print("✅ 合成候选生成身份、相似度、验证器与三态准入测试通过。")
test_empty_content_and_verifier_policy()
test_quality_threshold_boundaries()
test_handoff_evidence_and_consistency()🛑 STOP HERE 🛑
请先尝试自己完成代码并跑通测试。
如果遇到困难没有思路,可以向下滚动查看参考答案。
参考代码与解析
代码
def normalize_text(text: object) -> str:
"""统一大小写与空白,生成相似度及精确任务比较形式。"""
return " ".join(str(text or "").casefold().split())
def build_generation_identity(candidate: Dict[str, object]) -> tuple[str, str, str, str]:
"""读取蓝图、生成模型、模板版本与候选标识,拒绝缺项或空值。"""
fields = ("blueprint_id", "generator_id", "template_id", "candidate_id")
identity = tuple(str(candidate.get(field) or "").strip() for field in fields)
# TODO 1:检查生成来源是否齐全
missing = [field for field, value in zip(fields, identity) if not value]
if missing:
raise ValueError(f"生成身份字段缺失:{missing}")
return identity
def character_ngrams(text: object, n: int = 2) -> set[str]:
"""提取字符片段集合,短文本保留为单个片段。"""
normalized = normalize_text(text)
if not normalized:
return set()
if len(normalized) < n:
return {normalized}
return {normalized[index:index + n] for index in range(len(normalized) - n + 1)}
def max_reference_similarity(
instruction: object, references: Iterable[object], n: int = 2
) -> float:
"""比较字符集合,返回候选与参考指令的最大重合率。"""
if type(n) is not int or n <= 0:
raise ValueError("n 必须为正整数")
candidate_ngrams = character_ngrams(instruction, n=n)
similarities = []
for reference in references:
reference_ngrams = character_ngrams(reference, n=n)
union = candidate_ngrams | reference_ngrams
# TODO 2:计算集合相似度
similarity = len(candidate_ngrams & reference_ngrams) / len(union) if union else 1.0
similarities.append(similarity)
return max(similarities, default=0.0)
def decide_synthetic_admission(
candidate: Dict[str, object],
training_references: Sequence[str],
batch_references: Sequence[str],
evaluation_tasks: set[str],
required_verifiers: Sequence[str],
similarity_review_threshold: float = 0.72,
quality_admit_threshold: float = 0.80,
quality_review_threshold: float = 0.60,
) -> Dict[str, object]:
"""保留各项风险,按拒绝、复核、接纳的优先级选择结果。"""
if not required_verifiers or any(not isinstance(name, str) or not name.strip() for name in required_verifiers):
raise ValueError("必须声明至少一个具名验证器")
if not 0.0 <= similarity_review_threshold <= 1.0:
raise ValueError("similarity_review_threshold 必须位于 [0, 1]")
if not 0.0 <= quality_review_threshold <= quality_admit_threshold <= 1.0:
raise ValueError("质量阈值必须满足 0 <= review <= admit <= 1")
generation_identity = build_generation_identity(candidate)
instruction_key = normalize_text(candidate.get("instruction", ""))
training_similarity = max_reference_similarity(
candidate.get("instruction", ""), training_references
)
batch_similarity = max_reference_similarity(
candidate.get("instruction", ""), batch_references
)
quality_score = float(candidate.get("quality_score", 0.0))
if not 0.0 <= quality_score <= 1.0:
raise ValueError("quality_score 必须位于 [0, 1]")
verifier_evidence = deepcopy(candidate.get("verifier_results") or {})
if not isinstance(verifier_evidence, dict):
raise ValueError("verifier_results 必须为字典")
reasons = []
if not instruction_key:
reasons.append("missing_instruction")
if not normalize_text(candidate.get("response")):
reasons.append("missing_response")
if instruction_key in evaluation_tasks:
reasons.append("evaluation_task_leak")
for verifier in required_verifiers:
evidence = verifier_evidence.get(verifier)
if not isinstance(evidence, dict) or not evidence.get("version"):
reasons.append(f"required_verifier_missing:{verifier}")
elif evidence.get("passed") is not True:
reasons.append(f"required_verifier_failed:{verifier}")
if not str(evidence.get("failure_reason") or "").strip():
reasons.append(f"required_verifier_reason_missing:{verifier}")
if training_similarity >= similarity_review_threshold:
reasons.append("high_training_similarity")
if batch_similarity >= similarity_review_threshold:
reasons.append("high_batch_similarity")
if quality_review_threshold <= quality_score < quality_admit_threshold:
reasons.append("quality_needs_review")
elif quality_score < quality_review_threshold:
reasons.append("quality_below_minimum")
reject_reasons = [
reason for reason in reasons
if reason in {"evaluation_task_leak", "missing_instruction", "missing_response"}
or reason.startswith("required_verifier_missing:")
or reason.startswith("required_verifier_failed:")
or reason.startswith("required_verifier_reason_missing:")
or reason == "quality_below_minimum"
]
# TODO 3:选择三态准入结果
decision = "reject" if reject_reasons else "review" if reasons else "admit"
return {
"candidate_id": generation_identity[3],
"blueprint_id": generation_identity[0],
"generator_id": generation_identity[1],
"template_id": generation_identity[2],
"instruction": str(candidate.get("instruction") or "").strip(),
"response": str(candidate.get("response") or "").strip(),
"generation_identity": generation_identity,
"training_similarity": training_similarity,
"batch_similarity": batch_similarity,
"quality_score": quality_score,
"verifier_evidence": verifier_evidence,
"decision": decision,
"reasons": reasons,
}
def build_sft_candidate_records(rows: Sequence[Dict[str, object]]) -> List[Dict[str, object]]:
"""将通过筛查的候选转为SFT记录,保留来源版本和验证结果。"""
records = []
seen_ids = set()
for row in rows:
# TODO 4:筛选可继续审计的记录
handoff_allowed = row["decision"] == "admit" and not row["reasons"]
if not handoff_allowed:
continue
sample_id = str(row["candidate_id"])
if sample_id in seen_ids:
raise ValueError(f"接纳样本身份重复:{sample_id}")
seen_ids.add(sample_id)
verifier_versions = {
name: evidence["version"]
for name, evidence in row["verifier_evidence"].items()
if isinstance(evidence, dict) and evidence.get("version")
}
records.append({
"sample_id": sample_id,
"source_id": json.dumps(list(row["generation_identity"]), ensure_ascii=False, separators=(",", ":")),
"split": "train",
"instruction": row["instruction"],
"input": "",
"response": row["response"],
"admission_decision": row["decision"],
"verifier_versions": verifier_versions,
"provenance": {
"generation_identity": list(row["generation_identity"]),
"admission_decision": row["decision"],
"verifier_evidence": deepcopy(row["verifier_evidence"]),
"quality_score": row["quality_score"],
"training_similarity": row["training_similarity"],
"batch_similarity": row["batch_similarity"],
"reasons": list(row["reasons"]),
},
})
return records解析
TODO 1:检查生成来源
- 蓝图、生成模型、模板与候选标识共同定位来源;缺少任一项时,后续验证结果就难以对应到完整生成记录。
- 根据骨架给出的四个分量找出缺项。测试更换模板版本、清空生成器或传入空值,分别检查身份变化和缺项报错。
TODO 2:计算片段集合相似度
- 共有片段占全部不同片段的比例越高,文本表面越接近;没有参考对象时最大相似度取零。
- 用交集大小除以并集大小,并在两者均空时取一,由骨架汇总最大值。测试包括完全相同、不同任务、空参考集合及非法片段长度。
TODO 3:按全部原因选择去向
- 评测重合或必需验证失败时直接拒绝;通过这些检查后,近重复或中间质量分数才进入复核。
- 先检查拒绝原因,再检查是否还有复核原因。组合用例保留污染、验证失败和近重复等所有原因;分数恰好为 0.60 与 0.80 时分别进入复核与接纳。
TODO 4:保留接纳记录及其来源
- 决定为接纳且问题列表为空,记录才可继续用于训练数据审计。生成与验证信息随样本一起复制,便于后续回查。
- 过滤复核、拒绝和“写着接纳却仍有问题”的记录。测试验证重复标识报错,并在修改原始验证信息后,确认已复制记录仍保留原值。
框架一致性核对(可选)
用同一批三态记录检查 Hugging Face Datasets 的转换与过滤:核对接纳集合、行顺序和嵌套来源信息是否保持。准入决定仍由 Step 4 的实现产生,这里核对数据接口。
| 核对对象 | Step 4 教学实现 | 成熟框架 CPU 路径 | 共同输入与通过条件 |
|---|---|---|---|
| 接纳集合 | decide_synthetic_admission 生成三态结果 | Dataset.from_list 与 Dataset.filter | 同一批结果按接纳状态过滤后,样本标识及顺序相同 |
| 来源和验证记录 | generation_identity 与验证结果 | Dataset.to_list | 转换前后的完整行、嵌套字段值与顺序相同 |
| SFT 候选字段 | build_sft_candidate_records | Dataset.from_list 与 Dataset.to_list | 同一批接纳行转换后,样本、来源、指令、回答及附带信息保持 |
from importlib.metadata import version
try:
from datasets import Dataset
except ImportError:
framework_check = {
"status": "skipped",
"reason": "请使用 PT-CPU 环境运行本单元",
}
else:
def check_synthetic_data_interface():
"""核对本地准入决定与 Datasets 过滤、血缘和SFT交接字段。"""
def framework_candidate(**overrides):
"""构造用于接口往返的三态候选。"""
base = {
"candidate_id": "candidate-1",
"blueprint_id": "refund-v1",
"generator_id": "teacher-v2",
"template_id": "self-instruct-v1",
"instruction": "说明未发货订单如何修改收货地址",
"response": "先核对订单状态,再更新地址并确认结果。",
"quality_score": 0.92,
"verifier_results": {
"format": {"passed": True, "version": "format-v1", "failure_reason": None},
"answer": {"passed": True, "version": "answer-v2", "failure_reason": None},
},
}
return {**base, **overrides}
candidates = [
framework_candidate(),
framework_candidate(candidate_id="candidate-2", instruction="解释退款处理时限"),
framework_candidate(
candidate_id="candidate-3",
instruction="列出账户注销步骤",
verifier_results={
"format": {"passed": True, "version": "format-v1", "failure_reason": None},
"answer": {"passed": False, "version": "answer-v2", "failure_reason": "wrong_answer"},
},
),
]
references = ["解释退款处理时限"]
rows = [
decide_synthetic_admission(
candidate, references, [], set(), ["format", "answer"]
)
for candidate in candidates
]
serializable_rows = [
{
**row,
"generation_identity": list(row["generation_identity"]),
"reasons": list(row["reasons"]),
}
for row in rows
]
dataset = Dataset.from_list(serializable_rows)
admitted_dataset = dataset.filter(lambda row: row["decision"] == "admit")
local_admitted_ids = [row["candidate_id"] for row in rows if row["decision"] == "admit"]
framework_admitted_ids = admitted_dataset["candidate_id"]
sft_records = build_sft_candidate_records(rows)
sft_dataset = Dataset.from_list(sft_records)
checks = {
"admitted_ids_match": framework_admitted_ids == local_admitted_ids,
"rows_values_and_order_preserved": dataset.to_list() == serializable_rows,
"handoff_values_preserved": sft_dataset.to_list() == sft_records,
"lineage_fields_preserved": all(
row["blueprint_id"] and row["generator_id"] and row["template_id"]
for row in admitted_dataset
),
"generation_identity_preserved": all(
len(row["generation_identity"]) == 4
for row in admitted_dataset
),
"sft_handoff_fields_readable": all(
row["sample_id"] and row["source_id"] and row["split"] == "train"
and row["instruction"] and row["response"]
for row in sft_dataset
),
}
failed = [name for name, passed in checks.items() if not passed]
if failed:
raise AssertionError(f"合成数据框架核对失败:{failed}")
return {
"status": "passed",
"check_kind": "data_interface",
"versions": {"datasets": version("datasets")},
"checks": checks,
"candidate_rows": len(dataset),
"admitted_rows": len(admitted_dataset),
}
framework_check = check_synthetic_data_interface()
framework_check相关阅读
继续比较合成方法,再核对候选怎样进入实际数据处理。
- Self-Instruct 论文:观察种子任务、生成与过滤怎样共同影响数据分布。
- 32 SFT 数据契约与泄漏审计:接续检查候选字段、完整训练样本重复与评测任务重合。
- 64 SFT 数据质量项目:了解固定分词、清洗、切分与监督数量检查怎样形成可用数据清单。
