Skip to content

PERF-REGRESSION. Paired Benchmark and Regression Thresholds | 配对基准与性能回归阈值 ​

难度: Medium | 环境: CPU基础练习,GPU可选 | 标签: 性能分析、配对基准、回归阈值 | 目标人群: 具备性能测试基础的读者

🚀 云端运行环境

本章节的实战代码可以点击以下链接在免费 GPU 算力平台上直接运行:

Open In ColabOpen In Studio (国内推荐:魔搭社区免费实例)

本节导读 ​

同一任务改动后变慢,称为性能回归。但一次候选运行更快或更慢,也可能是温度、执行顺序或后台工作改变。把同一轮的基线与候选配成一对,可以保留轮次差异,再判断变化是否超过事先允许的幅度。

本节以延迟为例,先核对哪些运行能够配对,再从逐对比例估计典型变化及其不确定性,最后决定通过、回退或补测。比如允许慢5%时,通过检查只说明当前数据满足这条要求,并不等于已经加速。

关键词: 配对试验、延迟比例、自助重采样、中位数、性能回归

运行环境 ​

前置阅读 ​

先学习如何固定工作负载、计时范围和重复运行,再进入配对统计。

Step 1:哪些运行可以配成一对 ​

一对试验使用相同输入、模型权重、环境与计时范围,只改变待比较的方案。配对编号连接同一轮的基线与候选。不同轮次交替执行顺序,以减少固定先后带来的偏差;同时记录温度、设备状态和后台负载,分析跨轮波动。

基线与候选任一侧缺失、失败或计时无效时,该轮不能进入成功比例分布,仍需计入试验记录。若两侧都没有留下记录,仅靠编号并集无法知道它曾被计划;因此还要保留计划轮数,与完成轮数核对。

信息配对要求不满足时怎样处理
试验编号每侧唯一,连接同一轮运行重复编号拒绝;单侧缺失单独登记
固定条件标识编码输入、形状、模型及环境等固定条件;本次汇总只对应同一组固定条件任一有效运行固定条件不同则拒绝合并
延迟两侧均为有限正数,单位相同非法计时拒绝,不生成比例
执行状态两侧均成功才能比较时间失败保留编号;不能仅凭剩余成功样本采用候选
计划与完成轮数计划量独立保存,不能由成功数倒推缺失需修复或补测;当前严格规则不允许失败运行通过

配对运行与缺失失败记录

Step 2:从逐对延迟估计中位比例的范围 ​

设 N 个成功配对中,第 i 对的基线与候选延迟为 bi,ci,均为同单位正数。先算候选除以基线的比例 ri,再取其中位数 r^。例如基线10ms、候选11ms,比例为1.1,表示该轮慢10%;变慢占比 s 则统计有多少轮比例大于1。 1[⋅] 在条件成立时取1,否则取0。

ri=cibi,r^=median(r1,…,rN),s=1N∑i=1N1[ri>1].

自助重采样(Bootstrap)从已观测比例中有放回抽取 N 个数,每次求中位数;重复 K 次,再取这些中位数的5%与95%分位数,得到90%百分位区间 [L,U]。例如两个比例0.9和1.05,重采样可以重复抽到0.9,也可以各抽一个;这里重采样的是整对比例,不是拆散两侧时间。 Qp 表示分位函数,本节用排序后最接近的位置取值。

L=Q0.05(r^1∗,…,r^K∗),U=Q0.95(r^1∗,…,r^K∗).
统计对象回答什么问题解释时注意什么
逐对比例的中位数典型配对快还是慢一般不同于两组延迟中位数之比
中位比例的90%自助区间当前样本下中位比例有多不确定不是包含90%单次延迟的范围,也不是所有负载的联合保证
变慢占比成功配对中有多少轮更慢不表示变慢幅度,失败另计
独立重采样假设每对能否当作独立观测持续漂移或轮次相关会削弱区间解释
重采样次数与种子统计计算是否可重复增加计算次数不能代替新增真实配对

Step 3:区间相对允许回归边界怎样决定结果 ​

设事先允许的相对延迟增加量为 δ≥0,例如 δ=0.05 表示允许慢 5%;对应比例边界 τ 为:

τ=1+δ.

先检查正确性与执行失败,再检查缺失和有效配对数量,最后将中位比例区间 [L,U] 与 τ 比较。当前教学规则对执行失败零容忍,缺失则要求补齐;多个负载分别判断,任一拒绝则整体拒绝,其余存在待复测项则整体继续复测。 U≤τ表示通过本次允许变慢检查;是否更快,还要另与比例1比较。

条件,按顺序判断结果下一步
正确性未通过或存在执行失败reject修复或回退,保留失败记录
存在缺失配对或有效数量不足tune补齐运行后重新统计
L>τreject区间整体超过允许变慢幅度,回退候选
U≤τaccept通过当前回归检查,再核对其他任务要求
其他位置,包括 L=τ<Utune区间未能明确落在边界一侧,固定条件复测

性能回归的顺序判断

Step 4:实现配对检查、重采样与回归判断 ​

输入基线和候选的运行记录,先检查唯一编号与固定条件标识(workload_signature),再补全配对分支、比例重采样和决策条件。骨架负责字段校验与失败清单;决策函数同时接收成功样本的区间、失败数量和缺失数量。成功配对为空时,报告无法统计,并保留失败与缺失原因,供补测时定位。

TODO / 函数学习者完成的机制必须满足的约束测试证据
TODO 1:建立配对运行
pair_benchmark_runs
区分成功、缺失和失败编号唯一;有效记录同一签名;保留全部已见轮次乱序配对、两侧缺失、执行失败、重复编号、签名冲突
TODO 2:汇总比例分布
summarize_paired_ratios
逐对求比值,并有放回重采样中位数延迟有限为正;固定种子;每次抽取原样本量已知比例、单样本、固定种子复现、非法计时
TODO 3:判断性能回归
decide_performance_regression
按质量、失败、完整性和区间顺序判断失败优先拒绝;不足与跨界继续复测;允许边界相等三种决策、失败优先级、最小样本量和阈值等号
python
from math import isfinite
from random import Random
from statistics import median
from typing import Dict, List, Sequence
python
def _finite_latency(value):
    """要求真实计时为有限正数,不把布尔值当作时间。"""
    if isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value) or value <= 0:
        raise ValueError("延迟必须为有限正数")
    return float(value)


def _index_runs(runs):
    indexed = {}
    for run in runs:
        trial_id = run["trial_id"]
        if not isinstance(trial_id, int) or isinstance(trial_id, bool) or trial_id < 0:
            raise ValueError("试验编号必须为非负整数")
        if trial_id in indexed:
            raise ValueError("同侧试验编号重复")
        if not isinstance(run.get("workload_signature"), str) or not run["workload_signature"].strip():
            raise ValueError("工作负载签名不能为空")
        if "failure" not in run:
            raise ValueError("必须明确登记执行状态")
        if not run["failure"]:
            _finite_latency(run["latency_ms"])
        indexed[trial_id] = run
    return indexed


def pair_benchmark_runs(baseline_runs, candidate_runs):
    """配对同一负载的运行,同时保留缺失和执行失败。"""
    baseline, candidate = _index_runs(baseline_runs), _index_runs(candidate_runs)
    signatures = {run["workload_signature"] for run in list(baseline.values()) + list(candidate.values())}
    if len(signatures) > 1:
        raise ValueError("本次汇总必须使用同一工作负载签名")
    pairs = []
    excluded = {"missing_baseline": [], "missing_candidate": [], "failed": []}
    failed_runs = [{"side": side, "trial_id": trial_id, "failure": run["failure"]}
                   for side, runs in (("baseline", baseline), ("candidate", candidate))
                   for trial_id, run in sorted(runs.items()) if run["failure"]]
    for trial_id in sorted(set(baseline) | set(candidate)):
        # TODO 1:建立配对运行
        # missing_baseline = ???
        # missing_candidate = ???
        # failed_pair = ???
        raise NotImplementedError("TODO 1:请建立配对运行")
        if missing_baseline:
            excluded["missing_baseline"].append(trial_id)
            continue
        if missing_candidate:
            excluded["missing_candidate"].append(trial_id)
            continue
        if failed_pair:
            excluded["failed"].append(trial_id)
            continue
        pairs.append({"trial_id": trial_id,
                      "baseline_ms": float(baseline[trial_id]["latency_ms"]),
                      "candidate_ms": float(candidate[trial_id]["latency_ms"])})
    return {"pairs": pairs, "excluded": excluded, "failed_runs": failed_runs,
            "observed_trials": len(set(baseline) | set(candidate))}


def _quantile(values, probability):
    """对排序样本取最接近的离散位置,与 GPU 统计保持相同约定。"""
    ordered = sorted(values)
    index = int(round((len(ordered) - 1) * probability))
    return float(ordered[index])


def summarize_paired_ratios(pairs, bootstrap_samples=1000, seed=42):
    """对逐对比例进行独立重采样,返回中位数的90%百分位区间。"""
    if not pairs or not isinstance(bootstrap_samples, int) or isinstance(bootstrap_samples, bool) or bootstrap_samples <= 0:
        raise ValueError("需要成功配对与正整数重采样次数")
    if not isinstance(seed, int) or isinstance(seed, bool):
        raise ValueError("随机种子必须为整数")
    for pair in pairs:
        baseline_value = _finite_latency(pair["baseline_ms"])
        candidate_value = _finite_latency(pair["candidate_ms"])
        _finite_latency(candidate_value / baseline_value)
    rng = Random(seed)
    # TODO 2:汇总比例分布
    # ratios = ???
    # bootstrap_medians = ???
    # ci_low = ???
    # ci_high = ???
    raise NotImplementedError("TODO 2:请汇总配对比例")
    return {"pair_count": len(ratios), "median_ratio": float(median(ratios)),
            "slower_fraction": sum(ratio > 1.0 for ratio in ratios) / len(ratios),
            "ci_low": ci_low, "ci_high": ci_high}


def decide_performance_regression(summary, quality_passed, regression_threshold, minimum_pairs,
                                  failed_pairs=0, missing_pairs=0):
    """依次判断质量、失败、配对完整性、样本量及允许回归边界。"""
    if type(quality_passed) is not bool:
        raise ValueError("质量状态必须为布尔值")
    if isinstance(regression_threshold, bool) or not isinstance(regression_threshold, (int, float)) or not isfinite(regression_threshold) or regression_threshold < 0:
        raise ValueError("允许回归量必须有限且非负")
    for value in (minimum_pairs, failed_pairs, missing_pairs, summary["pair_count"]):
        if not isinstance(value, int) or isinstance(value, bool) or value < 0:
            raise ValueError("样本和失败计数必须为非负整数")
    if minimum_pairs == 0:
        raise ValueError("最少配对数必须为正")
    low, high = summary["ci_low"], summary["ci_high"]
    if any(isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value) or value <= 0
           for value in (low, high)) or low > high:
        raise ValueError("区间必须有限、为正且按低到高排列")
    boundary = 1.0 + regression_threshold
    # TODO 3:判断性能回归
    # quality_failed = ???
    # insufficient_pairs = ???
    # confirmed_regression = ???
    # confirmed_safe = ???
    raise NotImplementedError("TODO 3:请作出回归判断")
    if quality_failed:
        decision, reason = "reject", "execution_failed" if failed_pairs else "quality_guard_failed"
    elif insufficient_pairs:
        decision, reason = "tune", "incomplete_pairs" if missing_pairs else "insufficient_pairs"
    elif confirmed_regression:
        decision, reason = "reject", "latency_regression"
    elif confirmed_safe:
        decision, reason = "accept", "no_confirmed_regression"
    else:
        decision, reason = "tune", "interval_crosses_threshold"
    return {"decision": decision, "reason": reason, "regression_boundary": boundary}

测试 ​

python
def _runs(latencies, signature="shape-a", failed_trial=None):
    return [{"trial_id": trial_id, "workload_signature": signature, "latency_ms": latency,
             "failure": "oom" if trial_id == failed_trial else None}
            for trial_id, latency in enumerate(latencies)]


def _expect_regression_error(function, *args, **kwargs):
    try:
        function(*args, **kwargs)
    except ValueError:
        return
    raise AssertionError("非法数据未被拒绝")


def test_pairing_and_exclusions():
    baseline = _runs([10, 11, 9])
    candidate = _runs([10.5, 12], failed_trial=1)
    report = pair_benchmark_runs(list(reversed(baseline)), candidate)
    assert [p["trial_id"] for p in report["pairs"]] == [0]
    assert report["excluded"] == {"missing_baseline": [], "missing_candidate": [2], "failed": [1]}
    assert report["observed_trials"] == 3
    assert report["failed_runs"] == [{"side": "candidate", "trial_id": 1, "failure": "oom"}]
    reverse = pair_benchmark_runs(candidate, baseline)
    assert reverse["excluded"]["missing_baseline"] == [2]
    assert pair_benchmark_runs([], [])["observed_trials"] == 0


def test_duplicate_and_signature_checks():
    runs = _runs([10])
    _expect_regression_error(pair_benchmark_runs, runs + runs, runs)
    _expect_regression_error(pair_benchmark_runs, runs, _runs([10], signature="other"))
    for invalid in (True, 0.5, -1):
        _expect_regression_error(pair_benchmark_runs, [{**runs[0], "trial_id": invalid}], runs)
    _expect_regression_error(pair_benchmark_runs, [{**runs[0], "workload_signature": ""}], runs)


def test_known_ratios_and_reproducibility():
    pairs = [{"baseline_ms": 10, "candidate_ms": value} for value in (9, 10, 11)]
    first = summarize_paired_ratios(pairs, 200, 7)
    assert first == summarize_paired_ratios(pairs, 200, 7)
    assert first["pair_count"] == 3 and first["median_ratio"] == 1
    assert first["slower_fraction"] == 1 / 3
    assert first["ci_low"] == 0.9 and first["ci_high"] == 1.1
    single = summarize_paired_ratios([{"baseline_ms": 10, "candidate_ms": 9}], 50)
    assert single["ci_low"] == single["ci_high"] == single["median_ratio"] == 0.9
    # 逐对比例的中位数不能被两组中位数之比代替。
    different = [{"baseline_ms": b, "candidate_ms": c} for b, c in ((1, 100), (10, 1), (100, 10))]
    assert summarize_paired_ratios(different, 50)["median_ratio"] == 0.1
    assert median(p["candidate_ms"] for p in different) / median(p["baseline_ms"] for p in different) == 1.0


def test_invalid_statistical_inputs():
    for invalid in (0, -1, True, float("nan"), float("inf")):
        for key in ("baseline_ms", "candidate_ms"):
            pair = {"baseline_ms": 10, "candidate_ms": 10, key: invalid}
            _expect_regression_error(summarize_paired_ratios, [pair])
    # 两侧有限为正,仍可能在求比值时溢出或下溢。
    for baseline, candidate in ((1e-308, 1e308), (1e308, 1e-308)):
        _expect_regression_error(summarize_paired_ratios,
                                 [{"baseline_ms": baseline, "candidate_ms": candidate}])
    _expect_regression_error(summarize_paired_ratios, [])
    for invalid in (0, -1, True, 1.5):
        _expect_regression_error(summarize_paired_ratios, [{"baseline_ms": 1, "candidate_ms": 1}], invalid)


def test_decision_regions_and_equalities():
    for low, high, expected in ((0.98, 1.03, "accept"), (1.08, 1.12, "reject"),
                                (1.01, 1.08, "tune"), (1.0, 1.05, "accept"),
                                (1.05, 1.06, "tune"), (1.05, 1.05, "accept")):
        summary = {"pair_count": 20, "ci_low": low, "ci_high": high}
        assert decide_performance_regression(summary, True, 0.05, 20)["decision"] == expected


def test_quality_failure_and_missing_priority():
    summary = {"pair_count": 20, "ci_low": 0.9, "ci_high": 0.95}
    assert decide_performance_regression(summary, False, 0.05, 30)["reason"] == "quality_guard_failed"
    assert decide_performance_regression(summary, True, 0.05, 10, failed_pairs=1)["reason"] == "execution_failed"
    assert decide_performance_regression(summary, True, 0.05, 10, missing_pairs=1)["decision"] == "tune"
    assert decide_performance_regression(summary, True, 0.05, 21)["reason"] == "insufficient_pairs"


def test_failure_counts_reach_decision():
    report = pair_benchmark_runs(_runs([10] * 21), _runs([9] * 21, failed_trial=20))
    summary = summarize_paired_ratios(report["pairs"], 100)
    # 20 个较快成功样本不能掩盖第 21 轮执行失败。
    decision = decide_performance_regression(summary, True, 0.05, 20,
        failed_pairs=len(report["failed_runs"]),
        missing_pairs=len(report["excluded"]["missing_baseline"]) + len(report["excluded"]["missing_candidate"]))
    assert decision["decision"] == "reject"


def test_invalid_decision_inputs():
    summary = {"pair_count": 20, "ci_low": 0.98, "ci_high": 1.03}
    for invalid in (float("nan"), float("inf"), -0.1, True):
        _expect_regression_error(decide_performance_regression, summary, True, invalid, 10)
    for change in ({"pair_count": 1.5}, {"ci_low": float("nan")}, {"ci_high": 0.9}):
        _expect_regression_error(decide_performance_regression, {**summary, **change}, True, 0.05, 10)
    _expect_regression_error(decide_performance_regression, summary, True, 0.05, 0)


def run_perf_regression_tests():
    """核对配对、统计与决策的八组行为。"""
    for test in (test_pairing_and_exclusions, test_duplicate_and_signature_checks,
                 test_known_ratios_and_reproducibility, test_invalid_statistical_inputs,
                 test_decision_regions_and_equalities, test_quality_failure_and_missing_priority,
                 test_failure_counts_reach_decision, test_invalid_decision_inputs):
        test()
    print("✅ 配对完整性、自助区间与回归判断测试通过")


run_perf_regression_tests()

🛑 STOP HERE 🛑









请先尝试自己完成代码并跑通测试。
如果遇到困难,再向下查看参考代码与解析。










参考代码与解析 ​

代码 ​

python
def _finite_latency(value):
    """要求真实计时为有限正数,不把布尔值当作时间。"""
    if isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value) or value <= 0:
        raise ValueError("延迟必须为有限正数")
    return float(value)


def _index_runs(runs):
    indexed = {}
    for run in runs:
        trial_id = run["trial_id"]
        if not isinstance(trial_id, int) or isinstance(trial_id, bool) or trial_id < 0:
            raise ValueError("试验编号必须为非负整数")
        if trial_id in indexed:
            raise ValueError("同侧试验编号重复")
        if not isinstance(run.get("workload_signature"), str) or not run["workload_signature"].strip():
            raise ValueError("工作负载签名不能为空")
        if "failure" not in run:
            raise ValueError("必须明确登记执行状态")
        if not run["failure"]:
            _finite_latency(run["latency_ms"])
        indexed[trial_id] = run
    return indexed


def pair_benchmark_runs(baseline_runs, candidate_runs):
    """配对同一负载的运行,同时保留缺失和执行失败。"""
    baseline, candidate = _index_runs(baseline_runs), _index_runs(candidate_runs)
    signatures = {run["workload_signature"] for run in list(baseline.values()) + list(candidate.values())}
    if len(signatures) > 1:
        raise ValueError("本次汇总必须使用同一工作负载签名")
    pairs = []
    excluded = {"missing_baseline": [], "missing_candidate": [], "failed": []}
    failed_runs = [{"side": side, "trial_id": trial_id, "failure": run["failure"]}
                   for side, runs in (("baseline", baseline), ("candidate", candidate))
                   for trial_id, run in sorted(runs.items()) if run["failure"]]
    for trial_id in sorted(set(baseline) | set(candidate)):
        # TODO 1:建立配对运行
        missing_baseline = trial_id not in baseline
        missing_candidate = trial_id not in candidate
        failed_pair = (not missing_baseline and not missing_candidate
                       and bool(baseline[trial_id]["failure"] or candidate[trial_id]["failure"]))
        if missing_baseline:
            excluded["missing_baseline"].append(trial_id)
            continue
        if missing_candidate:
            excluded["missing_candidate"].append(trial_id)
            continue
        if failed_pair:
            excluded["failed"].append(trial_id)
            continue
        pairs.append({"trial_id": trial_id,
                      "baseline_ms": float(baseline[trial_id]["latency_ms"]),
                      "candidate_ms": float(candidate[trial_id]["latency_ms"])})
    return {"pairs": pairs, "excluded": excluded, "failed_runs": failed_runs,
            "observed_trials": len(set(baseline) | set(candidate))}


def _quantile(values, probability):
    """对排序样本取最接近的离散位置,与 GPU 统计保持相同约定。"""
    ordered = sorted(values)
    index = int(round((len(ordered) - 1) * probability))
    return float(ordered[index])


def summarize_paired_ratios(pairs, bootstrap_samples=1000, seed=42):
    """对逐对比例进行独立重采样,返回中位数的90%百分位区间。"""
    if not pairs or not isinstance(bootstrap_samples, int) or isinstance(bootstrap_samples, bool) or bootstrap_samples <= 0:
        raise ValueError("需要成功配对与正整数重采样次数")
    if not isinstance(seed, int) or isinstance(seed, bool):
        raise ValueError("随机种子必须为整数")
    for pair in pairs:
        baseline_value = _finite_latency(pair["baseline_ms"])
        candidate_value = _finite_latency(pair["candidate_ms"])
        _finite_latency(candidate_value / baseline_value)
    rng = Random(seed)
    # TODO 2:汇总比例分布
    ratios = [float(pair["candidate_ms"]) / float(pair["baseline_ms"]) for pair in pairs]
    if any(not isfinite(ratio) or ratio <= 0 for ratio in ratios):
        raise ValueError("比例超出有效数值范围")
    bootstrap_medians = [median(rng.choices(ratios, k=len(ratios))) for _ in range(bootstrap_samples)]
    ci_low = _quantile(bootstrap_medians, 0.05)
    ci_high = _quantile(bootstrap_medians, 0.95)
    return {"pair_count": len(ratios), "median_ratio": float(median(ratios)),
            "slower_fraction": sum(ratio > 1.0 for ratio in ratios) / len(ratios),
            "ci_low": ci_low, "ci_high": ci_high}


def decide_performance_regression(summary, quality_passed, regression_threshold, minimum_pairs,
                                  failed_pairs=0, missing_pairs=0):
    """依次判断质量、失败、配对完整性、样本量及允许回归边界。"""
    if type(quality_passed) is not bool:
        raise ValueError("质量状态必须为布尔值")
    if isinstance(regression_threshold, bool) or not isinstance(regression_threshold, (int, float)) or not isfinite(regression_threshold) or regression_threshold < 0:
        raise ValueError("允许回归量必须有限且非负")
    for value in (minimum_pairs, failed_pairs, missing_pairs, summary["pair_count"]):
        if not isinstance(value, int) or isinstance(value, bool) or value < 0:
            raise ValueError("样本和失败计数必须为非负整数")
    if minimum_pairs == 0:
        raise ValueError("最少配对数必须为正")
    low, high = summary["ci_low"], summary["ci_high"]
    if any(isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value) or value <= 0
           for value in (low, high)) or low > high:
        raise ValueError("区间必须有限、为正且按低到高排列")
    boundary = 1.0 + regression_threshold
    # TODO 3:判断性能回归
    quality_failed = not quality_passed or failed_pairs > 0
    insufficient_pairs = missing_pairs > 0 or summary["pair_count"] < minimum_pairs
    confirmed_regression = low > boundary
    confirmed_safe = high <= boundary
    if quality_failed:
        decision, reason = "reject", "execution_failed" if failed_pairs else "quality_guard_failed"
    elif insufficient_pairs:
        decision, reason = "tune", "incomplete_pairs" if missing_pairs else "insufficient_pairs"
    elif confirmed_regression:
        decision, reason = "reject", "latency_regression"
    elif confirmed_safe:
        decision, reason = "accept", "no_confirmed_regression"
    else:
        decision, reason = "tune", "interval_crosses_threshold"
    return {"decision": decision, "reason": reason, "regression_boundary": boundary}

解析 ​

TODO 1:同时保留成功、缺失与失败 ​

  • 两侧先各自检查编号唯一,再按相同编号连接;重复编号不能靠字典覆盖,执行失败也不能当成记录缺失。
  • 测试核对乱序配对、单侧缺失和失败明细,并用固定条件不同的记录检查是否错误混合比较。

TODO 2:对完整配对的比例重采样 ​

  • 先逐对求候选/基线,再有放回抽取同样数量的比例,计算每次中位数;不能把两侧分别打乱,或改成重采样均值。
  • 已知比例、单样本与固定种子测试检查结果,非法计时测试阻止无效数值进入区间。

TODO 3:先检查可用性,再比较区间 ​

  • 正确性不通过或执行失败时拒绝;缺失或成功数量不足时补测;其余再比较区间与允许边界。上界等于边界时通过,下界等于边界而上界超过时仍需补测。
  • 集成用例包含20个更快的成功配对和1次失败,检查是否把失败传入决策;仅统计成功样本会错误放行。

Step 5:GPU 机制实验——普通执行与编译执行的配对复测 ​

CPU 练习验证配对与统计规则;GPU 实验在同一微型多层感知机、相同权重与输入上交替运行普通执行和图编译入口torch.compile生成的执行方案。首次执行单独计时,避免把编译准备成本混入稳态比例。

学习问题CPU 已确认GPU 要确认与后续使用
编译方案是否通过各负载的允许回归检查成功配对、失败缺失记录、中位比例区间与判断顺序逐负载首次调用、稳态配对延迟与首次输出一致性;通过不等于加速,不能外推完整模型或服务收益

5.1 环境与固定工作负载 ​

本实验使用 PERF-CUDA-MECHANISM,首次运行按性能机制GPU环境创建并预检环境,结果登记见实验资产表。

配置层本节固定设置修改规则
运行环境单张 CUDA GPU;PERF-CUDA-MECHANISMProfile 预检未通过时停止
模型与输入本机随机初始化 TinyMLP,固定种子 42;1、32、256 三档 Token;无外部下载相同权重与输入下比较普通执行和编译执行;改变模型、负载或编译选项后使用新标签
配对协议每档先记录首次调用,再进行每条路径 10 次预热、30 对交替顺序复测首次执行使用同步墙钟,稳态使用 CUDA 事件时间;两者不能直接合并
正确性各档首次输出 allclose(atol=1e-3, rtol=1e-3)当前不是逐对输出核对;未验证不同输入与完整模型质量
统计方法2000 次配对比例重采样,90% 百分位区间;种子为 42+Token 数各负载分别估计,没有多负载同时置信保证
决策门槛正确性通过、无执行失败、完成计划配对、至少 20 对、允许慢 5%任一负载拒绝则整体拒绝;有待复测且无拒绝时整体继续复测
结果资产benchmarks/results/perf_regression/<workload_id>_<run_label>.json同标签不覆盖;5.3 可独立读取
python
RUN_MODE = "dry_run"  # 默认值 dry_run;改为 real_gpu 后执行真实配对复测。
RUN_LABEL = "default"  # 默认值 default;正式复测使用唯一标签。
ENVIRONMENT_PROFILE = "PERF-CUDA-MECHANISM"  # 预检必须通过的环境标识。
SEMANTIC_ID = "PERF-REGRESSION"  # 机制页稳定标识。
WORKLOAD_ID = "perf-paired-regression"  # 稳定工作负载标识。
WORKLOAD_VERSION = "v4"  # v4 增加多 Token 负载、首次调用成本和逐负载决策。
TOKEN_COUNTS = [1, 32, 256]  # 默认 Token 负载矩阵;同时覆盖低负载噪声区和较高计算负载。
HIDDEN_SIZE = 2048  # 默认隐藏维度;显存不足时下调。
WARMUP_STEPS = 10  # 每条路径的预热次数。
PAIRED_TRIALS = 30  # 默认配对试验数。
REGRESSION_THRESHOLD = 0.05  # 默认允许 5% 延迟回归。
MINIMUM_PAIRS = 20  # 作出 accept/reject 前要求的最少合法配对数。
BOOTSTRAP_SAMPLES = 2000  # 自助重采样次数;增大后区间更稳定但统计时间更长。
SEED = 42  # 固定输入和自助重采样的随机种子。
RESULT_RELATIVE_DIR = "benchmarks/results/perf_regression"  # 结果 JSON 的仓库相对目录。

5.2 执行配对复测并保存 JSON ​

默认 dry_run 检查配置;预检通过后改为 real_gpu 执行配对测量。中途失败时保存已完成配对和失败阶段,该负载判为 reject。

python
from pathlib import Path
import json
import sys
from random import Random
from statistics import median
from time import perf_counter
import subprocess
from math import isfinite


def _gpu_quantile(values, probability):
    ordered = sorted(values)
    index = int(round((len(ordered) - 1) * probability))
    return float(ordered[index])


def _gpu_summarize_paired_ratios(pairs, bootstrap_samples, seed):
    """按 CPU 练习的同一重采样约定独立统计真实计时。"""
    if not pairs or not isinstance(bootstrap_samples, int) or isinstance(bootstrap_samples, bool) or bootstrap_samples <= 0:
        raise ValueError("需要成功配对与正整数重采样次数")
    if any(isinstance(pair[key], bool) or not isinstance(pair[key], (int, float))
           or not isfinite(pair[key]) or pair[key] <= 0
           for pair in pairs for key in ("baseline_ms", "candidate_ms")):
        raise ValueError("每侧延迟必须为有限正数")
    ratios = [float(pair["candidate_ms"]) / float(pair["baseline_ms"]) for pair in pairs]
    if not ratios or any(not isfinite(ratio) or ratio <= 0 for ratio in ratios):
        raise ValueError("配对延迟必须为正且不能为空")
    rng = Random(seed)
    bootstrap_medians = [median(rng.choices(ratios, k=len(ratios))) for _ in range(bootstrap_samples)]
    return {
        "pair_count": len(ratios),
        "median_ratio": float(median(ratios)),
        "slower_fraction": sum(ratio > 1.0 for ratio in ratios) / len(ratios),
        "ci_low": _gpu_quantile(bootstrap_medians, 0.05),
        "ci_high": _gpu_quantile(bootstrap_medians, 0.95),
    }


def _gpu_decide_regression(summary, quality_passed, threshold, minimum_pairs):
    boundary = 1.0 + threshold
    if not quality_passed:
        decision, reason = "reject", "quality_guard_failed"
    elif summary["pair_count"] < minimum_pairs:
        decision, reason = "tune", "insufficient_pairs"
    elif summary["ci_low"] > boundary:
        decision, reason = "reject", "latency_regression"
    elif summary["ci_high"] <= boundary:
        decision, reason = "accept", "no_confirmed_regression"
    else:
        decision, reason = "tune", "interval_crosses_threshold"
    return {"decision": decision, "reason": reason, "regression_boundary": boundary}

if RUN_MODE not in {"dry_run", "real_gpu"}:
    raise ValueError("RUN_MODE 只能为 dry_run 或 real_gpu")
for name, value in {"HIDDEN_SIZE": HIDDEN_SIZE, "WARMUP_STEPS": WARMUP_STEPS,
                    "PAIRED_TRIALS": PAIRED_TRIALS, "MINIMUM_PAIRS": MINIMUM_PAIRS,
                    "BOOTSTRAP_SAMPLES": BOOTSTRAP_SAMPLES}.items():
    if not isinstance(value, int) or isinstance(value, bool) or value <= 0:
        raise ValueError(f"{name} 必须为正整数")
if not TOKEN_COUNTS or any(not isinstance(value, int) or isinstance(value, bool) or value <= 0
                           for value in TOKEN_COUNTS) or len(set(TOKEN_COUNTS)) != len(TOKEN_COUNTS):
    raise ValueError("Token 负载必须为不重复的正整数")
if isinstance(REGRESSION_THRESHOLD, bool) or not isinstance(REGRESSION_THRESHOLD, (int, float)) or not isfinite(REGRESSION_THRESHOLD) or REGRESSION_THRESHOLD < 0:
    raise ValueError("允许回归量必须有限且非负")

if RUN_MODE == "real_gpu":
    import torch
    from tools.project_runtime import resolve_project_root, save_json_record_once

    if not RUN_LABEL or RUN_LABEL == "default" or Path(RUN_LABEL).name != RUN_LABEL:
        raise ValueError("正式运行需要不含目录分隔符的唯一 RUN_LABEL")
    project_root = resolve_project_root()
    result_dir = project_root / RESULT_RELATIVE_DIR
    result_path = result_dir / f"{WORKLOAD_ID}_{RUN_LABEL}.json"
    if result_path.exists():
        raise FileExistsError("同标签结果已存在:读取已有结果,或更换 RUN_LABEL")
    subprocess.run([sys.executable, str(project_root / "tools/environment_preflight.py"),
                    "--profile", ENVIRONMENT_PROFILE], check=True)
    if not torch.cuda.is_available():
        raise RuntimeError("PERF-REGRESSION 真实实验需要 CUDA GPU")

    class TinyMLP(torch.nn.Module):
        def __init__(self, hidden_size):
            super().__init__()
            self.up = torch.nn.Linear(hidden_size, hidden_size * 2, bias=False)
            self.down = torch.nn.Linear(hidden_size * 2, hidden_size, bias=False)

        def forward(self, inputs):
            return self.down(torch.nn.functional.silu(self.up(inputs)))

    def measure_steady(function, inputs):
        start = torch.cuda.Event(enable_timing=True)
        end = torch.cuda.Event(enable_timing=True)
        start.record()
        with torch.no_grad():
            output = function(inputs)
        end.record()
        end.synchronize()
        return float(start.elapsed_time(end)), output

    def measure_first_execution(function, inputs):
        torch.cuda.synchronize()
        started_at = perf_counter()
        with torch.no_grad():
            output = function(inputs)
        torch.cuda.synchronize()
        return (perf_counter() - started_at) * 1000.0, output

    torch.manual_seed(SEED)
    torch.cuda.manual_seed_all(SEED)
    device = torch.device("cuda")
    eager = TinyMLP(HIDDEN_SIZE).to(device).eval()
    load_results = []
    for token_count in TOKEN_COUNTS:
        inputs = torch.randn(token_count, HIDDEN_SIZE, device=device)
        pairs = []
        load_failure = None
        quality_passed = False
        first_execution = None
        phase = "first_execution"
        attempted_trial_id = None
        try:
            candidate = torch.compile(eager, dynamic=False)
            eager_first_ms, eager_first_output = measure_first_execution(eager, inputs)
            candidate_first_ms, candidate_first_output = measure_first_execution(candidate, inputs)
            first_execution = {
                "eager_ms": eager_first_ms,
                "compile_ms": candidate_first_ms,
                "compile_to_eager_ratio": candidate_first_ms / eager_first_ms,
            }
            quality_passed = bool(torch.allclose(eager_first_output, candidate_first_output, atol=1e-3, rtol=1e-3))
            phase = "warmup"
            for _ in range(WARMUP_STEPS):
                measure_steady(eager, inputs)
                measure_steady(candidate, inputs)
            phase = "steady_pairs"
            for trial_id in range(PAIRED_TRIALS):
                attempted_trial_id = trial_id
                first, second = (eager, candidate) if trial_id % 2 == 0 else (candidate, eager)
                first_ms, _ = measure_steady(first, inputs)
                second_ms, _ = measure_steady(second, inputs)
                eager_ms, candidate_ms = (first_ms, second_ms) if trial_id % 2 == 0 else (second_ms, first_ms)
                pairs.append({"trial_id": trial_id, "baseline_ms": eager_ms, "candidate_ms": candidate_ms})
        except Exception as error:
            load_failure = {"type": type(error).__name__, "message": str(error),
                            "phase": phase, "trial_id": attempted_trial_id}
        summary = _gpu_summarize_paired_ratios(pairs, BOOTSTRAP_SAMPLES, SEED + token_count) if pairs else None
        if load_failure:
            load_decision = {"decision": "reject", "reason": "candidate_execution_failed"}
        elif len(pairs) != PAIRED_TRIALS:
            load_decision = {"decision": "tune", "reason": "incomplete_pairs"}
        else:
            load_decision = _gpu_decide_regression(summary, quality_passed, REGRESSION_THRESHOLD, MINIMUM_PAIRS)
        load_results.append({
            "token_count": token_count,
            "first_execution": first_execution,
            "quality": {"outputs_allclose": quality_passed},
            "summary": summary,
            "paired_runs": pairs,
            "planned_pairs": PAIRED_TRIALS,
            "completed_pairs": len(pairs),
            "failure": load_failure,
            "decision": load_decision,
        })

    result_dir.mkdir(parents=True, exist_ok=True)
    decisions = [item["decision"]["decision"] for item in load_results]
    if any(decision == "reject" for decision in decisions):
        overall_decision = {"decision": "reject", "reason": "one_or_more_loads_rejected"}
    elif any(decision == "tune" for decision in decisions):
        overall_decision = {"decision": "tune", "reason": "one_or_more_loads_inconclusive"}
    else:
        overall_decision = {"decision": "accept", "reason": "no_confirmed_regression_across_loads"}
    failures = [{"token_count": item["token_count"], **item["failure"]} for item in load_results if item["failure"]]
    record = {
        "schema_version": "performance-regression-mechanism/v4",
        "profile_id": ENVIRONMENT_PROFILE,
        "semantic_id": SEMANTIC_ID,
        "run_label": RUN_LABEL,
        "framework": {"python": sys.version.split()[0], "torch": torch.__version__, "cuda": torch.version.cuda},
        "hardware": {"device": torch.cuda.get_device_name(0), "total_memory_bytes": torch.cuda.get_device_properties(0).total_memory},
        "workload": {
            "id": WORKLOAD_ID,
            "version": WORKLOAD_VERSION,
            "token_counts": TOKEN_COUNTS,
            "hidden_size": HIDDEN_SIZE,
            "warmup_steps": WARMUP_STEPS,
            "paired_trials": PAIRED_TRIALS,
            "minimum_pairs": MINIMUM_PAIRS,
            "regression_threshold": REGRESSION_THRESHOLD,
            "bootstrap_samples": BOOTSTRAP_SAMPLES,
            "seed": SEED,
        },
        "metrics": {"load_matrix": load_results},
        "failure": failures or None,
        "evidence_level": "local_mechanism_benchmark",
        "decision": overall_decision,
    }
    save_json_record_once(result_path, record)
else:
    print("⏭️ RUN_MODE=dry_run;未执行 GPU 配对复测。")

5.3 读取结果 ​

按已有RUN_LABEL读取,从原始配对重算中位比例、自助区间与逐负载判断,再核对总决策。历史协议与当前协议的差异由读取器列出,原始记录保持不变。

python
from pathlib import Path
import json
from tools.project_runtime import resolve_project_root
from tools.performance_result_review import read_regression_result

result_path = resolve_project_root() / RESULT_RELATIVE_DIR / f"{WORKLOAD_ID}_{RUN_LABEL}.json"
if not result_path.is_file():
    if RUN_MODE == "real_gpu":
        raise FileNotFoundError(f"未找到结果:{result_path}")
    regression_record = {"status": "not_run", "result_path": str(result_path)}
    print(regression_record)
else:
    regression_record, result_review = read_regression_result(result_path, run_label=RUN_LABEL)
    print(json.dumps(result_review, ensure_ascii=False, indent=2))
    print(json.dumps({key: regression_record.get(key) for key in (
        "schema_version", "run_label", "framework", "hardware", "workload",
        "failure", "evidence_level", "decision",
    )}, ensure_ascii=False, indent=2))
    for load in regression_record.get("metrics", {}).get("load_matrix", []):
        print("Token 负载:", load["token_count"])
        print(json.dumps({key: load.get(key) for key in (
            "first_execution", "quality", "summary", "planned_pairs", "completed_pairs", "failure", "decision",
        )}, ensure_ascii=False, indent=2))
        print("轮次 | 基线 ms | 候选 ms | 候选/基线")
        for pair in load.get("paired_runs", []):
            print(pair["trial_id"], pair["baseline_ms"], pair["candidate_ms"],
                  pair["candidate_ms"] / pair["baseline_ms"])

5.4 解释配对结果 ​

当前登记为rtx5070ti_regression_v3,工作负载版本为v4;以下按决策记录中的1.05边界复算历史判断。历史文件未保存独立阈值配置;后续复测应在采样前保存阈值,方便核对其设定时点。RTX 5070 Ti 实测中,三档负载均通过输出一致性检查且没有执行失败。编译候选没有在任一负载下获得稳态加速;随着 Token 数增加,延迟差距由明确回归逐步缩小。

观察项读取字段当前结果解释与动作
正确性与执行failure、quality.outputs_allclose三档首次输出均 allclose=true;各 30 对,无失败当前差异可以进入性能判断,不由正确性失败解释
首次调用代价first_execution.eager_ms、compile_ms 与比例1 Token:7.27 / 404.44 ms;32 Token:0.46 / 29.00 ms;256 Token:1.11 / 245.67 ms候选首次调用包含编译准备且受形状与缓存影响,不混入稳态分布
稳态回归summary.median_ratio 与 90% Bootstrap 区间1 Token:1.089,区间 [1.082, 1.100];32 Token:1.055,[1.050, 1.063];256 Token:1.017,[1.014, 1.020]1 Token 确认超过 5% 回归;32 Token 跨越门槛;256 Token 未越过门槛但仍略慢
适用边界三档逐负载决策与总决策reject / tune / accept;总决策 rejectaccept表示区间上界未超过允许回归门槛;三档均未证明编译加速,256 Token的通过与加速是不同判断

相关阅读 ​

把配对方法用于训练或服务时,先核对各自的计时范围、输入和质量要求。

Released under the MIT License.