Skip to content

69. Prefix Caching Benchmark | 前缀缓存基准 ​

难度: Hard | 环境: CPU 主线,GPU 可选 | 标签: 推理优化、前缀缓存、基准对比 | 目标人群: 理解缓存复用、需要比较服务方案的读者

🚀 云端运行环境

本章节的实战代码可以点击以下链接在免费 GPU 算力平台上直接运行:

Open In ColabOpen In Studio (国内推荐:魔搭社区免费实例)

本节导读 ​

多个问题共用一段长系统提示,后来的请求可以复用此前保存的键值缓存,省去部分预填充。复用多少取决于词元开头是否一致、计算条件是否相同,以及旧缓存是否还在;保留缓存也会占用新请求需要的容量。

本项目固定请求内容与运行条件,比较缓存关闭和开启两条路径。先检查实际命中和输出,再比较首次返回时间、吞吐与容量记录,判断这组请求是否值得使用缓存。

关键词: 缓存身份、连续前缀、命中率、首词元延迟、输出等价性

运行环境 ​

  • 基础练习: 使用 BASE-CPU,验证缓存身份、前缀键和采用规则。
  • 真实服务实验: 使用 INF-VLLM-011-CU128,模型与请求集见 5.1。

前置阅读 ​

先区分可复用的逻辑前缀与实际占用的缓存块,再比较命中能否转成请求收益。

Step 1:从共享请求到缓存开关的配对比较 ​

键值缓存(K/V)保存模型处理历史词元时得到的键与值。前缀缓存复用请求开头已经计算出的这部分状态。模型权重、分词结果、聊天模板、适配器、位置和注意力配置、缓存精度与访问权限共同限定可共享的状态,称为缓存命名空间。请求开头词元相同且计算条件一致,才有复用机会。

先固定共享前缀请求集,把关闭缓存作为基线、开启缓存作为候选,再核对后端记录与输出。本页客户端以请求发出至首次非空内容返回的时间观察首响应,历史字段沿用首词元延迟(TTFT)的名称;一个返回内容块可能包含多个词元。跳过重复预填充可能缩短它,但排队、后缀计算和输出收尾仍需分别观察。

项目环节输入与操作需要保留的证据
定义比较条件固定模型版本、输入、生成规则与测量顺序同一请求和运行条件
识别可复用状态检查缓存身份和从开头连续命中的范围实际开关、命中与容量观测
真实执行依次运行缓存关闭与开启服务原始请求、日志、失败与指标
形成候选决策联合核对输出、延迟、吞吐与资源采用、补测或拒绝及下一动作

前缀缓存从身份检查到真实复用

Step 2:同一块内容还必须属于同一历史前缀 ​

完整块缓存按固定词元数分块,尾部残块继续计算。块键将当前位置和从开头至当前块的内容一起保存,使历史上下文参与匹配。例如每块两个词元,先处理 [1,2,3,4],再处理 [9,9,5,6],新请求 [1,2,5,6] 只能复用开头 [1,2];末块 [5,6] 的历史开头是 [9,9],新请求的历史开头却是 [1,2],因此这一块需要重新计算。

固定请求集、请求顺序和预热过程,才能解释重复机会。首次建立前缀与后续热复用分别记录;命中长度统计省去多少输入位置的计算,物理容量则统计仍保留多少缓存块。本页的最近最少使用(LRU)模型按最近访问顺序淘汰条目,真实系统还要保护活动引用与在途访问。

比较对象保持一致变化或观察
缓存关闭与开启模型、词元化输入、生成规则、并发和重复次数只改变前缀缓存开关
输入分布共享比例、请求顺序、冷启动与预热条件命中机会与首次建立成本
逻辑缓存模型同一批词元与命名空间连续完整前缀键、容量与 LRU 驱逐
真实服务记录后端版本与相同测量口径直接命中计数、容量、TTFT 与吞吐

Step 3:请求完成、输出等价和性能收益分别判断 ​

请求成功且返回非空输出后,继续核对同一请求的确定性输出,并比较同配置重复运行的结果。实际复用读取后端的命中计数及统计来源,时间收益则读取请求记录。请求级命中率按命中请求数计,词元复用比例按复用词元数计,两种分母不能混用。

候选只有在身份、请求、输出核对、稳定性与直接命中证据齐备后,才比较 TTFT 和输出吞吐;物理容量、驱逐与实际峰值要另看资源报告。CPU 规则检查容量字段是否齐全;评估容量是否足够,还要用目标并发和上下文测量实际峰值、驱逐与失败。

观察到的状态决策下一步
身份不一致、真实执行失败或请求不完整拒绝(reject)修复匹配或运行问题,保留失败记录
命中、指标、输出等价或稳定性证据缺少补测(tune)不交接采用候选,补采缺少的证据
请求完成,但输出等价性或重复稳定性未通过调整(tune)保留性能信号,定位差异后重新配对
命中成立,但延迟与吞吐改善不同时达标调整(tune)复核负载与缓存维护成本
输出核对、稳定性、命中和两个性能目标均通过局部采用(accept)仅将当前请求分布的候选送入部署复测

Step 4:实现缓存身份、配对证据与采用判断 ​

题目提供分块、完整前缀键、LRU 与结果统计骨架。你将完成命名空间匹配、证据收集和采用规则;输入布尔事实是对真实报告中请求完成、输出等价和重复稳定性核对的摘要,任务答案是否正确另用参考答案或任务评测规则判断。测试包括跨上下文同块、身份缺失、缺证和单项性能门槛未过。

任务与目的已知条件完成标准测试验证
TODO 1:核对缓存命名空间权重、分词器、模板、适配器、位置、精度、注意力和权限身份均有值且一致,开关形成关闭/开启对照字段变化、两边同时缺身份与空身份
TODO 2:建立配对证据基线、候选、命中与各项输出核对失败、缺失指标和质量事实分别保留缺少基线指标、缺命中、输出等价与稳定性未通过
TODO 3:形成采用状态命中率、TTFT、吞吐与证据质量事实通过且两项收益同时达标才采用精确阈值、单项不足与缺证
python
from typing import Dict, List
python
# 分块、驱逐和指标汇总构成机制底座;TODO 聚焦命名空间、后端证据和采用状态。
# CPU 结果验证证据关系,不替代真实缓存后端测量。
def complete_token_blocks(tokens: List[int], block_size: int) -> List[tuple[int, ...]]:
    """仅切出完整 token block;尾部不足一个 block 的 token 不参与缓存匹配。"""
    return [tuple(tokens[start:start + block_size]) for start in range(0, len(tokens) - block_size + 1, block_size)]


def _prefix_block_key(blocks, block_index):
    """键包含从位置零到当前块的完整历史,避免跨上下文误复用。"""
    return (block_index, tuple(tuple(block) for block in blocks[:block_index + 1]))


def count_continuous_prefix_hits(blocks, cache):
    """返回从 block 位置 0 开始连续命中的数量。

    blocks 是当前请求的完整 block 列表;cache 的 key 包含块位置及截至当前块的完整前缀。
    返回首个 miss 前的命中数,后续偶然命中不计入。
    """
    hit_blocks = 0
    for block_index, block in enumerate(blocks):
        if _prefix_block_key(blocks, block_index) not in cache:
            break
        hit_blocks += 1
    return hit_blocks


def write_blocks_with_lru(cache, blocks, clock, capacity_blocks):
    """写入当前请求的完整 block,并返回更新后的时钟与驱逐次数。

    已存在的 block 再次写入也要刷新访问时间;返回 (clock, eviction_count)。
    """
    eviction_count = 0
    for block_index, block in enumerate(blocks):
        clock += 1
        cache[_prefix_block_key(blocks, block_index)] = clock
        if len(cache) > capacity_blocks:
            oldest = min(cache, key=cache.get)
            del cache[oldest]
            eviction_count += 1
    return clock, eviction_count


def simulate_prefix_cache(token_sequences: List[List[int]], block_size: int = 4, capacity_blocks: int = 8) -> Dict[str, object]:
    """模拟完整 token block 的前缀复用和 LRU 驱逐;不测真实 backend 延迟。"""
    if block_size <= 0 or capacity_blocks <= 0:
        raise ValueError('block_size 和 capacity_blocks 必须为正数')
    if not isinstance(token_sequences, list) or not all(isinstance(tokens, list) for tokens in token_sequences):
        raise TypeError('token_sequences 必须是 list[list[int]]')
    cache, clock = {}, 0
    reused_tokens = total_prompt_tokens = hit_requests = eviction_count = 0
    for tokens in token_sequences:
        if not tokens:
            continue
        total_prompt_tokens += len(tokens)
        blocks = complete_token_blocks(tokens, block_size)
        hit_blocks = count_continuous_prefix_hits(blocks, cache)
        if hit_blocks:
            hit_requests += 1
        reused_tokens += hit_blocks * block_size
        clock, evicted = write_blocks_with_lru(cache, blocks, clock, capacity_blocks)
        eviction_count += evicted
    request_count = sum(bool(tokens) for tokens in token_sequences)
    return {'total_requests': request_count, 'total_prompt_tokens': total_prompt_tokens,
            'reused_tokens': reused_tokens, 'miss_tokens': total_prompt_tokens - reused_tokens,
            'hit_rate': hit_requests / request_count if request_count else 0.0,
            'token_reuse_rate': reused_tokens / total_prompt_tokens if total_prompt_tokens else 0.0,
            'prefill_work_reduction': reused_tokens / total_prompt_tokens if total_prompt_tokens else 0.0,
            'eviction_count': eviction_count, 'cache_size_blocks': len(cache)}


def summarize_prefix_cache(runs):
    """汇总同一 workload 的缓存运行记录;仅保留可比较的证据。"""
    if not runs:
        return {'run_count': 0, 'best_hit_rate_run': None, 'avg_hit_rate': 0.0, 'avg_ttft_ms': 0.0}
    required = {'name', 'workload_id', 'hit_rate', 'ttft_ms'}
    if any(not required.issubset(item) for item in runs):
        raise KeyError(f'每条 run 必须包含 {sorted(required)}')
    if len({item['workload_id'] for item in runs}) != 1:
        raise ValueError('缓存汇总必须使用同一 workload_id')
    best = max(runs, key=lambda item: item['hit_rate'])
    return {'run_count': len(runs), 'best_hit_rate_run': best['name'],
            'avg_hit_rate': sum(item['hit_rate'] for item in runs) / len(runs),
            'avg_ttft_ms': sum(item['ttft_ms'] for item in runs) / len(runs)}


def compare_prefix_cache_to_baseline(baseline, candidate):
    """计算同一 workload 下 candidate 相对 baseline 的复用、TTFT 与开销变化。"""
    required = {'workload_id', 'hit_rate', 'ttft_ms', 'overhead'}
    if not required.issubset(baseline) or not required.issubset(candidate):
        raise KeyError(f'比较记录必须包含 {sorted(required)}')
    if baseline['workload_id'] != candidate['workload_id']:
        raise ValueError('baseline 与 candidate 必须使用同一 workload_id')
    return {'hit_rate_gain': candidate['hit_rate'] - baseline['hit_rate'],
            'ttft_delta_ms': candidate['ttft_ms'] - baseline['ttft_ms'],
            'overhead_delta': candidate['overhead'] - baseline['overhead']}


def prefix_cache_rank_key(candidate):
    """返回一项证据合格缓存配置的升序排序 key。

    更大的 hit_rate_gain、更小的 ttft_delta_ms 和 overhead_delta 应排在前面;
    ttft_delta_ms 为负时表示 candidate 的 TTFT 更低。
    """
    summary = candidate['comparison']
    return (-summary['hit_rate_gain'], summary['ttft_delta_ms'], summary['overhead_delta'])


def select_prefix_cache_candidate(candidates):
    """排除无命中证据、失败或质量不合格候选后排序。"""
    eligible = [item for item in candidates if item['evidence_ok'] and item['quality_ok']]
    eligible.sort(key=prefix_cache_rank_key)
    return {'candidate_count': len(candidates), 'eligible_count': len(eligible),
            'selected': eligible[0] if eligible else None,
            'rejected_names': [item['name'] for item in candidates if item not in eligible]}


def validate_cache_namespace(baseline, candidate):
    """核对影响 K/V 计算与共享权限的身份,缺失身份不能视为一致。"""
    namespace_fields = ('model_revision', 'tokenizer_id', 'chat_template', 'adapter_id',
                        'position_key', 'cache_dtype', 'attention_key', 'tenant_key')
    mismatches = [field for field in namespace_fields
                  if baseline.get(field) in (None, '') or candidate.get(field) in (None, '')
                  or baseline.get(field) != candidate.get(field)]
    # TODO 1:命名空间有值且一致,策略开关形成关闭/开启对照。
    # ==========================================
    # TODO 1:核对缓存身份与开关对照。
    # ==========================================
    # namespace_ok = ???  # 全部身份有值且一致。
    # change_ok = ???  # 基线关闭且候选开启缓存。
    raise NotImplementedError("请完成 TODO 1:namespace_ok、change_ok")
    issues = list(mismatches) + ([] if change_ok else ['prefix_cache_enabled'])
    return {'contract_ok': namespace_ok and change_ok, 'issues': issues}


def build_prefix_cache_pair_evidence(baseline, candidate):
    """核对两组指标与输出事实;缺失指标保留为空而不是补成零。"""
    contract = validate_cache_namespace(baseline, candidate)
    failures = [name for name, record in (('baseline', baseline), ('candidate', candidate)) if record.get('failure')]
    required_metrics = ('hit_rate', 'ttft_ms', 'throughput_tps', 'output_complete',
                        'cache_capacity_blocks', 'output_equivalent', 'repeat_stable')
    missing = [f'{name}.{field}' for name, record in (('baseline', baseline), ('candidate', candidate))
               for field in required_metrics if record.get(field) is None]
    # ==========================================
    # TODO 2:区分硬失败与缺失证据。
    # ==========================================
    # hard_failures = ???  # 身份问题与运行失败的集合。
    # missing_evidence = ???  # 两组缺少的观测字段。
    raise NotImplementedError("请完成 TODO 2:hard_failures、missing_evidence")
    if candidate.get('backend_cache_hit_observed') is not True:
        missing_evidence.append('backend_cache_hit')
    def difference(field, reverse=False):
        """两侧指标齐备时计算候选改善量。"""
        left, right = baseline.get(field), candidate.get(field)
        return None if left is None or right is None else (left - right if reverse else right - left)
    return {'hard_failures': hard_failures, 'missing_evidence': missing_evidence,
            'hit_rate': candidate.get('hit_rate'),
            'ttft_gain_ms': difference('ttft_ms', reverse=True),
            'throughput_gain': difference('throughput_tps'),
            'output_ok': baseline.get('output_complete') is True and candidate.get('output_complete') is True,
            'output_equivalent': baseline.get('output_equivalent') is True and candidate.get('output_equivalent') is True,
            'repeat_stable': baseline.get('repeat_stable') is True and candidate.get('repeat_stable') is True,
            'cache_capacity_blocks': candidate.get('cache_capacity_blocks')}


def decide_prefix_cache_candidate(evidence, min_hit_rate, min_ttft_gain_ms, min_throughput_gain):
    """按请求、输出核对、命中和两项性能目标形成局部采用状态。"""
    if evidence['hard_failures']:
        return {'decision': 'reject', 'next_action': 'repair_namespace_or_backend'}
    if evidence['missing_evidence']:
        return {'decision': 'tune', 'next_action': 'collect_cache_execution_evidence'}
    if not evidence['output_ok']:
        return {'decision': 'reject', 'next_action': 'inspect_request_outputs'}
    if not evidence['output_equivalent'] or not evidence['repeat_stable']:
        return {'decision': 'tune', 'next_action': 'resolve_output_equivalence_and_stability'}
    # TODO 3:命中、TTFT 与吞吐三个门槛必须同时满足。
    # ==========================================
    # TODO 3:合并命中率与两项性能门槛。
    # ==========================================
    # reuse_ok = ???  # 三项下限均满足时为 True。
    raise NotImplementedError("请完成 TODO 3:reuse_ok")
    if reuse_ok:
        return {'decision': 'accept', 'next_action': 'promote_fixed_workload_result'}
    return {'decision': 'tune', 'next_action': 'inspect_workload_and_cache_cost'}

测试 ​

分别检查前缀上下文、身份与开关、真实命中证据,以及输出核对和联合性能门槛。

python
# 本测试区验证缓存机制、候选配置排序和同 workload 的报告骨架。
def _prefix_cache_sequences():
    """构造共享前缀、部分命中与完全不相关的三条请求。"""
    return [[1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 4, 5, 6, 9, 10], [20, 21, 22, 23, 24, 25, 26, 27]]

def _cache_report(name, hit_rate, ttft_ms, overhead, evidence=True, quality=True):
    """构造同一 workload 下可用于缓存配置排序的显式指标记录。"""
    return {'name': name, 'workload_id': 'shared-prefix-v1', 'hit_rate': hit_rate, 'ttft_ms': ttft_ms,
            'overhead': overhead, 'evidence_ok': evidence, 'quality_ok': quality}

def test_continuous_block_hits():
    """验证连续前缀命中不能跳过首个 miss。"""
    cache = {_prefix_block_key([(1, 2), (3, 4)], 0): 1,
             _prefix_block_key([(1, 2), (3, 4)], 1): 2}
    assert count_continuous_prefix_hits([(1, 2), (3, 4), (5, 6)], cache) == 2
    assert count_continuous_prefix_hits([(9, 9), (3, 4)], cache) == 0

def test_lru_write_and_eviction():
    """验证容量上限、重新访问刷新时间和尾块不命中。"""
    cache = {_prefix_block_key([(1, 2)], 0): 1}
    _, evicted = write_blocks_with_lru(cache, [(3, 4), (5, 6)], 1, 2)
    assert len(cache) == 2 and evicted == 1
    previous_clock = cache[_prefix_block_key([(3, 4)], 0)]
    refreshed_clock, no_extra_eviction = write_blocks_with_lru(cache, [(3, 4)], 3, 2)
    assert cache[_prefix_block_key([(3, 4)], 0)] == refreshed_clock > previous_clock and no_extra_eviction == 0
    simulated = simulate_prefix_cache(_prefix_cache_sequences(), block_size=2, capacity_blocks=4)
    assert simulated['reused_tokens'] == 6 and simulated['eviction_count'] > 0
    tail_case = simulate_prefix_cache([[1, 2, 3, 4, 5], [1, 2, 3, 4, 9]], block_size=4, capacity_blocks=8)
    assert tail_case['reused_tokens'] == 4
    for invalid in ({'block_size': 0}, {'capacity_blocks': 0}):
        try:
            simulate_prefix_cache([[1, 2]], **invalid)
        except ValueError:
            pass
        else:
            raise AssertionError('非法缓存参数应明确拒绝')

def test_candidate_ranking_and_contract():
    """验证证据和质量门槛先于缓存收益排序。"""
    baseline = _cache_report('off', 0., 130, 0.)
    small = _cache_report('small', .5, 100, .03)
    large = _cache_report('large', .7, 95, .08)
    candidates = [{'name': item['name'], 'evidence_ok': item['evidence_ok'], 'quality_ok': item['quality_ok'],
                   'comparison': compare_prefix_cache_to_baseline(baseline, item)} for item in [small, large]]
    assert select_prefix_cache_candidate(candidates)['selected']['name'] == 'large'
    blocked = {'name': 'blocked', 'evidence_ok': False, 'quality_ok': False,
               'comparison': {**candidates[1]['comparison'], 'hit_rate_gain': 99}}
    assert select_prefix_cache_candidate([blocked])['selected'] is None
    assert summarize_prefix_cache([baseline, small])['best_hit_rate_run'] == 'small'
    try:
        compare_prefix_cache_to_baseline(baseline, {**small, 'workload_id': 'other'})
    except ValueError:
        pass
    else:
        raise AssertionError('不同 workload 不应直接比较缓存收益')

def _cache_project_record(enabled, hit_rate, ttft_ms, throughput_tps, output_complete, cache_capacity_blocks):
    """构造命名空间一致、仅缓存开关不同的项目记录。"""
    return {'model_revision': 'commit', 'tokenizer_id': 'tokenizer', 'chat_template': 'template-v1',
            'adapter_id': 'none', 'position_key': 'rope-v1', 'prefix_cache_enabled': enabled,
            'backend_cache_hit_observed': enabled, 'hit_rate': hit_rate, 'ttft_ms': ttft_ms,
            'throughput_tps': throughput_tps, 'output_complete': output_complete,
            'cache_capacity_blocks': cache_capacity_blocks, 'failure': None,
            'cache_dtype': 'bfloat16', 'attention_key': 'causal', 'tenant_key': 'shared-scope',
            'output_equivalent': True, 'repeat_stable': True}


def test_cache_project_contract_and_evidence():
    """验证命名空间、直接命中、输出完整性和服务收益门禁。"""
    baseline = _cache_project_record(False, 0.0, 130.0, 80.0, True, 37000)
    candidate = _cache_project_record(True, 0.65, 95.0, 84.0, True, 37000)
    assert validate_cache_namespace(baseline, candidate)['contract_ok']
    assert not validate_cache_namespace(baseline, {**candidate, 'chat_template': 'other'})['contract_ok']
    evidence = build_prefix_cache_pair_evidence(baseline, candidate)
    assert evidence['hard_failures'] == [] and evidence['missing_evidence'] == []
    assert decide_prefix_cache_candidate(evidence, 0.5, 20.0, 2.0)['decision'] == 'accept'
    no_path = build_prefix_cache_pair_evidence(baseline, {**candidate, 'backend_cache_hit_observed': False})
    assert decide_prefix_cache_candidate(no_path, 0.5, 20.0, 2.0)['decision'] == 'tune'
    bad_output = build_prefix_cache_pair_evidence(baseline, {**candidate, 'output_complete': False})
    assert decide_prefix_cache_candidate(bad_output, 0.5, 20.0, 2.0)['decision'] == 'reject'


def test_prefix_keys_do_not_mix_histories():
    """同一位置的块内容相同,但历史不同仍不能复用。"""
    sequences = [[1, 2, 3, 4], [9, 9, 5, 6], [1, 2, 5, 6]]
    result = simulate_prefix_cache(sequences, block_size=2, capacity_blocks=16)
    assert result['reused_tokens'] == 2
    assert result['hit_rate'] == 1 / 3


def test_missing_identity_and_quality_cannot_pass():
    """都缺身份不是相同;质量事实不齐备时不能采用。"""
    baseline = _cache_project_record(False, 0.0, 130., 80., True, 37000)
    candidate = _cache_project_record(True, 0.65, 95., 84., True, 37000)
    for key in ('model_revision', 'cache_dtype', 'tenant_key', 'attention_key'):
        left, right = dict(baseline), dict(candidate)
        left.pop(key); right.pop(key)
        assert not validate_cache_namespace(left, right)['contract_ok']
    for key in ('output_equivalent', 'repeat_stable'):
        evidence = build_prefix_cache_pair_evidence(baseline, {**candidate, key: False})
        assert decide_prefix_cache_candidate(evidence, 0.5, 20., 2.)['decision'] == 'tune'
    evidence = build_prefix_cache_pair_evidence({**baseline, 'ttft_ms': None}, candidate)
    assert 'baseline.ttft_ms' in evidence['missing_evidence']
    assert decide_prefix_cache_candidate(evidence, 0.5, 20., 2.)['decision'] == 'tune'
    for update in ({'ttft_ms': 125.}, {'throughput_tps': 81.}):
        evidence = build_prefix_cache_pair_evidence(baseline, {**candidate, **update})
        assert decide_prefix_cache_candidate(evidence, 0.5, 20., 2.)['decision'] == 'tune'


def test_prefix_cache_benchmark_template():
    """运行本节全部 CPU 缓存机制测试。"""
    test_continuous_block_hits()
    test_lru_write_and_eviction()
    test_candidate_ranking_and_contract()
    test_cache_project_contract_and_evidence()
    test_prefix_keys_do_not_mix_histories()
    test_missing_identity_and_quality_cannot_pass()
    print('测试通过:缓存命名空间、后端证据与采用状态均通过。')

try:
    test_prefix_cache_benchmark_template()
except NotImplementedError:
    print('请先完成 TODO 部分的代码!')
    raise

🛑 STOP HERE

先完成题目并运行测试,再展开参考代码。











参考代码与解析 ​

代码 ​

python
# 分块、驱逐和指标汇总构成机制底座;TODO 聚焦命名空间、后端证据和采用状态。
# CPU 结果验证证据关系,不替代真实缓存后端测量。
def complete_token_blocks(tokens: List[int], block_size: int) -> List[tuple[int, ...]]:
    """仅切出完整 token block;尾部不足一个 block 的 token 不参与缓存匹配。"""
    return [tuple(tokens[start:start + block_size]) for start in range(0, len(tokens) - block_size + 1, block_size)]


def _prefix_block_key(blocks, block_index):
    """键包含从位置零到当前块的完整历史,避免跨上下文误复用。"""
    return (block_index, tuple(tuple(block) for block in blocks[:block_index + 1]))


def count_continuous_prefix_hits(blocks, cache):
    """返回从 block 位置 0 开始连续命中的数量。"""
    # 机制:位置和 token 内容共同构成 key,首个未命中后停止。
    hit_blocks = 0
    for block_index, block in enumerate(blocks):
        if _prefix_block_key(blocks, block_index) not in cache:
            break
        hit_blocks += 1
    return hit_blocks


def write_blocks_with_lru(cache, blocks, clock, capacity_blocks):
    """写入当前请求的完整 block,并返回更新后的时钟与驱逐次数。"""
    # 机制:访问刷新时钟,容量超限时驱逐最久未访问块。
    eviction_count = 0
    for block_index, block in enumerate(blocks):
        clock += 1
        cache[_prefix_block_key(blocks, block_index)] = clock
        while len(cache) > capacity_blocks:
            oldest = min(cache, key=cache.get)
            del cache[oldest]
            eviction_count += 1
    return clock, eviction_count


def simulate_prefix_cache(token_sequences: List[List[int]], block_size: int = 4, capacity_blocks: int = 8) -> Dict[str, object]:
    """模拟完整 token block 的前缀复用和 LRU 驱逐;不测真实 backend 延迟。"""
    if block_size <= 0 or capacity_blocks <= 0:
        raise ValueError('block_size 和 capacity_blocks 必须为正数')
    if not isinstance(token_sequences, list) or not all(isinstance(tokens, list) for tokens in token_sequences):
        raise TypeError('token_sequences 必须是 list[list[int]]')
    cache, clock = {}, 0
    reused_tokens = total_prompt_tokens = hit_requests = eviction_count = 0
    for tokens in token_sequences:
        if not tokens:
            continue
        total_prompt_tokens += len(tokens)
        blocks = complete_token_blocks(tokens, block_size)
        hit_blocks = count_continuous_prefix_hits(blocks, cache)
        if hit_blocks:
            hit_requests += 1
        reused_tokens += hit_blocks * block_size
        clock, evicted = write_blocks_with_lru(cache, blocks, clock, capacity_blocks)
        eviction_count += evicted
    request_count = sum(bool(tokens) for tokens in token_sequences)
    return {'total_requests': request_count, 'total_prompt_tokens': total_prompt_tokens,
            'reused_tokens': reused_tokens, 'miss_tokens': total_prompt_tokens - reused_tokens,
            'hit_rate': hit_requests / request_count if request_count else 0.0,
            'token_reuse_rate': reused_tokens / total_prompt_tokens if total_prompt_tokens else 0.0,
            'prefill_work_reduction': reused_tokens / total_prompt_tokens if total_prompt_tokens else 0.0,
            'eviction_count': eviction_count, 'cache_size_blocks': len(cache)}


def summarize_prefix_cache(runs):
    """汇总同一 workload 的缓存运行记录;仅保留可比较的证据。"""
    if not runs:
        return {'run_count': 0, 'best_hit_rate_run': None, 'avg_hit_rate': 0.0, 'avg_ttft_ms': 0.0}
    required = {'name', 'workload_id', 'hit_rate', 'ttft_ms'}
    if any(not required.issubset(item) for item in runs):
        raise KeyError(f'每条 run 必须包含 {sorted(required)}')
    if len({item['workload_id'] for item in runs}) != 1:
        raise ValueError('缓存汇总必须使用同一 workload_id')
    best = max(runs, key=lambda item: item['hit_rate'])
    return {'run_count': len(runs), 'best_hit_rate_run': best['name'],
            'avg_hit_rate': sum(item['hit_rate'] for item in runs) / len(runs),
            'avg_ttft_ms': sum(item['ttft_ms'] for item in runs) / len(runs)}


def compare_prefix_cache_to_baseline(baseline, candidate):
    """计算同一 workload 下 candidate 相对 baseline 的复用、TTFT 与开销变化。"""
    required = {'workload_id', 'hit_rate', 'ttft_ms', 'overhead'}
    if not required.issubset(baseline) or not required.issubset(candidate):
        raise KeyError(f'比较记录必须包含 {sorted(required)}')
    if baseline['workload_id'] != candidate['workload_id']:
        raise ValueError('baseline 与 candidate 必须使用同一 workload_id')
    return {'hit_rate_gain': candidate['hit_rate'] - baseline['hit_rate'],
            'ttft_delta_ms': candidate['ttft_ms'] - baseline['ttft_ms'],
            'overhead_delta': candidate['overhead'] - baseline['overhead']}


def prefix_cache_rank_key(candidate):
    """返回一项证据合格缓存配置的升序排序 key。"""
    summary = candidate['comparison']
    # 排序依据:更多复用、更低 TTFT 和更低维护开销优先。
    return (-summary['hit_rate_gain'], summary['ttft_delta_ms'], summary['overhead_delta'])


def select_prefix_cache_candidate(candidates):
    """排除无命中证据、失败或质量不合格候选后排序。"""
    eligible = [item for item in candidates if item['evidence_ok'] and item['quality_ok']]
    eligible.sort(key=prefix_cache_rank_key)
    return {'candidate_count': len(candidates), 'eligible_count': len(eligible),
            'selected': eligible[0] if eligible else None,
            'rejected_names': [item['name'] for item in candidates if item not in eligible]}


def validate_cache_namespace(baseline, candidate):
    """核对影响 K/V 计算与共享权限的身份,缺失身份不能视为一致。"""
    namespace_fields = ('model_revision', 'tokenizer_id', 'chat_template', 'adapter_id',
                        'position_key', 'cache_dtype', 'attention_key', 'tenant_key')
    mismatches = [field for field in namespace_fields
                  if baseline.get(field) in (None, '') or candidate.get(field) in (None, '')
                  or baseline.get(field) != candidate.get(field)]
    # TODO 1:命名空间有值且一致,策略开关形成关闭/开启对照。
    namespace_ok = not mismatches
    change_ok = baseline.get('prefix_cache_enabled') is False and candidate.get('prefix_cache_enabled') is True
    issues = list(mismatches) + ([] if change_ok else ['prefix_cache_enabled'])
    return {'contract_ok': namespace_ok and change_ok, 'issues': issues}


def build_prefix_cache_pair_evidence(baseline, candidate):
    """核对两组指标与输出事实;缺失指标保留为空而不是补成零。"""
    contract = validate_cache_namespace(baseline, candidate)
    failures = [name for name, record in (('baseline', baseline), ('candidate', candidate)) if record.get('failure')]
    required_metrics = ('hit_rate', 'ttft_ms', 'throughput_tps', 'output_complete',
                        'cache_capacity_blocks', 'output_equivalent', 'repeat_stable')
    missing = [f'{name}.{field}' for name, record in (('baseline', baseline), ('candidate', candidate))
               for field in required_metrics if record.get(field) is None]
    hard_failures = list(contract['issues']) + failures
    missing_evidence = list(missing)
    if candidate.get('backend_cache_hit_observed') is not True:
        missing_evidence.append('backend_cache_hit')
    def difference(field, reverse=False):
        """两侧指标齐备时计算候选改善量。"""
        left, right = baseline.get(field), candidate.get(field)
        return None if left is None or right is None else (left - right if reverse else right - left)
    return {'hard_failures': hard_failures, 'missing_evidence': missing_evidence,
            'hit_rate': candidate.get('hit_rate'),
            'ttft_gain_ms': difference('ttft_ms', reverse=True),
            'throughput_gain': difference('throughput_tps'),
            'output_ok': baseline.get('output_complete') is True and candidate.get('output_complete') is True,
            'output_equivalent': baseline.get('output_equivalent') is True and candidate.get('output_equivalent') is True,
            'repeat_stable': baseline.get('repeat_stable') is True and candidate.get('repeat_stable') is True,
            'cache_capacity_blocks': candidate.get('cache_capacity_blocks')}


def decide_prefix_cache_candidate(evidence, min_hit_rate, min_ttft_gain_ms, min_throughput_gain):
    """按请求、输出核对、命中和两项性能目标形成局部采用状态。"""
    if evidence['hard_failures']:
        return {'decision': 'reject', 'next_action': 'repair_namespace_or_backend'}
    if evidence['missing_evidence']:
        return {'decision': 'tune', 'next_action': 'collect_cache_execution_evidence'}
    if not evidence['output_ok']:
        return {'decision': 'reject', 'next_action': 'inspect_request_outputs'}
    if not evidence['output_equivalent'] or not evidence['repeat_stable']:
        return {'decision': 'tune', 'next_action': 'resolve_output_equivalence_and_stability'}
    # TODO 3:命中、TTFT 与吞吐三个门槛必须同时满足。
    reuse_ok = (evidence['hit_rate'] >= min_hit_rate
                and evidence['ttft_gain_ms'] >= min_ttft_gain_ms
                and evidence['throughput_gain'] >= min_throughput_gain)
    if reuse_ok:
        return {'decision': 'accept', 'next_action': 'promote_fixed_workload_result'}
    return {'decision': 'tune', 'next_action': 'inspect_workload_and_cache_cost'}

解析 ​

TODO 1:限定缓存身份 ​

  • 同一文本在不同权重、位置或适配器下可能对应不同 K/V;身份缺失也不能解释为相同。关闭/开启开关是本次唯一策略变化。
  • 测试逐个改变身份字段,再同时删除两边字段,区分“都未知”与“已确认一致”。

TODO 2:保留失败与缺证的区别 ​

  • 命名空间错误和运行失败阻止比较;命中、基线指标和输出核对缺失时,应保留未取得的证据,而非用零值补齐。
  • 测试移除显式指标与命中记录,并分别设置输出等价、重复稳定性未通过,确认这些事实进入决策。

TODO 3:同时满足正确复用与性能条件 ​

  • 请求完成不等于输出等价;输出核对与稳定性均通过后,命中率、TTFT 改善和吞吐改善三项条件采用合取判断。
  • 测试让一项性能改善、另一项未达标,核对联合门槛。历史 v4 的输出等价性和重复稳定性均未通过,登记为 tune。

Step 5(可选):真实服务实验——前缀缓存关闭与开启对照 ​

使用同一共享前缀请求分别运行缓存关闭与开启两条路径;每次运行保存分组 JSON,再从 manifest 读取证据并形成项目决策。

CPU 已验证真实环境补充本页输出
完整前缀身份、命中与驱逐统计、采用规则缓存关闭/开启的真实命中、请求输出、延迟、吞吐和资源记录原始 JSON、配对清单与当前负载下的候选决策

5.1 环境与固定工作负载 ​

使用 INF-VLLM-011-CU128;固定模型与请求配置按单卡实验登记,实际容量与执行能力由当前预检和真实运行确认。

本次实验回答一个具体问题:在同一组共享前缀请求上,开启 prefix cache 是否带来可观测收益。先记录实验契约,再让 G0 与 G1 只改变 cache policy;模型、backend、请求分布、生成长度、并发和测量流程保持一致。

实验要素本轮契约G0 / G1 如何保持一致证据输出
实验对象G0 cache off;G1 prefix cache使用同一模型 revision、backend 和服务版本两组可比较结果
固定 workloadshared-prefix 请求文件、生成长度、batch / concurrency使用同一请求顺序和 token 化结果统一 workload 记录
测量流程warmup、重复次数、超时和输出长度两组使用相同流程可复查的 JSON 结果
唯一变量cache policyG0 关闭,G1 只开启 prefix cache命中、延迟、吞吐和资源对照

前缀缓存 GPU/backend 对照实验契约

python
# 5.1:唯一可编辑配置。先改这一处,再运行 5.2–5.6。
RUN_REAL_BACKEND = False  # 默认值 False:改为 True 才启动真实 G0/G1 服务对照。
RUN_LABEL = 'replace_with_unique_label'  # 正式复测改为唯一标签;旧标签的 JSON 保留为历史证据。
SEMANTIC_ID = 'INF-PREFIX-CACHE'
SCHEMA_VERSION = 'inference-benchmark/v1'
MODEL_ID = 'Qwen/Qwen2.5-0.5B-Instruct'  # G0/G1 使用同一模型与 revision。
WORKLOAD = 'benchmarks/workloads/prefix_reuse_long.jsonl'  # 长共享前缀;两组必须使用同一冻结请求集。
EXECUTION_ORDER = 'baseline_then_candidate'  # 可选 baseline_then_candidate / candidate_then_baseline;反向复测需新标签。
NUM_PROMPTS = 30  # 按固定顺序循环 workload,不改变唯一请求集合。
CONCURRENCY = 1  # 先隔离 prefix cache;并发扫描属于 70,不在本页混入。
REPEATS = 3  # 避免把单次服务波动当作缓存结论。
WARMUP = 5  # 预热不计入正式统计;G1 的预热也会建立可复用条目。
MAX_TOKENS = 64  # 两组保持相同的生成长度。
BACKEND = 'vllm'  # 改变 backend 应形成新的专项对照。
DTYPE = 'bfloat16'  # 记录 backend 实际选择的 dtype;两组必须一致。

MAX_MODEL_LEN = 2048  # G0/G1 固定相同 KV 容量上限。
GPU_MEMORY_UTILIZATION = 0.80
ENFORCE_EAGER = True
RUN_ID = RUN_LABEL  # 派生内部标识;不单独修改。
MANIFEST_PATH = f'benchmarks/results/69_manifest_{RUN_ID}.json'  # 对照清单。
CANONICAL_GROUPS = (
    {'name': 'g0_cache_off', 'role': 'baseline', 'cache_policy': 'off', 'enabled': False},
    {'name': 'g1_prefix_cache', 'role': 'candidate', 'cache_policy': 'prefix_cache', 'enabled': True},
)
GROUPS = CANONICAL_GROUPS if EXECUTION_ORDER == 'baseline_then_candidate' else tuple(reversed(CANONICAL_GROUPS))

5.2 环境启动检查 ​

确认共享前缀 workload 可被读取,并记录 backend、GPU 与缓存开关。预检结果用于解释后续启动失败或缺少命中指标的原因。

检查项需要确认的内容记录用途
模型与 backendG0/G1 使用同一模型 revision 与服务版本保持对照条件一致
GPU 与运行时GPU、驱动、CUDA、PyTorch、dtype环境变更后重新采集
workload 与开关请求文件、端口、cache policy确认共享前缀请求和策略可加载
命中证据backend metrics、日志或专用接口不能由 TTFT 或吞吐反推命中率
python
# 5.2:只读取 5.1 配置并检查运行条件,不启动服务。
from pathlib import Path
from tools.inference_project_runtime import locate_repo_root
import importlib.util
import torch

REPO_ROOT = locate_repo_root()  # 运行基础设施:将 5.1 的仓库相对路径锚定到项目根目录。
WORKLOAD_FILE = REPO_ROOT / WORKLOAD
MANIFEST_FILE = REPO_ROOT / MANIFEST_PATH
preflight = {
    'workload_exists': WORKLOAD_FILE.is_file(), 'backend': BACKEND,
    'cuda_available': torch.cuda.is_available(),
    'vllm_on_current_kernel': importlib.util.find_spec('vllm') is not None,
}
print('prefix cache preflight:', preflight)

5.3 候选与比较契约检查 ​

核对 5.1 中的请求集、两组缓存策略和结果路径,确认它们组成同负载的关闭/开启对照。

python
from pathlib import Path
from tools.backend_runtime import start_vllm as loaded_start_vllm

if '--no-enable-prefix-caching' not in loaded_start_vllm.__code__.co_consts:
    raise RuntimeError(
        '当前 Kernel 仍加载旧版 backend_runtime,无法构造 cache-off 基线;'
        '请重启 Kernel 后从 5.1 重新运行。'
    )
if not WORKLOAD_FILE.is_file():
    raise FileNotFoundError(f'找不到共享前缀 workload:{WORKLOAD_FILE}')
if EXECUTION_ORDER not in {'baseline_then_candidate', 'candidate_then_baseline'}:
    raise ValueError('EXECUTION_ORDER 只能是 baseline_then_candidate 或 candidate_then_baseline')
if {group['name'] for group in GROUPS} != {'g0_cache_off', 'g1_prefix_cache'}:
    raise ValueError('本页必须保留 cache off 与 prefix cache 两个受控组')
if NUM_PROMPTS <= 0 or WARMUP < 0 or REPEATS <= 0:
    raise ValueError('num_prompts、repeats 必须为正数,warmup 不能为负数')
project_config = {
    'model': MODEL_ID, 'backend': BACKEND, 'dtype': DTYPE, 'workload': WORKLOAD,
    'generated_tokens': MAX_TOKENS, 'num_prompts': NUM_PROMPTS,
    'concurrency': CONCURRENCY, 'warmup': WARMUP, 'repeats': REPEATS,
    'execution_order': EXECUTION_ORDER,
    'groups': [item['name'] for item in GROUPS],
}
print(project_config)

5.4 执行实验并保存原始 JSON ​

按 5.1 中声明的顺序运行 G0 与 G1。每一组完成 warmup 后再测量;单组报告和本次 manifest 保存原始配置、请求结果、缓存观测与失败信息。5.6 读取这些记录并单独生成决策,原始文件保留运行时内容。

分支执行内容原始产物
G0cache off,固定共享前缀 workload69_g0_cache_off_...json
G1prefix cache,其他条件不变69_g1_prefix_cache_...json
manifest汇总本次两组、结果路径与运行状态69_manifest_...json
python
# 5.4:执行 G0/G1;在服务仍运行时登记 prefix-cache 观测,再保存分组 JSON。
from pathlib import Path
import json
import re
import urllib.request
from tools.inference_project_runtime import runtime_snapshot, start_optional_vllm, stop_optional_vllm, run_backend_benchmark
from tools.project_runtime import save_json_record_once


def observe_prefix_cache(base_url, log_handle):
    """从当前 vLLM 的 /metrics 与服务日志记录缓存命中证据。"""
    observation = {
        'hit_rate': None,
        'cache_queries': None,
        'cache_hits': None,
        'cache_capacity_blocks': None,
        'hit_rate_evidence': None,
        'metrics_source': f'{base_url}/metrics',
        'prefix_cache_enabled': None,
        'prefix_cache_metric_series': [],
        'log_path': getattr(log_handle, 'name', None),
    }
    try:
        with urllib.request.urlopen(f'{base_url}/metrics', timeout=10) as response:
            metric_text = response.read().decode('utf-8', errors='replace')
        series = []
        for line in metric_text.splitlines():
            normalized = line.lower()
            if 'prefix' in normalized and 'cache' in normalized and not line.startswith('#'):
                series.append(line)
        observation['prefix_cache_metric_series'] = series
        config_line = next((line for line in series if 'cache_config_info' in line), '')
        enabled = re.search(r'enable_prefix_caching=\"(True|False)\"', config_line)
        if enabled:
            observation['prefix_cache_enabled'] = enabled.group(1) == 'True'
        queries = re.search(r'prefix_cache_queries_total[^}]*}\s+([0-9.]+)', metric_text)
        hits = re.search(r'prefix_cache_hits_total[^}]*}\s+([0-9.]+)', metric_text)
        blocks = re.search(r'num_gpu_blocks=\"([0-9]+)\"', config_line)
        if queries and hits:
            observation['cache_queries'] = float(queries.group(1))
            observation['cache_hits'] = float(hits.group(1))
            if observation['cache_queries'] > 0:
                observation['hit_rate'] = round(observation['cache_hits'] / observation['cache_queries'], 6)
        if blocks:
            observation['cache_capacity_blocks'] = int(blocks.group(1))
        if series:
            observation['hit_rate_evidence'] = 'vllm_metrics_prefix_cache_series'
    except Exception as exc:
        observation['metrics_error'] = str(exc)

    try:
        log_handle.flush()
        log_text = Path(log_handle.name).read_text(encoding='utf-8', errors='replace')
        matches = re.findall(r'Prefix cache hit rate:\s*([0-9.]+)%', log_text)
        if matches:
            observation['hit_rate'] = round(float(matches[-1]) / 100.0, 6)
            observation['hit_rate_evidence'] = 'vllm_server_log_prefix_cache_hit_rate'
    except Exception as exc:
        observation['log_parse_error'] = str(exc)
    return observation


if RUN_REAL_BACKEND:
    final_paths = {group['name']: REPO_ROOT / f"benchmarks/results/69_{group['name']}_{BACKEND}_{RUN_ID}.json" for group in GROUPS}
    raw_paths = {group['name']: REPO_ROOT / f"benchmarks/results/69_{group['name']}_{BACKEND}_raw_{RUN_ID}.json" for group in GROUPS}
    collisions = [path for path in (*final_paths.values(), *raw_paths.values(), MANIFEST_FILE) if path.exists()]
    if collisions:
        raise FileExistsError(f'当前 RUN_LABEL 已有结果,请更换标签:{collisions}')
    reports = []
    for group in GROUPS:
        result_path = final_paths[group['name']]
        raw_result_path = raw_paths[group['name']]
        server = log_path = None
        try:
            server, log_path, port, selected_dtype, _ = start_optional_vllm(
                model_id=MODEL_ID, model_source='auto', dtype=DTYPE,
                served_model_name=MODEL_ID, enable_prefix_caching=group['enabled'],
                max_model_len=MAX_MODEL_LEN, gpu_memory_utilization=GPU_MEMORY_UTILIZATION,
                enforce_eager=ENFORCE_EAGER,
                log_path=REPO_ROOT / f"benchmarks/results/69_{group['name']}_{BACKEND}_{RUN_ID}.log",
            )
            base_url = f'http://127.0.0.1:{port}'
            report = run_backend_benchmark(
                project='69', base_url=base_url, model=MODEL_ID,
                label=f"{BACKEND}-{group['name']}", output=str(raw_result_path), workload=str(WORKLOAD_FILE),
                backend=BACKEND, batch=1, max_tokens=MAX_TOKENS, num_prompts=NUM_PROMPTS,
                concurrency=CONCURRENCY, warmup=WARMUP, repeats=REPEATS, dtype=selected_dtype,
                cache_policy=group['cache_policy'], include_requests=True,
            )
            report.update({'schema_version': SCHEMA_VERSION, 'semantic_id': SEMANTIC_ID, 'run_label': RUN_LABEL, 'run_mode': 'real_backend', 'raw_result_path': str(raw_result_path)})
            report['strategy_metrics'] = observe_prefix_cache(base_url, log_path)
            save_json_record_once(result_path, report)
            reports.append({'group': group, 'role': group['role'], 'report_path': str(result_path),
                            'status': report.get('status', 'ok'), 'quality': report.get('quality', {'status': 'not_evaluated'}),
                            'failure': report.get('failure'), 'evidence_level': 'real_backend_paired'})
        except Exception as error:
            failure_record = {'schema_version': SCHEMA_VERSION, 'semantic_id': SEMANTIC_ID, 'project': '69', 'run_label': RUN_LABEL, 'run_mode': 'real_backend', 'status': 'failed', 'group': group, 'failure': {'type': type(error).__name__, 'message': str(error)}}
            if not result_path.exists():
                save_json_record_once(result_path, failure_record)
            reports.append({'group': group, 'role': group['role'], 'report_path': str(result_path),
                            'status': 'failed', 'quality': {'status': 'failed'},
                            'failure': {'failure_reason': str(error), 'retest_path': 'recheck_backend_and_cache_policy'},
                            'evidence_level': 'backend_execution_failure'})
        finally:
            if server is not None:
                stop_optional_vllm(server, log_path)
    manifest = {'schema_version': 'inference-project-manifest/v1', 'semantic_id': SEMANTIC_ID, 'project': '69', 'run_label': RUN_LABEL, 'run_mode': 'real_backend',
        'experiment': 'prefix_cache_reuse', 'config': project_config, 'environment': runtime_snapshot(), 'groups': reports,
        'artifact': {'manifest_path': MANIFEST_PATH, 'group_result_paths': [item['report_path'] for item in reports]},
        'failure': [item for item in reports if item.get('status') != 'ok'] or None,
        'quality': {'status': 'not_evaluated'}, 'evidence_level': 'real_backend_paired',
        'decision': {'decision': 'tune', 'reason': '运行 5.5、5.6 汇总命中、质量和性能证据后再判断'},}
    output = MANIFEST_FILE
    save_json_record_once(output, manifest)
    print(f'对照清单已保存: {output}')
else:
    print('实验未启动:确认 5.1 条件后,将 RUN_REAL_BACKEND 改为 True。')

5.5 读取原始结果并规整证据 ​

逐条读取 manifest 引用的 G0/G1 JSON,确认两组共享模型、backend、dtype、workload、生成长度和并发。先检查请求完整返回与非空输出,再读取独立证据记录中的输出等价和重复一致性。历史 TTFT/TPOT 沿用原采集器口径,TPOT 包含结束尾部;复测时按 66 保存独立时间戳,以区分首内容、内容跨度和流结束。

组别唯一变化缓存机制证据请求完成与性能证据用途
G0cache off实际开关为关闭TTFT / TPOT / 吞吐 / 显存;成功与失败请求可比较的基线
G1prefix cache实际开关、hit rate、证据来源与 G0 同口径;请求完成状态判断复用是否带来净收益
失败记录unsupported / OOM / failed已采集字段failure reason、retest path不与成功运行混合平均
python
# 5.5 读取分组 JSON,核对共同配置并规整命中、质量与性能证据。
import json
from pathlib import Path
from tools.inference_project_runtime import locate_repo_root

REPO_ROOT = locate_repo_root()
RESULT_CONFIG_FIELDS = ('model', 'backend', 'dtype', 'workload', 'generated_tokens', 'concurrency')


def scalar_metric(metrics, name, statistic='p50'):
    """读取标量指标或嵌套分位数。"""
    value = metrics.get(name)
    if isinstance(value, dict):
        return value.get(statistic)
    return value


def read_prefix_result(result_path):
    """读取单组报告,展平延迟分位数并保留可追溯的缓存观测来源。"""
    result_path = Path(result_path)
    if not result_path.is_absolute():
        result_path = REPO_ROOT / result_path
    if not result_path.exists():
        return {'path': str(result_path), 'status': 'missing'}
    raw = json.loads(result_path.read_text(encoding='utf-8'))
    normalized = raw.get('normalized_result') or raw
    config = raw.get('config') or normalized.get('config', {})
    raw_metrics = raw.get('metrics') or normalized.get('metrics', {})
    results = raw.get('results') or []
    expected = raw.get('workload', {}).get('requests')
    successful = raw_metrics.get('successful_requests')
    failed = raw_metrics.get('failed_requests')
    nonempty_outputs = all(bool(item.get('output_text', '')) for item in results if item.get('ok'))
    quality_status = 'passed' if expected == successful and failed == 0 and nonempty_outputs else 'not_evaluated'
    strategy = raw.get('strategy_metrics') or normalized.get('strategy_metrics', {})
    status = raw.get('status') or normalized.get('status')
    if status in {None, 'not_recorded'}:
        status = 'failed' if raw.get('failure') else 'ok'
    return {
        'path': str(result_path),
        'status': status,
        'config': {field: config.get(field) for field in RESULT_CONFIG_FIELDS},
        'metrics': {
            'ttft_p50_ms': scalar_metric(raw_metrics, 'ttft_ms', 'p50'),
            'ttft_p95_ms': scalar_metric(raw_metrics, 'ttft_ms', 'p95'),
            'tpot_p50_ms': scalar_metric(raw_metrics, 'tpot_ms', 'p50'),
            'tpot_p95_ms': scalar_metric(raw_metrics, 'tpot_ms', 'p95'),
            'throughput_tokens_per_s': raw_metrics.get('output_token_throughput_per_s'),
            'e2e_p95_ms': scalar_metric(raw_metrics, 'e2e_ms', 'p95'),
            'successful_requests': successful,
            'failed_requests': failed,
        },
        'cache_metrics': {field: strategy.get(field) for field in (
            'hit_rate', 'cache_queries', 'cache_hits', 'cache_capacity_blocks', 'hit_rate_evidence',
            'metrics_source', 'prefix_cache_enabled', 'prefix_cache_metric_series', 'log_path',
        )},
        'output_completeness': {'status': quality_status, 'expected_requests': expected,
                                'successful_requests': successful, 'nonempty_outputs': nonempty_outputs},
        'failure': raw.get('failure') or normalized.get('failure', {}),
    }


def summarize_prefix_pair(records):
    """汇总缓存候选相对基线的配对差值。"""
    indexed = {item.get('group'): item for item in records}
    g0, g1 = indexed.get('g0_cache_off'), indexed.get('g1_prefix_cache')
    if not g0 or not g1:
        return {}
    base, candidate = g0.get('metrics', {}), g1.get('metrics', {})
    def delta(name):
        left, right = base.get(name), candidate.get(name)
        return None if left is None or right is None else round(right - left, 4)
    return {
        'ttft_p50_delta_ms': delta('ttft_p50_ms'),
        'ttft_p95_delta_ms': delta('ttft_p95_ms'),
        'tpot_p50_delta_ms': delta('tpot_p50_ms'),
        'throughput_delta_tokens_per_s': delta('throughput_tokens_per_s'),
        'e2e_p95_delta_ms': delta('e2e_p95_ms'),
    }


def read_prefix_manifest(manifest_path):
    """读取 manifest 及分组结果,并检查 G0/G1 是否具有可比性。"""
    manifest_path = Path(manifest_path)
    if not manifest_path.is_absolute():
        manifest_path = REPO_ROOT / manifest_path
    if not manifest_path.exists():
        return {'status': 'not_run', 'manifest': str(manifest_path), 'results': []}
    manifest = json.loads(manifest_path.read_text(encoding='utf-8'))
    groups = manifest.get('groups', [])
    results = []
    for group in groups:
        item = read_prefix_result(group.get('report_path', ''))
        item['group'] = group.get('group', {}).get('name')
        item['role'] = group.get('role')
        if item['status'] == 'missing' and group.get('status') == 'failed':
            item.update({'status': 'failed', 'failure': group.get('failure', {})})
        results.append(item)
    readable = [item for item in results if item.get('status') not in {'missing', 'failed'}]
    signatures = {tuple(item['config'].get(field) for field in RESULT_CONFIG_FIELDS) for item in readable}
    return {
        'status': 'ok', 'manifest': str(manifest_path), 'results': results,
        'shared_config_ok': len(readable) == 2 and len(signatures) == 1,
        'paired_summary': summarize_prefix_pair(results),
        'output_completeness': {item.get('group'): item.get('output_completeness') for item in results},
        'failure': manifest.get('failure'),
    }

prefix_evidence = read_prefix_manifest(globals().get('MANIFEST_PATH', 'benchmarks/results/69_manifest_smoke.json'))
print(json.dumps(prefix_evidence, ensure_ascii=False, indent=2))

5.6 形成派生决策 ​

先确认比较口径和直接命中证据,再查看请求完成状态、输出等价性、重复稳定性、TTFT、吞吐与容量观察。5.6 只读取已有证据并生成派生决策;不修改 G0/G1 原始 JSON 或 manifest。

观察到的结果决策下一步
命中、请求、输出等价与重复一致性通过,且TTFT与吞吐均改善accept补容量压力验证,再扩展请求分布做回归测试
任一质量证据缺失或未通过、命中证据缺失,或性能信号不一致tune补充正确性证据或调整 block、容量、驱逐策略
unsupported、OOM、请求失败或端到端性能同时退化reject保留失败记录,回退 cache off 或更换候选

当前登记结果

字段登记内容
项目与标签69 节前缀缓存;rtx5070ti_prefix_cache_v4
运行条件RTX 5070 Ti Laptop GPU;cache-off 与 prefix-cache 配对请求
原始证据cache-off 日志、prefix-cache 日志
正式证据证据记录
清单与交接结果清单 是原始运行清单,其决策字段是运行前占位;最终结论以 证据记录 为准,未生成 72 节采用交接
结论tune:直接命中与性能改善已记录;30组输出仅12组相同,缓存开启后5个重复提示组中仅1组输出保持一致,需先定位差异
容量与统计范围已记录缓存块容量,独立显存峰值及驱逐压力待测;后端命中比例按原生指标单位与采集窗口解释,CPU示例另统计请求命中率
python
# 5.6 只根据 5.5 的证据给出派生决策;不修改 5.4 写出的原始 JSON 或 manifest。
def decide_prefix_cache_evidence(evidence, registered_quality=None):
    """检查配对、质量与缓存命中证据,再判断当前 workload 的性能取舍。"""
    if evidence.get('status') != 'ok':
        return {'decision': 'tune', 'reason': '尚未获得可读取的分组结果'}
    if evidence.get('failure'):
        return {'decision': 'reject', 'reason': '存在 backend 失败记录'}
    records = {item.get('group'): item for item in evidence.get('results', [])}
    g0, g1 = records.get('g0_cache_off'), records.get('g1_prefix_cache')
    if not g0 or not g1 or not evidence.get('shared_config_ok'):
        return {'decision': 'tune', 'reason': 'G0/G1 结果或共同配置尚未完整'}
    if any(item.get('output_completeness', {}).get('status') != 'passed' for item in (g0, g1)):
        return {'decision': 'reject', 'reason': 'G0 或 G1 请求未完整返回,不能解释缓存性能'}
    quality = registered_quality or {}
    exact = quality.get('exact_output_equivalence') or {}
    stability = quality.get('repeat_stability') or {}
    if exact.get('status') != 'passed' or exact.get('matched') != exact.get('total'):
        return {'decision': 'tune', 'reason': '输出等价性未通过;性能收益不能替代缓存正确性核对'}
    g0_stability = stability.get('g0_cache_off') or {}
    g1_stability = stability.get('g1_prefix_cache') or {}
    if (not g0_stability or not g1_stability
            or g0_stability.get('stable_prompt_groups') != g0_stability.get('total_prompt_groups')
            or g1_stability.get('stable_prompt_groups') != g1_stability.get('total_prompt_groups')):
        return {'decision': 'tune', 'reason': '重复运行稳定性证据缺失或未通过'}
    g0_cache, g1_cache = g0.get('cache_metrics', {}), g1.get('cache_metrics', {})
    if g0_cache.get('prefix_cache_enabled') is not False or g1_cache.get('prefix_cache_enabled') is not True:
        return {'decision': 'reject', 'reason': 'G0/G1 的实际 prefix-cache 开关未形成关闭/开启对照'}
    if (g1_cache.get('hit_rate') is None or not g1_cache.get('hit_rate_evidence')
            or g1_cache.get('cache_capacity_blocks') is None):
        return {'decision': 'tune', 'reason': '缺少来自 backend 指标或服务日志的直接命中证据'}
    base, candidate = g0.get('metrics', {}), g1.get('metrics', {})
    ttft95_0, ttft95_1 = base.get('ttft_p95_ms'), candidate.get('ttft_p95_ms')
    throughput_0, throughput_1 = base.get('throughput_tokens_per_s'), candidate.get('throughput_tokens_per_s')
    if None in {ttft95_0, ttft95_1, throughput_0, throughput_1}:
        return {'decision': 'tune', 'reason': 'TTFT P95 或输出吞吐字段不完整'}
    if ttft95_1 < ttft95_0 and throughput_1 > throughput_0:
        return {
            'decision': 'accept',
            'reason': '当前共享前缀 workload 中,直接命中证据、请求完整性、TTFT P95 与吞吐对照均通过',
            'scope': '单卡、单并发、固定共享前缀 workload;输出完整性已检查,但未覆盖高并发、驱逐压力或语义质量评测',
        }
    if ttft95_1 > ttft95_0 and throughput_1 < throughput_0:
        return {'decision': 'reject', 'reason': '缓存配置使 TTFT P95 与吞吐同时退化'}
    return {'decision': 'tune', 'reason': '命中存在但性能信号不一致;升级共享前缀长度或并发后复测'}

quality_path = REPO_ROOT / f'benchmarks/results/69_evidence_record_{RUN_LABEL}.json'
registered_record = json.loads(quality_path.read_text(encoding='utf-8')) if quality_path.is_file() else {}
prefix_cache_decision = decide_prefix_cache_evidence(
    globals().get('prefix_evidence', {}), registered_record.get('quality'))
print({'decision': prefix_cache_decision, 'evidence_source': '5.5_and_registered_evidence',
       'quality_record': str(quality_path), 'quality_record_found': quality_path.is_file()})

相关阅读 ​

继续核对缓存的保留与回收、共享索引,以及请求竞争对命中的影响。

Released under the MIT License.