Exploratory case study · ep1-f005探索性案例研究 · ep1-f005
After the original test passes,
is the vulnerability actually fixed?
原题通过之后，
漏洞是否真的修好了？
Using one real credential-leak bug, we look at whether the patches produced when different models drive the same Hermes Agent also cover an extra boundary: copying the client.
以一个真实的凭据泄漏缺陷为例，观察不同模型驱动同一 Hermes Agent 完成修复后，补丁能否覆盖客户端复制这一额外边界。
- Research question研究问题
- Is passing the original regression test enough to show that a fix meets the fuller credential-isolation requirement? When a model's answer identifies a risk, does that risk get handled in the code?通过原始回归测试，是否足以说明修复满足更完整的凭据隔离要求？模型回答中识别出的风险，是否落实到了代码？
- Method方法
- 9 models, two runs each, with a 420-second budget for the main experiment; plus two 1200-second probes for GLM-5.3. Afterwards, the archived constructor functions, the unfixed baseline and the upstream reference function all went through the same HTTP boundary checks.9 个模型各运行两次，主实验预算 420 秒；另对 GLM-5.3 做两次 1200 秒探针。随后对归档构造函数、未修复基线和上游参考函数实施统一 HTTP 边界检查。
- Main finding主要发现
- Scope解释范围
- This report compares system runs under a fixed task, prompt, Agent, tools and budget. Single task, small sample; it does not estimate any model's overall success rate. The per-dimension placings in section 05 describe only these 18 runs; they are not a model capability ranking.本报告比较的是指定任务、提示、Agent、工具与预算下的系统运行。单任务、小样本；不估计模型总体成功率。第 05 节按维度给出的名次只描述这 18 次运行，不是模型能力排名。
01 · Task background01 · 任务背景
Correct behavior: a request may carry only the selected credential
正确行为：请求只能携带选定的凭据
After an API key is set for a third-party vendor, the SDK still reads a separate Bearer token from the environment, so a single request carries two sets of credentials. The task requires stopping the extra credential from being sent while keeping legitimate authentication. The fix adopted upstream is the reference implementation; other behaviorally equivalent implementations are allowed.
给第三方供应商设置 API key 后，SDK 仍会从环境读取另一份 Bearer token，导致同一请求携带两套凭据。任务要求阻止多余凭据被发送，同时保留合法鉴权。上游采纳的方案是参考实现；允许行为等价的其他实现。
| Scenario | Keep | Exclude | How this report checks it |
|---|---|---|---|
| Original API key request | The specified X-Api-Key | Bearer token from the environment | Original regression test + function re-check |
| Original Bearer request | The specified Authorization | API key from the environment | Function re-check; the baseline already passes |
| Both requests after copying the client | Same as the original request | Environment credentials brought back in by the copy | Post-experiment re-check; not scored in the original task |
| 场景 | 应保留 | 应排除 | 本报告如何检查 |
|---|---|---|---|
| API key 原始请求 | 指定的 X-Api-Key | 环境中的 Bearer token | 原题回归测试 + 函数补测 |
| Bearer 原始请求 | 指定的 Authorization | 环境中的 API key | 函数补测；基线已通过 |
| 复制客户端后的两种请求 | 与原始请求相同 | 复制时重新带入的环境凭据 | 实验后补测；非原题评分项 |
Expand: issue timeline and upstream reference answer展开：问题时间线与上游参考答案
0genlab found and reproduced a cross-vendor credential leak
0genlab 发现并复现凭据跨厂商泄漏
With a third-party API key explicitly configured, the SDK still attached the Anthropic Bearer credential from the environment to requests. Reproduced with a local HTTP server and filed as an issue.
显式配置了第三方 API key，SDK 仍把环境里的 Anthropic Bearer 凭据附加到请求中。用本地 HTTP 服务复现并提交 issue。
Issue #105774 ↗问题 #105774 ↗After plain requests were fixed, copied clients still leaked
修复普通请求后，发现复制客户端仍会泄漏
kokhlo proposed a fix; 0genlab re-tested the with_options() copy path and proposed a header-level Omit approach, which the author adopted.
kokhlo 提出修复；0genlab 补测 with_options() 复制路径，提出请求头层的 Omit 方案，作者采纳。
Copy-path evidence ↗复制路径证据 ↗ · Author adopts it ↗作者采纳 ↗Further discussion on the compatibility fallback and reverse authentication
多人继续讨论兼容兜底与反向鉴权
egilewski pointed out that the compatibility branch loses protection when Omit is unavailable; 0genlab added HTTP test results and a fix contribution, and flagged the copy risk for the reverse credential.
egilewski 指出 Omit 不可用时兼容分支会失去保护；0genlab 补充 HTTP 实测和修复贡献，同时指出反向凭据的复制风险。
Review comment ↗评审意见 ↗ · Repro and contribution ↗复现与贡献 ↗Upstream merged the two-way Omit approach
上游最终合入双向 Omit 方案
kshitijk4poor took over the original PR, using sdk.Omit() to drop the unselected auth header and keeping the protection after the client is copied. Merge commit 036a20b3.
kshitijk4poor 接续原 PR，通过 sdk.Omit() 删除未选定的鉴权头，并保持客户端复制后的保护。合并提交 036a20b3。
Final fix #107978 ↗最终方案 #107978 ↗
All of the above are public GitHub records; each entry's date is on its left. This page re-checked the links and merge status on 2026-09-21 (#105774 closed on 2026-09-11; #107978 merged the same day).
以上均为 GitHub 公开记录，事件日期见每条左侧；本页于 2026-09-21 复核过链接与合并状态（#105774 于 2026-09-11 关闭，#107978 同日合并）。
The upstream PR was verified on SDK 0.87.0; the supplementary checks in this report all use SDK 0.94.0 from the frozen image.
上游 PR 在 SDK 0.87.0 上验证；本报告的补充检查统一使用冻结镜像的 SDK 0.94.0。
02 · Study design02 · 研究设计
Three layers of evidence, each answering one question
三层证据，分别回答三个问题
| Evidence layer | Question | Criterion and use |
|---|---|---|
| Original-task acceptance | Did it pass the acceptance check that was runnable at the time? | Scored after restoring the original regression_test.py; the only regression test covers the original API key request. The task text also asks to keep OAuth working, but that test does not cover it. |
| Fix robustness | Do the archived functions pass the four boundary checks? | Two auth types × original/copied request; same SDK and local HTTP. Post-hoc exploration; does not rewrite the historical score. |
| Answer evidence | How do the replies describe risk and verification? | Visible content read under one rule set; “mentions it”, “plans it”, “claims to have verified it” and “implements the protection” are recorded separately and not combined into a thoroughness score. |
| 证据层 | 问题 | 判据与用途 |
|---|---|---|
| 原题验收 | 是否完成当时可执行的验收？ | 恢复原始 regression_test.py 后判分；只有 API key 原始请求这一项回归测试。任务文字另要求保留 OAuth，但该测试未覆盖它。 |
| 修复稳健性 | 归档函数能否满足四项边界检查？ | 两种鉴权 × 原始/复制请求；统一 SDK 与本地 HTTP。属于事后探索，不改写历史 score。 |
| 回答证据 | 答复如何描述风险与验证？ | 按统一规则阅读可见内容；“说到、计划、声称验证、实现保护”分别记录，不合成全面性分数。 |
Experimental conditions and evidence completeness
实验条件与证据完整性
| Item项目 | Recorded for this run本次记录 | Comparability and limits可比性与限制 |
|---|---|---|
| Task and prompt任务与提示 | Same old source, TASK.md and original test相同旧源码、TASK.md、原始测试 | The root cause, file location and the None trap were given; this evaluates implementing a fix, not finding the root cause independently. Neither the upstream patch nor copy acceptance was given.已给根因、文件位置与 None 陷阱；评估实施修复，不是独立发现根因。未给上游补丁和复制验收。 |
| Sample and inclusion rule样本与纳入规则 | Two fixed snapshots, 18 main runs + 2 probes固定两份快照，18 次主实验 + 2 次探针 | All timeouts kept; 1 p1 smoke test excluded by task. Earlier calibration runs are not part of this cohort.保留全部超时；按 task 排除 1 次 p1 冒烟测试。较早校准运行不属于本队列。 |
| Models and channel模型与渠道 | 9 archived model identifiers, accessed through AIHubMix9 个归档模型标识，经 AIHubMix 接入 | Model names are not reconstructable weight versions; runs span several days. Upstream service changes are not fully controlled.模型名称不是可重建权重版本；运行跨日。上游服务变化未被完全控制。 |
| Agent and environmentAgent 与环境 | Same image, separate containers, 2 CPU / 4 GB, pinned SDK相同镜像、独立容器、2 CPU / 4 GB、固定 SDK | Agent and task source are pinned separately. Host-side scoring dependencies are not fully frozen.Agent 与任务源码分别锁定。宿主判分依赖未完整冻结。 |
| Budget预算 | 420 s for the main experiment; a separate 1200 s group for GLM; max_turns=20420 秒主实验；GLM 单独 1200 秒组；max_turns=20 | The extended group consists of new runs, not continuations; it is not pooled with the main experiment to estimate success rates.延时组为新运行，非续跑；不与主实验混合估计成功率。 |
| Sampling and inference settings采样与推理配置 | This report does not yet aggregate temperature, seed, reasoning budget etc. per request本报告尚未逐请求汇总 temperature、seed、推理预算等 | Marked as pending verification; the same Agent configuration alone does not establish that every upstream applied the same effective parameters.标为待核对；不能仅凭同一 Agent 配置断言各上游的有效参数一致。 |
| Run schedule运行安排 | Actual times kept in the snapshots实际时间由快照保留 | No randomized execution order or time blocking; platform load, retries and similar effects are not independently controlled.未建立随机执行顺序或时间分块设计；平台负载、重试等影响未独立控制。 |
| Re-checks and text review补测与文本审阅 | Function-level re-check; a single AI-assisted text review with no independent second reviewer; tool-execution claims in the answers were not verified one by one.函数级补测；单次 AI 辅助文本审阅，无独立第二审阅者，未逐项复核回答中的工具执行声明。 |
Expand: versions, resources, network and reproduction limits展开：版本、资源、网络与复现限制
Expand: metric definitions, scoring implementation and data sources展开：指标定义、判分实现与数据来源
- score / exit
- score=1 means the original regression test passed; exit=0 means the Agent ended normally and exit=124 means the outer timeout fired. They are different criteria.
- Test-modification flag
- score.sh restores the original test on the host before running it; test_modified_by_model is recorded separately and does not deduct points automatically. It is false for all 20 runs in this episode.
- wall_s
- Timed from just before the run starts until the container has ended and its logs have been collected and removed; later scoring is not included.
- API call count
- official uses Hermes api_calls; wire counts only recorded responses with valid usage and may miss some failures and retries. Not the same as Agent main-loop turns.
- Input accounting
- For wire, uncached input = prompt_tokens − cached_tokens; official uses the exported input_tokens, with cache reads listed separately. Listed means are arithmetic means of two runs.
- Re-check scope
- Archived constructor functions + anthropic 0.94.0 + a loopback HTTP server inside the container; the container has no internet access and calls no model. Re-check date: .
- score / exit
- score=1 表示原始回归测试通过；exit=0 表示 Agent 正常结束，exit=124 表示外层超时。两者不是同一判据。
- 测试修改标记
- score.sh 在宿主恢复原始测试再执行；test_modified_by_model 独立记录，不自动扣分。本期20次均为 false。
- wall_s
- 从运行启动前到容器结束、日志收集及移除后计时，不含后续判分。
- API 调用数
- official 取 Hermes api_calls；wire 只计带有效 usage 的实录响应，未必覆盖全部失败和重试。不等于 Agent 主循环轮次。
- 输入口径
- wire 的未缓存输入 = prompt_tokens − cached_tokens；official 使用导出 input_tokens，缓存读取另列。列表均值为两次算术平均。
- 补测范围
- 归档构造函数 + anthropic 0.94.0 + 容器内回环 HTTP 服务；容器无外网，不调用模型。补测日期：。
Task text given to the model模型任务原文 · Task source version任务源码版本 · Image lock file镜像锁文件 · Scoring script判分脚本
03 · Results03 · 结果
Every patch that passed the original task still misses the copy boundary
通过原题的补丁，仍全部漏掉复制边界
Figure 1 · Original-task result and re-check results for each run
图 1 · 每次运行的原题结果与补测结果
Models sorted by name · click a run id to compare the evidence side by side模型按名称排列 · 点击运行编号并排核对证据| Implementation / run实现 / 运行 | Original task原题结果 | API key re-checkAPI key 补测 | Bearer re-checkBearer 补测 | ||
|---|---|---|---|---|---|
| Original request原始请求 | Copied request复制请求 | Original request原始请求 | Copied request复制请求 | ||
Figure 2 · Duration of two runs under the same budget
图 2 · 相同预算下的两次运行耗时
Each dot is one main-experiment run; a model's two runs sit on separate lines. ● / ○ mean passed and ■ means timed out. Models are sorted by name; the right column shows original-task passes across the two runs.
每个点是一场主实验；同一模型的两次运行上下分开。● / ○ 为通过，■ 为超时。模型按名称排列；右侧为两次原题通过数。
Supplementary observation: GLM extended-budget probe (two independent runs)补充观察：GLM 延时探针（独立两次运行）
B · Separate extended-budget probe / glm-5.3B · 单独延时探针 / glm-5.3
With a larger budget, the result went from fail to pass
增加预算后，结果从未通过变为通过
Figure A · two groups of independent runs附图 A · 两组独立运行Same image and resources, with the outer budget raised from 420 to 1200 seconds, run twice from scratch. The chart below uses a single time scale; the dashed lines mark each group's budget.
固定同一镜像和资源，把外层预算从 420 秒改为 1200 秒，重新运行两次。下图是同一时间刻度；虚线表示各组预算。
04 · Mechanism and qualitative review04 · 机制解释与定性审阅
The protection stays on the client instance and does not survive a copy
保护停在客户端实例上，复制后没有延续
Figure 3 · From construction to copy, where does the protection fail?
图 3 · 从构造到复制，保护在哪一步失效？
| Approach实现方式 | ① Construct client① 构造客户端 | ② Original request② 原始请求 | ③ Copy client③ 复制客户端 | ④ Request after copy④ 复制后的请求 |
|---|---|---|---|---|
| Unfixed baseline未修复基线 | SDK reads the environment tokenSDK 读取环境 token | Extra credential sent ×带出额外凭据 × | Constructed again, reads the environment again再次构造，继续读取环境 | Extra credential still sent ×仍带出额外凭据 × |
| Clear attribute after construction构造后清空属性 | Read environment → clear auth_token读取环境 → 清空 auth_token | Current instance sends no extra credential ✓当前实例不带额外凭据 ✓ | None makes the SDK read the environment againNone 触发 SDK 再次读取环境 | Extra credential reappears ×额外凭据重新出现 × |
| Temporarily remove the environment variable临时移除环境变量 | Remove variable → construct → restore variable移除变量 → 构造 → 恢复变量 | Current instance sends no extra credential ✓当前实例不带额外凭据 ✓ | Reads the token from the restored environment从已恢复的环境读取 token | Extra credential reappears ×额外凭据重新出现 × |
| Upstream header exclusion上游请求头排除 Reference function参考函数 | Stores a rule that explicitly excludes the header保存显式排除头的规则 | Omit removes the extra header when the request is built ✓构建请求时由 Omit 删除多余头 ✓ | The copy keeps the default_headers rule复制保留 default_headers 规则 | Still excluded at the header layer ✓仍在请求头层排除 ✓ |
Did the answers identify the copy risk? One rule set applied to every run
回答是否识别了复制风险？逐次使用同一套规则
145 non-empty visible replies, including progress notes and final replies; 21 auxiliary title-generation replies are excluded. reasoning_content is not read, and tool arguments or tool output are not treated as model answers. What follows is a single AI-assisted qualitative review with no independent re-check yet.
145 条非空可见答复，包含过程说明与最终答复；另排除 21 条辅助标题生成回复。不读取 reasoning_content，不把工具参数或工具输出当作模型回答。以下是单次 AI 辅助定性审阅，尚无独立复核。
Review rules and decision boundaries (version 1.0)审阅规则与判定边界（版本 1.0）
All 20 runs use the same review scope. “Not stated explicitly” describes only the saved visible content and cannot be used to infer that the model did not consider it. Mentioning Omit counts as an intent to check, not as a completed fix. Restating the root cause and the None trap already given in the task does not count as an independent finding.
所有 20 次运行使用相同审阅范围。“未见明确表述”只描述已保存的可见内容，不能推断模型没有考虑。提及 Omit 只记为排查意图，不记为完成方案。复述任务已给出的根因与 None 陷阱，不算独立发现。
Expand: unified text-review table for all 20 runs展开：全部 20 次运行的统一文本审阅表
| Model / run模型 / 运行 | Copy-boundary statement复制边界表述 | Omit statementOmit 表述 | Final delivery claim最终交付声明 | Independent result: original / copy独立结果：原题 / 复制 |
|---|
Three representative excerpts: identifying the risk, scoping it out, mentioning the mechanism三个代表性片段：识别风险、限定范围、提及机制
05 · Stratified ranking05 · 分层排名
Four questions ranked separately: who located the problem accurately, and who delivered fast
拆成四个问题分别排：谁定位得准，谁交付得快
Figure 4 · Per-model details on four dimensions, with overall placing
图 4 · 四个维度的逐模型明细与综合名次
| Overall综合 | Model模型 | Original passes原题通过 | Mean time耗时均值 | Copy-boundary statement复制边界表述 | Copy re-check复制补测 | Mean API callsAPI 调用均值 | Approach实现方式 |
|---|
06 · Discussion and limitations06 · 讨论与局限
What does this experiment support, and what can it not support yet?
这次实验支持什么，还不能支持什么？
| Supported by this evidence | Cannot be extrapolated from it | Evidence needed next |
|---|---|---|
| All 16 implementations that passed the original task failed the copy re-check | “These models generally can't fix security bugs” | Several independent tasks, more repetitions, and a task-sampling rule fixed in advance |
| GLM r14 explicitly described the risk of the copy re-reading the environment, but that run did not finish | “GLM thinks most thoroughly”, or “naming the risk means solving it” | An independent re-check of the unified review rules, combined with tool execution and what actually landed in code |
| Both runs in the GLM extended group passed the original task | A larger budget is the only reason the result changed | A controlled budget comparison, execution-order control and more repetitions |
| Under the specified SDK, the reference function passes all four checks | The reference fix has passed a full security audit | Checks on full call paths, concurrency, multiple versions and other runtime environments |
| 本次证据支持 | 不能据此外推 | 下一步所需证据 |
|---|---|---|
| 16 份原题通过实现均未通过复制补测 | “这些模型普遍不能修安全漏洞” | 多个独立任务、更多重复、预先确定的任务抽样规则 |
| GLM r14 明确表述复制重读环境风险，但该次未完成 | “GLM 思考最全面”或“说到风险就代表会解决” | 统一审阅规则的独立复核，结合工具执行与代码落实 |
| GLM 延时组另外两次通过原题 | 预算增加是结果变化的唯一原因 | 受控预算对照、执行顺序控制及更多重复 |
| 在指定 SDK 下，参考函数四项检查均通过 | 参考方案已通过完整安全审计 | 完整调用路径、并发、多个版本与其他运行环境检查 |
Validity limits: the task supplied the root cause; model calls spanned several days and versions were not fully pinned; each model had only two main runs; the re-check is post-hoc exploration; the copy check covers functions, not the full Agent; the text reviewer knew the experiment results, so interpretation bias is possible. These limits are kept next to the conclusions rather than hidden at the bottom of the page.
效度限制：任务已提供根因；模型调用跨日且版本未完全固定；单模型主实验仅两次；补测为事后探索；复制检查覆盖函数而非完整 Agent；文本审阅者知道实验结果，存在解释偏差。以上限制保留在结论附近，不藏在页尾。
07 · Developer actions and evidence07 · 开发者行动与证据
Turn this finding into acceptance requirements
把这次发现转成验收要求
- Extend from one request to the client lifecycle: acceptance checks both the original instance and the copied instance; for both auth types, the specified credential is kept and the extra one is excluded.
- Make completion claims match the checks actually run: the report lists the scenarios that were executed and keeps untested items explicit, instead of letting “fixed” stand in for the verified scope.
- Fix the rules before the next round: if the copy scenario becomes a formal scoring item, create a new task version, a new cohort and new results; do not rewrite this round's scores.
- 从一次请求扩展到客户端生命周期：验收同时检查原始实例和复制实例；两种鉴权均保留指定凭据、排除额外凭据。
- 让完成声明对应实际检查：报告列出执行过的场景，未测项明确保留，不用“已修复”替代验证范围。
- 下一轮先固定规则再运行：若把复制场景升级为正式评分项，应建立新任务版本、新队列与新结果，不回写本次成绩。
Appendix A · Per-run details and input usage (20 runs · with tokens and sources)附录 A · 逐次运行明细与输入用量（20 次 · 含 token 与来源）
Putting two budgets in one table would make the probes read as part of the main experiment, so the runs are split by budget: 9 models × 2 runs in the main experiment, and two extended-budget probes for GLM-5.3 only. “Result” combines the original-task score with the exit status; the three token columns use the same accounting as the notes in section 02. Source label official is the Agent's exported ledger and wire is recomputed from relay recordings (only the 4 timed-out runs needed backfilling this time). None of the 20 runs modified the test file, and scoring ran after the original test was restored on the host, so that field is not listed. The two gpt-6-astra runs show only 24 uncached input tokens against nearly 100k cache reads — that route's cache covers almost everything, so a single-digit net input is normal, not missing data.
同一张表里混放两种预算会把探针读成主实验的一部分，所以按预算分开：主实验 9 个模型各 2 次，延时探针只针对 GLM-5.3 两次。「结果」合并了原题判分与退出状态；token 三列与 02 节口径说明同源，来源标签 official 为 Agent 导出账本、wire 为 relay 录制重算（本次仅 4 次超时运行需要回填）。20 次运行均未修改测试文件，判分在宿主恢复原始测试后执行，因此不单列该字段。其中 gpt-6-astra 两次的未缓存输入只有 24、缓存读取却近 10 万——该路缓存接近全覆盖，净输入只剩个位数是正常态，不是缺数据。
Main experiment · 420-second budget · 18 runs主实验 · 420 秒预算 · 18 次运行
| Model · run模型 · 运行 | Result结果 | Time耗时 | API callsAPI 调用 | Uncached input未缓存输入 | Cache reads缓存读取 | Output输出 |
|---|
Extended-budget probe · 1200-second budget · 2 runs延时探针 · 1200 秒预算 · 2 次运行
| Model · run模型 · 运行 | Result结果 | Time耗时 | API callsAPI 调用 | Uncached input未缓存输入 | Cache reads缓存读取 | Output输出 |
|---|
Input and cache usage输入与缓存用量
How much of the total input came from cache?
总输入里，有多少来自缓存？
Arithmetic mean of two runs per model · 420-second main experiment only每模型两次算术均值 · 仅 420 秒主实验The two tables above hold per-run values; here the same batch is shown per model, averaging the two runs and splitting the input into uncached input and cache reads. Timed-out runs only include usage recorded before they stopped; data sources and coverage are described in the section 02 accounting notes, and these numbers cannot be used directly to compare cost.
上面两张表是逐次数值，这里换成模型级的同批数字：两次运行取算术平均，看未缓存输入与缓存读取各占多少。超时运行只累计停止前的已记录用量；数据来源与覆盖范围详见 02 节口径说明，不能直接据此比较费用。
Appendix B · Other issues and toolchain notes (7 entries)附录 B · 其他问题与工具链记录（7 条）
Supplementary records补充记录
What else turned up during the experiment?
实验过程中还发现了什么？
7 entries · expand as needed7 条记录 · 按需展开Includes real bugs, early calibration issues and toolchain observations; each phase is labeled separately, and historically contaminated runs are not counted in this episode's results.
包括真实漏洞、早期校准问题和工具链观察；不同阶段分开标注，不把历史污染运行计入本期结果。
Evidence index and review status
证据索引与复核状态
| Artifact | Entry point | Current status |
|---|---|---|
| Run snapshots | Main experiment · Extended group | Raw values kept; charts generated from the snapshots |
| Boundary re-check | Results and function hashes · Check script | Baseline, reference and 20 functions; 88 requests actually captured |
| Visible answers and review | 145 visible replies · Rules and per-run decisions | Checked against raw response hashes and excerpts; not independently re-reviewed |
| Report build | Build script · Visible-answer extraction script | Can be regenerated locally; a full environment rebuild is still limited by the image and dependency records |