Measured 2026-08-19 on dgx:gpu0, in three separate rc run jobs, each running as root in the worker pod:
$ nvidia-smi -lgc 2190
The current user does not have permission to change clocks for GPU 0000000F:01:00.0.
LGC_RC=4
The container is root but lacks the capability the driver requires.
Why this outlives the campaign that found it
.agents/benchmarking.md §"The clock is part of the measurement" instructs "Pin the clocks before measuring, under the lock", and offers sudo nvidia-smi -lgc 2100 as the recipe. That recipe works on the host path. Every clock-pinned figure in this repository was taken over the host + ssh + flock path, and AGENTS.md now forbids ssh to a fleet device. So the migration to rc leases silently removed clock pinning, and no record said so.
Same class as #1265: a capability the records assume, which the current access path does not provide. It is visible debt rather than a surprise now, but it was a surprise on 2026-08-19.
What it costs, measured
Inside a lease the SM clock can only be SAMPLED. tools/bench/gpu_clock_state.py still works and is the only attribution such a figure carries. On a thermally throttling GB10 that is not enough for a PAIR:
- Nine timed windows across two arms on 2026-08-19 recorded within-run SM-clock spreads of 12.92% to 26.36% against the 5% ceiling
.agents/benchmarking.md sets. SwThermalSlowdown was active in every window; one also carried HwSlowdown + HwThermal.
gpu_clock_state compare returned PAIRING_VERDICT=DISCARD on all three c1 pairings, even though the cross-arm rule passed perfectly — same boot id, both arms at a 2489 MHz median, median offset 0.0%.
- The consequence is that the Qwen3.8-27B bf16 c1 cell has two clean, complete absolutes from both arms and no ratio, and the ratio is owed rather than measurable. That is GPU time spent on a pair that could not be divided.
What would settle it
One of:
- A lease that grants the driver capability
-lgc needs (an rc worker change), which restores the documented method.
- A recorded, ratified alternative: a thermal settle-and-hold procedure that gets the within-run spread under 5% without pinning, demonstrated rather than assumed.
- An explicit decision that lease-measured pairs use a different clock rule, argued in a spec, with
gpu_clock_state's ceiling reconciled to it. Not a widening of the assertion to make a red go green.
Records updated in the meantime: .agents/environment.md records the refusal beside the other measured lease capabilities, .agents/benchmarking.md names the exception where it instructs the pin, and .agents/benchmark-record.md carries the measurement.
Found while recording the Qwen3.8-27B bf16 c1/c8 re-measure (#915, #979). Not fixed in flow: the fix is an infrastructure capability this row has no authority over, and options 2 and 3 are each a spec of their own.
Measured 2026-08-19 on
dgx:gpu0, in three separaterc runjobs, each running as root in the worker pod:The container is root but lacks the capability the driver requires.
Why this outlives the campaign that found it
.agents/benchmarking.md§"The clock is part of the measurement" instructs "Pin the clocks before measuring, under the lock", and offerssudo nvidia-smi -lgc 2100as the recipe. That recipe works on the host path. Every clock-pinned figure in this repository was taken over the host +ssh+flockpath, and AGENTS.md now forbidssshto a fleet device. So the migration torcleases silently removed clock pinning, and no record said so.Same class as #1265: a capability the records assume, which the current access path does not provide. It is visible debt rather than a surprise now, but it was a surprise on 2026-08-19.
What it costs, measured
Inside a lease the SM clock can only be SAMPLED.
tools/bench/gpu_clock_state.pystill works and is the only attribution such a figure carries. On a thermally throttling GB10 that is not enough for a PAIR:.agents/benchmarking.mdsets.SwThermalSlowdownwas active in every window; one also carriedHwSlowdown + HwThermal.gpu_clock_state comparereturnedPAIRING_VERDICT=DISCARDon all three c1 pairings, even though the cross-arm rule passed perfectly — same boot id, both arms at a 2489 MHz median, median offset 0.0%.What would settle it
One of:
-lgcneeds (anrcworker change), which restores the documented method.gpu_clock_state's ceiling reconciled to it. Not a widening of the assertion to make a red go green.Records updated in the meantime:
.agents/environment.mdrecords the refusal beside the other measured lease capabilities,.agents/benchmarking.mdnames the exception where it instructs the pin, and.agents/benchmark-record.mdcarries the measurement.Found while recording the Qwen3.8-27B bf16 c1/c8 re-measure (#915, #979). Not fixed in flow: the fix is an infrastructure capability this row has no authority over, and options 2 and 3 are each a spec of their own.