Skip to content

Clock pinning is UNAVAILABLE inside an rc lease: nvidia-smi -lgc returns LGC_RC=4 as root, and every clock-pinned figure in the records was taken over the retired ssh+flock path #1354

Description

@localai-bot

Measured 2026-08-19 on dgx:gpu0, in three separate rc run jobs, each running as root in the worker pod:

$ nvidia-smi -lgc 2190
The current user does not have permission to change clocks for GPU 0000000F:01:00.0.
LGC_RC=4

The container is root but lacks the capability the driver requires.

Why this outlives the campaign that found it

.agents/benchmarking.md §"The clock is part of the measurement" instructs "Pin the clocks before measuring, under the lock", and offers sudo nvidia-smi -lgc 2100 as the recipe. That recipe works on the host path. Every clock-pinned figure in this repository was taken over the host + ssh + flock path, and AGENTS.md now forbids ssh to a fleet device. So the migration to rc leases silently removed clock pinning, and no record said so.

Same class as #1265: a capability the records assume, which the current access path does not provide. It is visible debt rather than a surprise now, but it was a surprise on 2026-08-19.

What it costs, measured

Inside a lease the SM clock can only be SAMPLED. tools/bench/gpu_clock_state.py still works and is the only attribution such a figure carries. On a thermally throttling GB10 that is not enough for a PAIR:

  • Nine timed windows across two arms on 2026-08-19 recorded within-run SM-clock spreads of 12.92% to 26.36% against the 5% ceiling .agents/benchmarking.md sets. SwThermalSlowdown was active in every window; one also carried HwSlowdown + HwThermal.
  • gpu_clock_state compare returned PAIRING_VERDICT=DISCARD on all three c1 pairings, even though the cross-arm rule passed perfectly — same boot id, both arms at a 2489 MHz median, median offset 0.0%.
  • The consequence is that the Qwen3.8-27B bf16 c1 cell has two clean, complete absolutes from both arms and no ratio, and the ratio is owed rather than measurable. That is GPU time spent on a pair that could not be divided.

What would settle it

One of:

  1. A lease that grants the driver capability -lgc needs (an rc worker change), which restores the documented method.
  2. A recorded, ratified alternative: a thermal settle-and-hold procedure that gets the within-run spread under 5% without pinning, demonstrated rather than assumed.
  3. An explicit decision that lease-measured pairs use a different clock rule, argued in a spec, with gpu_clock_state's ceiling reconciled to it. Not a widening of the assertion to make a red go green.

Records updated in the meantime: .agents/environment.md records the refusal beside the other measured lease capabilities, .agents/benchmarking.md names the exception where it instructs the pin, and .agents/benchmark-record.md carries the measurement.

Found while recording the Qwen3.8-27B bf16 c1/c8 re-measure (#915, #979). Not fixed in flow: the fix is an infrastructure capability this row has no authority over, and options 2 and 3 are each a spec of their own.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions