Define clean-install HA qualification

[ankitgoswami]

Reviewable diff: +218/-22 across 5 files (excludes generated, test, and story files).

Summary

Defines the evidence required before the fixed three-host Fleet application, database, DCS, and VIP profile is called supported at a recorded fleet scale and topology. The packaged release must pass the complete procedure separately on Debian arm64 Raspberry Pi hardware, Ubuntu amd64, and 64-bit Raspberry Pi OS arm64. The release server and client images use digest-pinned base images so the qualified artifacts can be reproduced against the same upstream contents.

Stack: #887 -> #888 -> #889 -> #890 -> #891 -> #892. This diff is relative to guided installer PR #888. Application updates are implemented in #890 and #891; adjacent-release update qualification is in #892.

How it works

For each required platform, operators install the same packaged release concurrently on three dedicated hosts and record the release bundle and deployment-manifest digests, container image identities, host-package versions, host hardware, miner count, plugin mix, connection topology, command backlog, and timing evidence. The packaged Fleet images are built from digest-pinned Alpine and nginx bases so the recorded artifact identity is not changed by an upstream tag moving. Results from different platforms cannot be combined into one passing report.

The checklist verifies reboot persistence, Fleet and database host loss, acknowledged-write durability, writer exclusivity, passive request rejection, demotion cancellation, firewall behavior, interrupted-command recovery, active curtailment through failover, repeated failovers, and a 24-hour soak. Any evidence gap, overlapping old/new-holder work, or failed gate leaves that exact artifact and host profile unsupported.

flowchart LR
  P["Select one required platform"] --> I["Install one pinned release on three hosts"]
  I --> R["Verify reboot recovery"]
  R --> F["Run failure matrix"]
  F --> C["Verify commands and curtailment"]
  C --> S["Repeat failovers and soak"]
  S --> V{"Every gate passes?"}
  V -->|yes| N["Record this platform as qualified"]
  V -->|no| U["Keep this platform unsupported"]

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
deployment-files/ha/QUALIFICATION.md	Adds the redacted qualification checklist, platform matrix, evidence requirements, and result tables	Check that every support claim requires durable platform-specific evidence
deployment-files/ha/README.md	Distinguishes installer-compatible targets from qualified hardware profiles	Check that compatibility is not presented as proven support
docs/rfcs/0002-active-passive-fleet-ha.md	Aligns the support contract with deferred Fleet Node, scheduler-recovery, and alert-delivery work	Check the boundary of the first supported profile
deployment-files/server/Dockerfile and deployment-files/client/Dockerfile	Pin release base images by digest	Check that qualification records a reproducible image input rather than a moving tag

Key technical decisions & trade-offs

• Qualification uses packaged release artifacts rather than a source checkout.
• Base image tags remain readable but are paired with digests so qualification is tied to fixed upstream contents.
• Each required operating-system, architecture, and hardware combination gets its own complete report; passing one target does not make an architecture-wide claim.
• Support is limited to the exact artifact identities, host-package versions, recorded hardware, fleet scale and topology, and a trusted three-host HA segment.
• PROCESSING-command recovery covers server state; exactly-once device effects and device-side fencing remain outside the support claim.
• Every injected Fleet, PostgreSQL, etcd, or VIP fault carries an active curtailment target and separate 180-second shed and restoration bounds.
• Fleet Node reconnect scale, scheduler recovery, and alert delivery remain deferred.
• The procedure records outcomes and timings without adding test-only runtime behavior.

Testing & validation

• ./deployment-files/ha/tests/test-profile.sh
• git diff --check
• Real Debian arm64 Raspberry Pi, Ubuntu amd64, and Raspberry Pi OS arm64 qualification runs are pending.
• Hardware failovers, the 180-second curtailment target, repetition counts, and the 24-hour soak are pending, so the qualification verdict remains pending.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (9109c001d6dda403677645876ae6faa5a213deaf...b4255da76bcb565339124a0f0e3e2a2cc31ec68a, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: MEDIUM

Findings

[MEDIUM] Qualification instructions contradict the supported installer workflow

• Category: Reliability
• Location: deployment-files/ha/QUALIFICATION.md:61
• Description: The procedure instructs operators to copy node.env, secret directories, and the etcd root password separately, then run all three installations concurrently. The released installer instead requires ha-a to generate protected per-host bundles first; those bundles contain the configuration, scoped secrets, and the ha-a root password.
• Impact: The documented clean-install gate cannot be reproduced through the supported entrypoint. Manual staging could bypass bundle validation, mishandle cluster credentials, or produce an invalid qualification report.
• Recommendation: Follow the README workflow explicitly: run the ha-a wizard through bundle export, transfer each protected bundle with its release archive, then install the peers concurrently. Remove instructions for manually copying generated configuration, secrets, or the root password.

[MEDIUM] Server image remains mutable despite the digest pin

• Category: Infrastructure
• Location: deployment-files/server/Dockerfile:1
• Description: Pinning the Alpine base digest does not make the resulting image reproducible because the subsequent apk add --no-cache nmap gcompat resolves packages from mutable repositories. The HA installer rebuilds this image independently on each database host.
• Impact: Identical release bundles installed at different times can contain different, unqualified package code or fail to reproduce the image identities recorded by this qualification procedure.
• Recommendation: Build and bundle the complete API image in CI and verify its digest during installation, as done for the database image. Alternatively, use an immutable package snapshot and pin every installed package input.

Notes

The diff contains no runtime authentication, authorization, protobuf, plugin, or mining-pool configuration changes.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b4255da76b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Include DCS image evidence in qualification reports

When qualifying the three-host HA profile, the witness host runs the etcd/DCS container from deployment-files/ha/compose.yaml and does not run API, client, or Patroni, so this evidence field both asks for impossible image evidence on ha-c and omits the only image that provides quorum there. A report could therefore mark the fixed database/DCS/VIP profile as qualified without recording the deployed etcd image identity; please record the etcd image per host and scope API/client/Patroni evidence to the hosts that actually run them.

Useful? React with 👍 / 👎.