Skip to content

nodejumper-org/monad-preflight

Repository files navigation

monad-preflight

CI Go Report Card Go Reference License: MIT

Hardware readiness harness for Monad validators. A single static Go binary that you run on a bare-metal host before installing the node, to answer one question the existing tooling doesn't:

Will this machine hold MonadBFT's timing budget under load — or will it earn demerits once it's in the active set?

Monad validators must run on bare metal because MonadBFT proposes and votes in sub-second windows, and virtualization jitter, CPU downclocking, unpredictable NVMe tail latency, and cross-CCD core pinning all quietly break that budget. The knowledge about which settings matter is scattered across the official docs, the shipped systemd units, and the community Hardware Compatibility List as loose grep commands and BIOS tips. monad-preflight collects it into one tool that measures and delivers a verdict.

It is measurement, not monitoring. Existing community tools (validator managers with "doctor" checks, health-check cron stacks, dashboards) watch a node that is already running. This runs before the node exists and stresses the host to expose the latency tail that idle checks miss.

The tool never modifies your system. It only reads /proc and /sys, makes read-only syscalls (adjtimex, statfs), and writes a temp file to a directory you explicitly specify for the storage probe.

What it checks

Requirements marked (official) come from docs.monad.xyz or the systemd units shipped in the monad .deb; the rest are community operator knowledge (monadhcl.xyz, linked from the official hardware page) or engineering heuristics.

Probe What it verifies
virtualization Bare metal (official) — hypervisor ⇒ FAIL, container ⇒ WARN
cpu-spec 16 physical cores, ~4.5GHz base clock (official)
cpu-isa x86-64-v3 feature level monad-execution is compiled for, + 1GiB hugepage support (official)
ram 32 GB+ (official), with tolerance for kernel-reserved memory
os-release Ubuntu 24.04+ (official)
kernel-version ≥ 6.8.0-60 and not in the 6.8.0-56..59 hang-bug range (official)
io-uring kernel.io_uring_disabled — MonadDb does all TrieDB I/O via io_uring (official)
fd-limits fs.nr_open fits the units' LimitNOFILE=1048576 (official)
hugepages 1GiB hugepage support; 2048×2MiB + 4×1GiB reservation state (official)
smt SMT/HyperThreading disabled in BIOS (official)
cpu-governor performance governor on every core (community)
cstates No enabled deep C-states (community)
clock-sync Kernel clock NTP-disciplined, via read-only adjtimex(2) (community)
nic-speed Default-route link ≥ 300 Mbit/s (official), 1 Gbps baseline (community)
port-conflicts 8000 TCP+UDP and 8001 UDP not already bound (official ports)
ccd-topology monad-execution's SQPOLL cores share an L3/CCD domain (community)
nvme-devices No community-flagged NVMe models; PCIe Gen4 x4+ link (official)
triedb-device TrieDB drive: 2TB+, 512-byte LBA, non-rotational (official)
disk-space 500GB+ filesystem for MonadBFT/OS (official)
cpu-jitter Timer wake-up latency under self-generated load, p99.9 tail vs budget (heuristic)
nvme-latency O_DIRECT 4KiB random-write tail latency on the target drive (heuristic)

Usage

# Full run. --disk: writable dir on the OS/MonadBFT NVMe (e.g. future ledger dir).
# --triedb-dev: the drive you intend to dedicate to TrieDB (raw, no filesystem).
sudo ./monad-preflight --disk /home/monad --triedb-dev /dev/nvme1n1

# Fast smoke test.
./monad-preflight --quick

# Machine-readable output for CI / bi-weekly readiness reports.
./monad-preflight --json --disk /mnt/scratch > readiness.json

# Run a subset (see --list for names).
./monad-preflight --only cpu-jitter,nvme-latency --disk /mnt/scratch

Exit codes: 0 pass · 1 warn · 2 usage error · 3 fail · 4 inconclusive (nothing could be verified, the run was interrupted, or the report could not be written). A failing check beats interruption: an interrupted run that already produced a FAIL exits 3. Suitable for gating a deploy script — any non-zero code means "do not proceed blindly".

Some probes require privileges or a real host and will report SKIP rather than fail when run in a container, VM, or without root. A skip never counts as a pass: if every probe skips, the overall verdict is INCONCLUSIVE, not READY.

Example output

monad-preflight — validator hardware timing readiness
────────────────────────────────────────────────────────────
[PASS] Bare metal (no hypervisor or container)
    no hypervisor or container detected

[PASS] CPU meets core count and base clock minimums
    16 cores, base_clock 4.5GHz

[WARN] CPU frequency governor set to performance
    32 of 32 CPUs are not on 'performance' (schedutil on 32); downclocking adds wake latency
    fix: Set the governor to 'performance' on all cores: cpupower frequency-set -g performance

[FAIL] Storage write tail latency (O_DIRECT random 4KiB)
    storage tail latency too high (p99 2635.0us, p99.9 5898.0us, O_DIRECT random write); per-command latency this erratic will stall TrieDB
      p50                      120 us
      p99                      2635 us   (limit 1000 us)
      p99.9                    5898 us   (limit 2000 us)
      max                      11021 us
    fix: Use a PCIe Gen4x4+ NVMe from the community-vetted list...

────────────────────────────────────────────────────────────
17 pass  1 warn  1 fail  2 skip
Overall: NOT READY — fix the failing checks before running a validator
Note: thresholds are engineering heuristics, not official Monad limits. See README.

A machine-readable sample lives in docs/sample-output.json.

About the thresholds (read this)

Monad does not publish explicit per-metric timing limits for MonadBFT. Static requirements (cores, RAM, kernel, ports, hugepages, LBA format, ...) are taken directly from the official docs and shipped packaging. The timing thresholds, however, are engineering heuristics:

  • cpu-jitter measures a normal-priority sampler through the Go runtime — not a SCHED_FIFO cyclictest figure. Its limits (warn 500µs / fail 2000µs at p99.9) sit far above that stack's measurement floor, so only genuine host-side stalls (virtualization steal, downclocking, SMT contention) trip them.
  • nvme-latency runs a few seconds of O_DIRECT QD1 random writes. That exposes erratic per-command latency, but cannot reproduce sustained-load failure modes (SLC-cache exhaustion, thermal throttling). Treat PASS as necessary, not sufficient, and soak-test new drive models with fio.

Every limit lives in one file (internal/probe/thresholds.go) with its rationale and provenance, so it can be challenged and recalibrated as reference-node data becomes available.

What it deliberately does not check

  • Firewall rules and PPS policy (the anti-spam iptables rule, hashlimit sizing): reading nftables/iptables state needs root netlink dumps; verify with iptables-save per the install docs.
  • True WAN bandwidth/latency: needs an active speed test against real endpoints; the NIC link speed is checked as a local proxy.
  • On-chain requirements (stake, registration): not host properties.

Corrections and calibration data from operators are welcome — see CONTRIBUTING.md.

Build

No external dependencies (stdlib only).

make            # fmt-check + vet + test + build
make build      # just the host binary
make release    # cross-compiled static binaries + SHA256SUMS in dist/

Or directly with the Go toolchain:

go build -o monad-preflight ./cmd/monad-preflight
go test ./...

# static cross-compile
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o monad-preflight ./cmd/monad-preflight

Requires Go 1.22+.

Status

Beta. The measurement cores (CPU jitter, storage tail) and the twenty-odd config/spec probes work today. Planned: calibration against a published reference-node profile, a long-soak storage mode, and optional network-path latency checks against known peers.

License

MIT.

About

Pre-install hardware timing checks for Monad validators — measures CPU jitter and NVMe tail latency under load, flags a host that will miss MonadBFT deadlines before it earns demerits.

Resources

License

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors