GPU Batch Reboot (NVIDIA B200, 66-day uptime bug)

Systemd timers + scripts to schedule periodic reboots of GPU compute nodes, with error handling, active health polling, and alerting.

Prerequisites

• ipmi-clush: used for IPMI power cycle commands. Install it from https://github.com/doitnowgroup/ipmi-clush
• Slurm (scontrol): for node drain/resume and reservations.
• ssh + nvidia-smi: for active health polling after reboot.

Installation

1. Install files

# Scripts
sudo install -m 755 -o root -g root scripts/gpu-batch-drain.sh   /usr/local/sbin/
sudo install -m 755 -o root -g root scripts/gpu-batch-reboot.sh  /usr/local/sbin/
sudo install -m 755 -o root -g root scripts/gpu-batch-alert.sh   /usr/local/sbin/

# Configuration
sudo mkdir -p /etc/gpu-batch
sudo install -m 644 -o root -g root etc/gpu-batch/batch1.conf /etc/gpu-batch/
sudo install -m 644 -o root -g root etc/gpu-batch/batch2.conf /etc/gpu-batch/

# IPMI credentials (required)
sudo install -m 600 -o root -g root etc/gpu-batch/ipmi-batch1.cred.example /etc/gpu-batch/ipmi-batch1.cred
sudo install -m 600 -o root -g root etc/gpu-batch/ipmi-batch1.cred.example /etc/gpu-batch/ipmi-batch2.cred

# Alerts (optional but recommended)
sudo install -m 600 -o root -g root etc/gpu-batch/alert.conf.example /etc/gpu-batch/alert.conf

# Systemd units
sudo install -m 644 -o root -g root systemd/*.service systemd/*.timer /etc/systemd/system/
sudo systemctl daemon-reload

2. Configure

Edit all config files with your actual values:

sudo vi /etc/gpu-batch/batch1.conf       # set NODES and RESNAME for batch 1
sudo vi /etc/gpu-batch/batch2.conf       # set NODES and RESNAME for batch 2
sudo vi /etc/gpu-batch/ipmi-batch1.cred  # set IPMI_USER and IPMI_PASSWORD
sudo vi /etc/gpu-batch/ipmi-batch2.cred
sudo vi /etc/gpu-batch/alert.conf        # set WEBHOOK_URL and/or ALERT_MAIL

The NODES field uses Slurm node range syntax (e.g. b200n00[1-8] or gpu[001-032]). The RESNAME is the reservation name shown in sinfo/scontrol.

3. Enable

sudo systemctl enable --now gpu-batch-drain@batch1.timer
sudo systemctl enable --now gpu-batch-drain@batch2.timer
sudo systemctl enable --now gpu-batch-reboot@batch1.timer
sudo systemctl enable --now gpu-batch-reboot@batch2.timer

Schedule

Batches alternate so both are never down at the same time. Each batch reboots every 2 months with a ~17 day gap between drain and reboot (well within the max QOS of 14 days).

	Drain	Reboot
Batch 1 (odd nodes)	Day 15 of odd months, 08:00	Day 1 of even months, 08:00
Batch 2 (even nodes)	Day 15 of even months, 08:00	Day 1 of odd months, 08:00

The maintenance reservation is created with unlimited duration and automatically deleted after the reboot cycle completes.

Verification

# View next execution of each timer
systemctl list-timers 'gpu-batch-*'

# Test a service manually (recommended before trusting the timer)
sudo systemctl start gpu-batch-drain@batch1.service
sudo journalctl -u gpu-batch-drain@batch1.service -f

# Review history of a reboot cycle
sudo journalctl -u gpu-batch-reboot@batch1.service --since "-1 day"

Pending adjustments for your environment

• ssh in gpu-batch-reboot.sh assumes root can connect by key to compute nodes without a password; adjust user/key if that is not the case.
• POLL_TIMEOUT (15 min) and POLL_INTERVAL (30s) in gpu-batch-reboot.sh are adjustable if POST + init of the B200s takes longer or shorter on your hardware.

Author

SergioZ3R0 · @SergioZ3R0