Skip to content

Commit 60e20ec

Browse files
SunrisepeakSunrisepeak
committed
ci(mirror): 每资源计时 + 2m 上传上限(超时 warn 并跳过)
v0.0.94 发布时 publish-ecosystem 被 30min job timeout 杀掉,日志里看不出是 哪个资产在慢 —— 事后查是 gtc 跨境传 linux-x86_64(8.4MB)/linux-aarch64 (30.5MB)两个大件卡住,最后靠本地 gtc 手工补齐。 - **每资源计时**:上传返回即打印 `<github|gitcode> <资产> (<大小>) uploaded in Ns`; 已 serving 的仍先 probe → 打印 skipping,不重传(原有行为,补上 host 标签)。 - **2m 上限**(`MIRROR_UPLOAD_TIMEOUT`,默认 120s):超时打印 WARN + **放弃该资产**, 后续轮次不再重试。 - 每条 mirror leg 结束打印耗时;失败时 hint 出手工补传命令。 **为何这个 cap 与 v0.0.90 事故不同**:那次 `timeout 300` 杀掉慢但**在推进**的上传 后**重传**,每次从字节零开始,永不收敛,把 20min job 预算拖垮。本 cap 的安全性来自 **超时即放弃、绝不重试**;完整性 gate 仍是唯一 pass/fail,漏传会响亮报错而不是 静默发半个镜像。脚本顶部的血泪注释已按此改写。 验证: - 真实 0.0.94 镜像干跑 → 16 资产两端全 200、已存在全部 skip、206s 通过。 - stub 双分支(PATH 上的真可执行文件,非 shell function —— timeout 对函数无效): 快传 → `github big.tar.gz (2.9MB) uploaded in 1s`; 慢传 → `WARN: gitcode ... exceeded the 2s cap after 2s — skipping`。
1 parent e243856 commit 60e20ec

1 file changed

Lines changed: 74 additions & 15 deletions

File tree

‎.github/tools/mirror_res.sh‎

Lines changed: 74 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,56 @@ read -r -a ASSETS <<< "${ASSETS:-$DEFAULT_ASSETS}"
4242

4343
info() { echo "[mirror] $*"; }
4444

45+
# Per-asset upload cap. Exceeding it WARNs and abandons that asset (no retry,
46+
# no delete) so one slow cross-border PUT can't eat the whole job budget — the
47+
# v0.0.94 release lost `publish-ecosystem` to a 30min job timeout with zero
48+
# per-asset visibility, and the two biggest linux tarballs had to be pushed by
49+
# hand afterwards.
50+
#
51+
# CAUTION (v0.0.90 postmortem, still binding): a cap that kills a
52+
# slow-but-PROGRESSING upload and then RETRIES it is strictly worse than no cap
53+
# — every restart resumes from byte zero and the mirror never converges. This
54+
# cap is safe only because a capped asset is SKIPPED, never re-uploaded. The
55+
# completeness gate at the bottom is still the pass/fail, so a skipped asset
56+
# fails the release loudly instead of silently shipping a half mirror.
57+
#
58+
# Raise it (or set it per-host) when the cross-border leg legitimately needs
59+
# longer than the cap for the big archives; the gate will tell you.
60+
: "${MIRROR_UPLOAD_TIMEOUT:=120}"
61+
4562
DL="$(mktemp -d)"; trap 'rm -rf "$DL"' EXIT
4663

64+
human_size() { # path → e.g. 31.9MB
65+
local b; b=$(stat -c %s "$1" 2>/dev/null || stat -f %z "$1" 2>/dev/null || echo 0)
66+
awk -v b="$b" 'BEGIN{ if (b>=1048576) printf "%.1fMB", b/1048576; else if (b>=1024) printf "%.1fKB", b/1024; else printf "%dB", b }'
67+
}
68+
69+
host_label() { [[ "$1" == gh ]] && echo github || echo gitcode; }
70+
71+
# Upload one asset under the cap, timing it. 0 = the command returned within
72+
# the cap (NOT proof it landed — gtc's exit code lies both ways, so the probe /
73+
# gate remains the only source of truth); 1 = the cap fired, asset abandoned.
74+
upload_asset() { # kind(gh|gtc) asset → 0 returned / 1 capped
75+
local kind="$1" a="$2" start elapsed rc=0 sz host
76+
sz=$(human_size "$DL/$a")
77+
host=$(host_label "$kind")
78+
start=$SECONDS
79+
if [[ "$kind" == gh ]]; then
80+
GH_TOKEN="${XLINGS_RES_TOKEN:-}" timeout "$MIRROR_UPLOAD_TIMEOUT" \
81+
gh release upload "$VER" "$DL/$a" -R "$GH_DST" --clobber >/dev/null 2>&1 || rc=$?
82+
else
83+
timeout "$MIRROR_UPLOAD_TIMEOUT" gtc release upload "$GTC_DST" "$DL/$a" --tag "$VER" \
84+
>/dev/null 2>&1 || rc=$?
85+
fi
86+
elapsed=$((SECONDS - start))
87+
if [[ $rc == 124 || $rc == 137 ]]; then
88+
info "WARN: $host $a ($sz) exceeded the ${MIRROR_UPLOAD_TIMEOUT}s cap after ${elapsed}s — skipping (not retried; the verify gate below decides the release)"
89+
return 1
90+
fi
91+
info "$host $a ($sz) uploaded in ${elapsed}s"
92+
return 0
93+
}
94+
4795
info "downloading $SRC_REPO v$VER assets ($PROJ)"
4896
for a in "${ASSETS[@]}"; do
4997
gh release download "v$VER" -R "$SRC_REPO" -D "$DL" -p "$a" 2>/dev/null || { echo "[mirror] FAIL: missing $a in $SRC_REPO v$VER" >&2; exit 1; }
@@ -93,36 +141,43 @@ verify_batch() { # base_url asset... → prints assets still not serving
93141
# Hard-won rules (0.0.86 / 0.0.89 / 0.0.90 postmortems):
94142
# - NEVER delete on a verify timeout — the eager 404→delete loop repeatedly
95143
# deleted GOOD uploads whose propagation was merely slow.
96-
# - NEVER kill a slow-but-progressing upload: the outer `timeout` MUST
97-
# exceed gtc's inner PUT timeout (600s). v0.0.90 wrapped uploads in
98-
# `timeout 300`, so every cross-border PUT >5min was SIGKILLed at 60%%
99-
# and restarted from byte zero — the 20min job ceiling fell to this.
144+
# - NEVER kill a slow-but-progressing upload AND RETRY IT. v0.0.90 wrapped
145+
# uploads in `timeout 300`, so every cross-border PUT >5min was SIGKILLed
146+
# at 60%% and restarted from byte zero — the 20min job ceiling fell to
147+
# this. MIRROR_UPLOAD_TIMEOUT keeps a cap but ABANDONS the asset instead of
148+
# retrying it, which is what makes the cap safe; the gate then fails the
149+
# release loudly rather than thrashing until the job is killed.
100150
# - gtc's exit code lies both ways (obs_callback flakiness); the download
101151
# probe is the only source of truth.
102152
mirror_host() { # kind(gh|gtc) base_url
103153
local kind="$1" base="$2" try a
104154
local pending failed
155+
local -A capped=()
156+
local host_start=$SECONDS
157+
local host; host=$(host_label "$kind")
105158
for try in 1 2 3; do
106159
pending=()
107160
for a in "${ASSETS[@]}"; do
161+
# A capped asset is never re-attempted (see MIRROR_UPLOAD_TIMEOUT).
162+
[[ -n "${capped[$a]:-}" ]] && continue
163+
# Step 1: already serving? then it's mirrored — never re-upload it.
108164
if probe "${base}/${a}"; then
109-
[[ $try == 1 ]] && info "$kind $a already mirrored, skipping"
165+
[[ $try == 1 ]] && info "$host $a already mirrored, skipping"
110166
continue
111167
fi
112-
pending+=("$a")
113-
if [[ "$kind" == gh ]]; then
114-
GH_TOKEN="${XLINGS_RES_TOKEN:-}" timeout 900 \
115-
gh release upload "$VER" "$DL/$a" -R "$GH_DST" --clobber || true
168+
if upload_asset "$kind" "$a"; then
169+
pending+=("$a")
116170
else
117-
timeout 900 gtc release upload "$GTC_DST" "$DL/$a" --tag "$VER" \
118-
>/dev/null 2>&1 || true
171+
capped[$a]=1
119172
fi
120173
done
121-
[[ ${#pending[@]} == 0 ]] && return 0
174+
[[ ${#pending[@]} == 0 ]] && break
122175
failed=$(verify_batch "$base" "${pending[@]}")
123-
[[ -z "$failed" ]] && return 0
124-
echo "[mirror] $kind not serving after patience (try $try): $failed — re-uploading (no delete)"
176+
[[ -z "$failed" ]] && break
177+
info "$host not serving after patience (try $try): $failed — re-uploading (no delete)"
125178
done
179+
((${#capped[@]})) && info "WARN: $host abandoned ${#capped[@]} asset(s) at the ${MIRROR_UPLOAD_TIMEOUT}s cap: ${!capped[*]}"
180+
info "$host mirror leg finished in $((SECONDS - host_start))s"
126181
return 0 # the completeness gate below is the real pass/fail
127182
}
128183

@@ -174,5 +229,9 @@ for host in "${hosts[@]}"; do
174229
[[ "$code" == 200 ]] || { rc=1; echo "[mirror] FAIL: missing/unverified: https://${host}/releases/download/${VER}/${a}" >&2; }
175230
done
176231
done
177-
[[ $rc == 0 ]] && info "all assets mirrored + verified on ${#hosts[@]} host(s)"
232+
if [[ $rc != 0 ]]; then
233+
echo "[mirror] hint: if the asset above was WARNed as capped at ${MIRROR_UPLOAD_TIMEOUT}s, either raise MIRROR_UPLOAD_TIMEOUT for this run or push it by hand:" >&2
234+
echo "[mirror] gh release download v$VER -R $SRC_REPO -p '<asset>' && gtc release upload $GTC_DST '<asset>' --tag $VER" >&2
235+
fi
236+
[[ $rc == 0 ]] && info "all assets mirrored + verified on ${#hosts[@]} host(s) in ${SECONDS}s"
178237
exit $rc

0 commit comments

Comments
 (0)