fix(pvc): complete restore populate handoff#4
Conversation
…ke pv mode The restore populate handoff patches the host PV claimRef even when PV sync to host is disabled (fake PV mode: sync.toHost.persistentVolumeClaims.enabled + sync.toHost.persistentVolumes.enabled=false). In that mode the chart granted no persistentvolumes patch/watch verbs, so the very first host PV patch of the handoff fails with Forbidden and the restore stalls forever. Two layers were missing: - clusterrole.yaml: grant persistentvolumes get/list/watch/patch when PVC sync is on and PV sync is off. - _rbac.tpl: with rbac.clusterRole.enabled=auto, PVC-sync-only setups did not trigger ClusterRole creation at all, so even a correct rule set would not be rendered. Verified with helm template for both modes: fake PV mode now renders get/list/watch/patch; PV sync mode is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Sync() deletion guard only covers the window where the virtual populate helper PVC still exists with a deletionTimestamp. Once the guest removes the pvc-protection finalizer the virtual helper object disappears entirely and the next reconcile takes the SyncToVirtual path, which deleted the host helper PVC unconditionally. That releases the populated host PV (and lets a Delete reclaim policy destroy it) before the claimRef handoff to the target PVC has happened, losing the restored data. Add the same handoff-pending guard to SyncToVirtual, identifying the helper through the host object's virtual name mapping. Also treat a missing host target PVC as handoff-pending in the shared check: releasing the helper while the target host PVC does not exist yet reclaims the populated PV just the same. Regression tests verified to fail against the previous head: the orphaned helper was deleted with no requeue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When the guest side had materialized (guest PVC volumeName + bound guest PV) but the populated host PV did not exist, the syncer still synthesized a Bound status on the virtual PVC. The workload then reported a completed restore over a volume that does not exist on the host - the same false-success failure the PG frozen scene exposed: host PV reclaimed, guest Bound, restore reported complete, disk empty. ensureExternalPopulatorHostMaterialization now reports whether the host side has converged (host PV exists and its claimRef references the host target PVC, either already or after a successful handoff patch), and Sync() only synthesizes the virtual Bound status after convergence. Otherwise the virtual PVC stays Pending so the real state remains visible, and the sync requeues. Also remove the materialization request ConfigMap: nothing consumes it anywhere, so it only leaked pending ConfigMaps into the host namespace while masking the missing-PV state. Updated the three tests that encoded the old expectation (host PV missing => virtual Bound + request ConfigMap); verified they fail against the previous head with the synthesized Bound status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In fake PV mode, pvNeeded() only kept a fake PV if some PVC referenced it via spec.volumeName. When the populate helper PVC is deleted before the restore target PVC's volumeName is written (populator crash recovery, premature helper deletion), no PVC references the PV by volumeName in that window and the fake PV gets garbage collected, even though its claimRef already points at the live target PVC. Runtime phase-B validation showed this delays handoff convergence by roughly 30-60s until the fake PV is recreated; the populated host PV itself survives thanks to the helper preserve guards, so this is a robustness gap, not data loss. pvNeeded now also keeps a fake PV whose claimRef references a live PVC with a matching (or unset) UID. PVCs that are deleting, missing, or have a mismatched UID do not keep the PV, so released fake PVs are still cleaned. Regression test verified to fail against the previous head (fake PV was deleted in the window). This ports the same fix previously reviewed as 59bdd19 on the old populator-bridge base. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Phase A / Phase C runtime evidence for exact head Identity: local build Phase A (normal KB order, script Phase C (syncer killed inside the W5 race window, script Cleanup: guest namespaces deleted; zero host PVC/PV residue beyond the vcluster data volume itself. Boundary: kind/arm64, no-KB simulation, N=1 per phase. This is not a PG/IDC real-environment result and does not authorize release. |
Problem
DataProtection restore in a vCluster needs a four-step handoff between the guest target PVC, the temporary
kb-populate-<target PVC UID>helper, the translated host PVCs, and the populated host PV. The previous syncer could reclaim the populated PV, synthesize guestBoundtoo early, or switch the host PV claimRef to the target while the populator helper was still active. The last race producedClaimMisboundin a fresh PostgreSQL restore scene.Change
volumeName.Boundonly after the populated host PV exists and points at the exact host target PVC.volumeName;Populating=Truewith terminal reasonSucceedorProvisioned;NotFound(a deleting helper still counts as present).Required KubeBlocks contract
This head intentionally requires KubeBlocks DataProtection to publish a terminal
Populatingcondition. KubeBlocks main/1.2 does this after helper cleanup (True/Succeed, withProvisionedalso accepted). A legacy/custom populator that never writes this terminal condition will remain pending at the handoff gate instead of receiving an unsafe claimRef switch. Do not roll this image into such an environment without first adding the equivalent terminal condition contract.The helper name contract remains deliberately narrow:
kb-populate-<target PVC UID>. This is not generic external-populator support.Validation
Exact head:
4b4091829b8f992b37e96356535c375441246ddf(baseb98b1ae5a603fa504daf1e745d65784c5a0be0ea).All pass. Seven of the eight new focused cases fail on
b98b1ae5a; the one case that stays green is the negative guard proving unrelated helpers do not over-block.Independent XP review walked design-contract classes 1-8 and found no remaining source blocker. The reviewer also verified the KubeBlocks ordering is non-cyclic: rebind guest PV -> delete helper -> publish terminal Populating condition; none of those steps waits for the host handoff.
Independent kind Phase-B race validation at the exact head passed after the simulator was corrected to publish the real KubeBlocks terminal condition:
race-data-pr4read back successfully;Evidence SHA:
ef09ad51e5bc5e72de0c66fdd1cb0ddfd3d3d8110b12393ddd9dae9ff742d2fc.The first simulator run without a terminal Populating condition failed at the new gate as designed (evidence SHA
f8cf56011fc35082e10b6759b07d0e9be715478ea9d5518078043c64db20bf75); this is harness-contract evidence, not a product regression.Evidence boundary
The focused source and kind gates are green. Fresh PostgreSQL R1k runtime validation of this exact head is still N=0 and remains a separate gate. This PR is not broad restore compatibility or release-readiness evidence.