feat: carry a VPC pod to the edge only when served - #415
Merged
scotwells merged 1 commit intoAug 27, 2026
Conversation
Every VPC pod's EndpointSlice is federated to every edge cluster, so a cell publishes one tenant's pod addresses and SRv6 SIDs onto edges that serve other tenants and other locations. The staging lab cell carries 178 slices this way. Nothing filters them, because the two facts that decide whether an edge needs a pod, the NetworkService selecting it and the HTTPProxy naming that service, live in a project control plane a cell cannot read. EdgeReachability records the answer where both planes meet. The control plane resolves, per project namespace, the workload addresses behind a proxy, and writes them to the federation hub. The write-back reads the record from the hub it already publishes into and carries a slice only when the record names one of its addresses. The record answers per namespace rather than per service so a reader can tell "nothing is served here" from "the control plane has not answered". Silence keeps a pod published: a route withdrawn under a pod that is still serving black-holes live traffic, while a route left up for a pod nothing sends to costs a table entry. Key changes: - Add EdgeReachability, a hub-only record of a project's served addresses - Add EdgeReachabilityReconciler, resolving proxies, services and the interfaces they select in each project control plane - Filter the VPC EndpointSlice write-back on the record, collecting a copy that stops serving - Resync published slices every minute, since nothing in a cell watches the hub and a pod put behind a proxy has no local event to act on
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #411 in part.
Today every VPC pod's EndpointSlice is federated to every edge cluster. On the staging lab cell that is 178 slices spanning several tenants and several locations, which means one tenant's pod addresses and SRv6 SIDs are installed on edges that serve other tenants and never route to them. Nothing filters this, and the fan-out grows with every workload anyone runs.
This carries a pod to the edge only while something serves it. The control plane resolves, per project namespace, the workload addresses that are behind an HTTPProxy, and records them on the federation hub as an
EdgeReachability. The write-back in each cell reads that record from the hub it already publishes into, and carries a slice only when the record names one of the slice's addresses. A pod put behind a proxy becomes reachable; a pod taken out from behind one stops being carried.The filter is enforced at the write-back rather than at the edge, so an address that no edge needs is never published in the first place.
Failure direction
Withdrawing reachability for a pod that is still serving black-holes its traffic. Leaving a route up for a pod nothing sends to costs a table entry. The two are not symmetric, so the record is built to distinguish "the control plane says nothing is served in this namespace" from "the control plane has not answered".
That is why the record answers per namespace instead of one object per service. With per-service objects those two states are the same absence, and a producer that had never run would read as a withdrawal of everything. With one record per namespace, a missing record means keep publishing, an empty record means withdraw, and a hub that cannot be read returns an error and changes nothing. A producer that stops running leaves its last record in place rather than an empty one.
Lifecycle
Deleting a proxy, narrowing a selector, or releasing an interface all take an address back out of the record on the next pass, and the copy is collected. Nothing in a cell watches the hub, so a minute-paced resync republishes what the cell holds and is what acts on a change only the hub saw. The existing ten-minute sweep still collects copies whose source went while the cell could not reach the hub.
Note for deployment
EdgeReachabilitymust be installed on the hub before the control plane can write records, and it is added toconfig/crd/downstream, which infra applies to the hub and to every cell. Until a record exists for a namespace, that namespace keeps publishing exactly as it does today, so the rollout order is safe in either direction. The cell's hub credentials need read access to the new resource.Test plan
make test(envtest) covers the record the control plane writes and the filter the write-back applies, including the silence-is-not-a-withdrawal case and the resync that brings a pod back.