Docker tag cleanup#274
Draft
ericmj wants to merge 3 commits into
Draft
Conversation
Track each tag's last-pulled time and use it, with build age, to retire tags Bob no longer needs to keep, while letting a build request pin an image permanently. - Parse Docker Hub's `tag_last_pulled` and thread it through reconcile, staging and the swap into a new `docker_tags.last_pulled` column. - Add `Bob.Job.DockerCleanup`, scheduled daily and shipping in `:dry_run` (logs per-repo counts, deletes nothing). Live mode deletes, rate-limited and in bounded batches, removing a row only once Docker Hub confirms: per-arch tags built over 30 days ago, and manifest tags not pulled in 180 days (falling back to build age when a tag has no recorded pull). - A pending or completed build request reserves its tags permanently and exempts them from cleanup. Requesting an already-built image records the reservation without rebuilding; `prune` now keeps completed reservations and only reclaims expired dead-ends. Reservation is matched by a tag-name hash anti-join against the small build_requests table, not per-row JSONB. - Document retention in the README and on the request page, and flag reserved tags on the Docker tags page.
Give users a way to see whether a tag is due for cleanup. - Add a "last pulled" column alongside "built". - Flag each tag's retention outlook with a badge: green "reserved" for tags pinned by a build request, or amber "removing"/"removing ~Nd" once a tag is within 14 days of (or past) its removal date. - The outlook comes from `Bob.DockerCleanup.removal_at/3`, the same rule the cleanup deletes by, so the page and the deleter never disagree: per-arch tags by build age, manifest tags by last-pulled (falling back to build age). Reserved tags are never flagged as removing.
- Report every Docker Hub request outcome to the rate limiter via a shared paced request, reclaiming the calibration probe's slot on failure so a failed delete in the long-lived cleanup job can't wedge the gate for all Docker Hub traffic; the pager and delete_tag now share one throttle protocol - Preserve last_pulled when a reconcile page omits it, so a transient null from Docker Hub can't expose a pulled tag to cleanup - Treat a manifest tag as stale only when neither pulled nor built since the cutoff, mirrored in removal_at, so a re-pushed tag isn't deleted and immediately rebuilt with a 404 window in between - Re-check reservations immediately before each delete: the rate-limited batch runs for hours and a request made mid-run must still pin its tag - Complete fully built requests before os_version rotation or TTL expiry so a reservation backed by a pending row can't lapse after the image exists - Satisfy requests from a complete manifest so an image whose per-arch staging tags were cleaned is reserved for free instead of rebuilt - Budget the cleanup batch limit across both repo groups instead of per query - Serialize reserve() on the target so concurrent submits of the same already-built image can't insert duplicate permanent rows - Assert the SQL reservation predicate matches request_target/1 so the two tag-format encodings can't drift silently
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.