CI: Use main Ubuntu archive instead of azure.archive.ubuntu.com - #4336
Merged
Conversation
The azure.archive.ubuntu.com apt mirror on GitHub-hosted runners regularly degrades to trickle speeds for extended periods, stalling package downloads until the job hits its timeout. This is a recurring upstream problem, see actions/runner-images issues #7048 and #12949. apt retries do not help because the connection stays alive, only slow. Switch affected jobs to archive.ubuntu.com, which is consistently fast from Azure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI jobs that install build dependencies on GitHub-hosted runners occasionally stall until the 45 minute job timeout. The cause is the default apt mirror on the runners, azure.archive.ubuntu.com, which degrades to trickle speeds (tens of kB/s) for extended periods. Example: rip-rtai on #4335 spent 25 minutes downloading fonts-noto-cjk (61 MB) and was cancelled at the timeout.
This is a recurring upstream problem with the Azure mirror, reported many times in actions/runner-images (issues #7048, #12949, #675, #6913). GitHub's official response there is to use an alternative mirror.
Note that apt retries (Acquire::Retries) do not help here: the connection stays alive, it is only slow, so apt never sees an error to retry.
Fix
Switch the apt sources on the runner from azure.archive.ubuntu.com to archive.ubuntu.com before installing packages, in a small script called from install-deps.sh and from the cppcheck job (which calls apt directly). This covers all jobs that install dependencies on the GitHub-hosted Ubuntu runners. The container based package jobs are unaffected, they use Debian mirrors.
The sed handles both the traditional /etc/apt/sources.list and the deb822 format .sources files used on 24.04 runners.
Possible follow up
If we still see stalls after this (i.e. archive.ubuntu.com itself degrading from Azure), the next step would be a wrapper that runs apt under a timeout and, on expiry, rewrites the sources to a different mirror and retries. I did not include that here because it adds complexity for a failure mode we have not seen yet; the plain mirror swap removes the known offender. If stalls reappear I will implement the wrapper as a follow up.