feat(runners): Add nightly recycle for macOS 26 finch runners#1102
feat(runners): Add nightly recycle for macOS 26 finch runners#1102ayush-panta wants to merge 1 commit into
Conversation
Add scheduled ASG actions to spin down and spin up macOS 26 finch runners nightly at 10:00/10:30 UTC (2:00/2:30 AM PST). This prevents state accumulation that causes runners to become unresponsive over time, which previously required manual termination to resolve. Signed-off-by: ayush-panta <ayushkp@amazon.com>
d1429f5 to
b81402f
Compare
| this.repo === 'finch'; | ||
|
|
||
| if (isMac26Finch) { | ||
| new autoscaling.CfnScheduledAction(this, 'NightlySpinDown', { |
There was a problem hiding this comment.
Would replacing the root-volume nightly instead of provisioning a new instance altogher be better? will that solve our host contamination issues?
https://docs.aws.amazon.com/ebs/latest/userguide/ebs-restoring-volume.html#replace-root
There was a problem hiding this comment.
Also note that there is a 24hour cycle for mac hosts to become available after termination, we should be sure that we will not hit our capacity quota if a termination fails
Swapnanil-Gupta
left a comment
There was a problem hiding this comment.
Not a fan of this. Should we try to figure out why this is happening first?
It is difficult to determine why it is happening because the instances lose connection, so I can't SSM in to look at logs. I'm looking at an alternative approach with root volume replacement. |
Description of changes: Add scheduled ASG actions to spin down and spin up macOS 26 finch self-hosted runners nightly at 10:00/10:30 UTC (2:00/2:30 AM PST). This prevents state accumulation that causes runners to become unresponsive over time, which previously required manual termination to resolve.
This PR only targets these specific runners as they are the only ones frequently becoming unresponsive.
Testing done: Ran
npx jest -c ./jest.config.unit.js:License Acceptance
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.