ECS autoscaling with graceful shutdown issue #4807
Description
I have an ECS (EC2 launch type) setup running a PHP worker.
The ECS service scales based on an SQS queue, with desired = 1, min = 1, max = 2.
The Auto Scaling Group (ASG) has min = 1, desired = 1, max = 3.
The ECS task performs a graceful shutdown that can take up to 1 hour (the EC2 instance itself stays running).
The issue I’m seeing is:
When a scale-out event happens during that graceful shutdown, ECS keeps trying to place the new task on the same EC2 instance where the old task is shutting down — even though that instance has no remaining capacity, and there are available unprovisioned instances in the ASG.
I already use the distinctInstance placement constraint, but the scheduler still attempts placement on the busy instance.
Has anyone seen this before or found a way to prevent ECS from scheduling new tasks on instances that are hosting tasks in graceful shutdown (without draining the instance itself)?
Any guidance or best practices would be really appreciated