EC2 Simplified Automatic Recovery conflicts with Karpenter's termination behavior

[ellistarn]

Description

EC2 Simplified Automatic Recovery is enabled by default for many instance types. When triggered, it can hold a node for much longer than it would take for Karpenter to simply replace it.

Karpenter's interruption controller handles spot interruptions and scheduled maintenance via EventBridge, but doesn't watch the system status checks that trigger EC2 Auto Recovery.

As a potential design option, we could:

1. Always disable auto recovery in maintenanceOptions on Karpenter-managed launch templates
2. Watch for status check failures via DescribeInstanceStatus and trigger node replacement

---

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[youwalther65]

Looking at State change events for Amazon EC2 instances it seems there are no events for status check failures, so integration with EventBridge is currently not possible (even though I would prefer this way).
So using DescribeInstanceStatus is completely valid even though we now have to handle potential API throttling

[cnmcavoy]

Are there any side-effects to using the maintenanceOptions to disable the auto-recovery? It seems like that approach would avoid polling the AWS API.

[youwalther65]

@cnmcavoy Turning it off means, we just ignore potential errors and don't get visibility into real issues, which requires the DescribeInstanceStatus calls.

[bwagner5]

Ideally, I think we want to know about instance status checks to take care of unhealthy nodes more quickly than waiting on other things to start failing. I'm digging in to see what we could do with polling since EventBridge is not supported.