Chore: Explore autoscaler behavior when external limits are reached

[bernardhalas]

Description

The autoscaler behaves well on a sunny weather.
Its behavior is not thoroughly predictable on situations like:

• cloud provider outage
• quota hit
• multiple scale-out requests across cloud providers out of which some some get stuck on external reasons
• combination of subsequent scale-out and down-scale requests

Exit criteria

• Create a list of scenarios to test (take an example from the description above), describe them in this user-story
• Execute the test scenarios and observe in which of them the autoscaler behaves in a predictable manner
• Open follow-up issues to fix any observed misbehavior and/or align the documentation

[Despire]

What has been tested:

A single kubernetes cluster with multiple autoscaled nodepools (3).

Two of the nodepools had invalid OS images to "simulated" a failing scale-up from claudie.

The results are:

Cluster-autoscaler has a 15 minutes timeout for a scale-up operation, if the scale-up fails during that period the scale-up is removed and the NodePool is marked as unhealthy for a certain amount of time (5-10mins)

On the next iterations of the cluster-autoscaler the unhealthy nodepool will be ignored (until the unhealthy backoff expires) and other nodepools will be selected that match the requirements for the scale-up.

Which I would argue is generally how we would expect it to work for multiple autoscaled nodepools.

There seems to be an issue with the scale-up timing out which is described in #2050