ClusterAutoscalerMaxNodesReached

This alert is being raised when the cluster autoscaler has reached its maximum allowed number of nodes. This issue is rarely a show-stopping incident but it should be addressed as soon as possible because there is an impact on the scheduling of new pods.

In the cluster autoscaler, the maximum number of nodes can be set at a node group (or equivalent) level and at the cluster autoscaler level.

When setting the max nodes at the node group level, we are limiting the number of nodes within the group to a certain maximum value.

The max nodes setting at the autoscaler level, limits the nodes to a certain maximum value across all node groups and across the cluster.

In FE, the default value is 10 and this means that even if the node group max value is set to 20, the autoscaler will stop adding nodes once it reaches its maximum node number of 10.

Troubleshooting Hints

Before doing anything we need to understant why the cluster has reached this state. The first thing to check for should be the pending pods waiting to be scheduled. Once we decide that there is a real need to add additional nodes, we can either modify the trackbone configuation accordingly or, in the case of client clusters, approach the client with the problem and the proposed solution.

Do not automatically increase the max nodes value for the autoscaler on client clusters!

Resolution

To increase the max allowed number of autoscaler nodes, we need to modify the trackbone configuration by changing/adding the value maxNodesTotal. Below is an example for a client zone:

addons: {
                "metrics-server": #CCEMetricsServerAddon
                autoscaler:       #CCEAutoscalerAddon & {
                        "values": {
                                "custom": {
                                        maxNodesTotal:                 100
                                        scaleDownDelayAfterDelete:     5
                                        scaleDownDelayAfterAdd:        5
                                        scaleDownUtilizationThreshold: 0.4
                                        scaleDownDelayAfterFailure:    3
                                        maxEmptyBulkDeleteFlag:        10
                                        unremovableNodeRecheckTimeout: 5
                                        scaleDownUnneededTime:         5
                                }
                        }
                }
        }