KubeAPILatencyHigh

This alert triggers with severity warning if the latency is greater than 1s, and with severity critical if it is greater than 4s.

Troobleshooting hints

Web hooks

This alert may be related to admission controller webhooks used in the cluster.

If the alert triggers for GET endpoints among others, webhooks might not be responsible for triggering it, because they are not called for these.

Otherwise, you need to check the logs of the pods hosting webhooks for problems that could cause latencies. To find them, you need to start from the mutatingwebhookconfigurations and validatingwebhookconfigurations kubernetes objects which are linked to services referring to the pods that we are searching for.

• Firstly, you need to get the list of webhook objets:

  kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations
  

• Secondly, for each webhook object returned, you need to find the name of the corresponding services:

  webhookObject=<type_of_webhook/name>
  # Get services corresponding to $webhookObject
  go_tpl='
  {{- range .webhooks -}}
    {{- with .clientConfig.service -}}
      SERVICE: namespace={{.namespace}}, name={{.name}}{{"\n"}}
    {{- end -}}
  {{- end -}}'
  kubectl get "$webhookObject" -o go-template="$go_tpl"
  

• Thirdly, for each service, you need to get the name of the pods matching the service's selector:

  serviceNamespace=<service_namespace>
  serviceName=<service_name>
  # Get service selectors formatted as k1=v1,k2=v2,... list
  go_tpl='{{range $k,$v := .spec.selector}},{{$k}}={{$v}}{{end}}'
  selectors=$(kubectl get -n $serviceNamespace svc $serviceName -o go-template="${go_tpl}" | cut -c2-)
  # Get pods filtered by selectors
  kubectl -n $serviceNamespace get pod -l "${selectors}"
  

• Finally, you can check the logs for the pods:

  podName=<pod_name>
  kubectl -n $serviceNamespace logs $podName
  

Check compute resources

Investigate the issue by searching for shortages of compute resources for clusters/nodes/webhook pods around the time of the incident.

You can copy/paste the expression of the alert KubeAPILatencyHigh into the Explore tab of Grafana to view the time interval of the incident.

(cluster_quantile:apiserver_request_latencies:histogram_quantile{job="apiserver",quantile="0.99",subresource!="log",verb!~"^(?:LIST|WATCH|WATCHLIST|PROXY|CONNECT)$"} > 1

Other causes

The API Server relies on etcd to read or write to the cluster's config. As a consequence, latencies on etcd will cause latencies on the API Server.

Unfortunately, etcd logs are not available to the Caascad team.