Aller au contenu

NginxAlertreceiverTooMany500s#

This alert is triggered when NGINX ingress requests fail with http error code 5xx for more than 5 minutes.

The criticity of this alert depends on the percentage of the querys that fail due to 5xx errors:

  • warning: if more than 5% of the total number of requests fail with 5xx error code
  • critical: if more than 10% of the total number of requests fail with 5xx error code

NGINX Ingress Controller documentation here.

Identify if Alertreceiver incident#

Before starting to troubleshoot, please make sure Alertreceiver works correctly. You can check Alertreceiver status here.

Alert Receiver status

Warning

If you encounter any issues to connect to the link above, make sure the following proxy is configured in your browser: http://proxypac.si.francetelecom.fr:8080/.

If any issue on Alertreceiver side, NGOT team need to contact the team in charge of it and provide all the information that could help them to fix the issue.

Troubleshooting hints#

  • Check that two ingress pods are present and running (replica=2):

    kubectl get  pod -n nginx-alertreceiver

  • Check that the ingress is present and the ressource events: 

    kubectl get ingress -n nginx-alertreceiver
kubectl describe ingress <nginx-ingress-name> -n nginx-alertreceiver

  • Check the logs of the pods.

  • In Grafana, check the following dashboards (Datasource: Thanos-client, cluster: prdcasa):

    • NGINX Ingress controller (Alertreceiver)
    • Nginx Request Handling Performance (Alertreceiver)

    Tip

    In order to identify which client Alertmanager sends the failing requests, check the alerts_source label of the alert.

Solution#

If the problem is on Alertreceiver side, NGOT support team need to contact the team in charge if this server and provide all the information that could help them to fix the issue.

Client communication#

If NGINX ingress fails to forward alerts to AlertReceiver(CASA), impacted client needs to be informed. Detailed procedure regarding client contacts/communication here.

If NGINX ingress fails to forward alerts due to AlertReceiver(CASA) issue, it is necessary to contact the team in charge of this server via an OCEANE/SWAN ticket. Detailed procedure regarding ticket creation here.