Aller au contenu

ConcoursePipelineErrorPostgresBackup#

Concourse Pipeline has failed when backuping postgresql database to S3 object storage.

If a job has failed, the error could be:

  • error to connect on kubernetes cluster
  • error to get the leader pod of the postgresql cluster
  • error to connect on the leader pod of the postgresql cluster
  • error to create the database SQL dump from the leader pod
  • error to upload the file to the S3 storage

You can check the job details in order to see the logs of the script.

Troubleshooting hints#

  • Trigger a new build for failed job in Concourse CI. If it keeps failing:
  • Identify zone / team / pipeline / job (in the alert)
  • Pipeline URL: https://ci-infra.$ZONE.caascad.com/teams/$TEAM/pipelines/$PIPELINE

  • Job URL: https://ci-infra.$ZONE.caascad.com/teams/$TEAM/pipelines/$PIPELINE/jobs/$JOB

  • If the job URL of the alert returns a 404 error, see Decommissionned jobs.

Known cases#

Decommissionned jobs#

When you click on the job link (in the description of the error) you get a 404 error.

Important: first investigate why you get a 404 error.

When the 404 error is normal, for example when a zone is decommissionned, you can fix by redeploying a fake job, run it successfully, then remove the fake job.

fly switch -z infra-stg -n cron
fly gp -p postgres-backup > /tmp/pipeline.yaml
cp /tmp/pipeline.yaml /tmp/pipeline.orig.yaml

Edit the file /tmp/pipeline.yaml as explained below:

  1. Choose a reference model to copy/paste.

  2. Copy the jobs section and replace the current name with the one that does not exist any more. Example for multiple jobs:

    - jobs:
  - postgres-status-keycloak-ocb-xxx
  - postgres-status-concourse-infra-ocb-xxx
  - postgres-status-concourse-ocb-xxx
  - postgres-status-gitea-ocb-xxx
  - postgres-status-quay-ocb-xxx
  name: ocb-xxx

  3. Copy the jobs. For each job, replace the current name with the one that reports errors. Also replace the beginning of the plan like this:

    - name: postgres-status-quay-ocb-xxx
  plan:
  - config:
      image_resource:
        name: ""
        source:
          username: ((docker-registry-internal.username))
          password: ((docker-registry-internal.password))
          repository: ((docker-registry-internal.url))/alpine
          tag: 3.13
        type: registry-image
      platform: linux
      run:
        path: echo
    task: postgres-status-quay
  serial: true
  serial_groups:
  - group-quay

    You must keep

    • the first line name
    • the 4 last lines, including task and the group name (group-quay above). The other lines need to be replaced like above.

Deploy the new pipeline (ensure with the diff that there are only adds):

fly sp -p postgres-backup -c /tmp/pipeline.yaml

Force run the jobs (they are listed in the description of the alert).

Ensure that all the jobs are now "green".

Redeploy the original pipeline (ensure with the diff that there are only removals for the removed zone name):

fly sp -p postgres-backup -c /tmp/pipeline.orig.yaml

The alert should disappear soon.

Pipeline URL (samples)#

Job URL (samples)#