Aller au contenu

Quay - Opérations#

Problèmes connus#

Transaction: cannot execute INSERT/UPDATE in a read-only#

Description : Suite à une bascule de l'instance de base de données PostgreSQL, il peut arriver que la connexion PostgreSQL de Quay soit resté sur l'ancien pod Master (devenu un replica). Celà peut générer :

  • des erreurs HTTP 500 lors des Push / Pull d'images 
    2023-09-11 12:27:12 gunicorn-registry stdout | 2023-09-11 10:27:12,190 [289] [INFO] [gunicorn.access]  - - [11/Sep/2023:10:27:12 +0000] "GET /v2/auth?account=atos_gsoprod%2Bgsops&scope=repository%3Aatos_gsoprod%2Forchestrator%3Apull&service=docker-registry.ocb-esagso.caascad.com HTTP/1.1" 500 0 "-" "-"
  • des errreurs tracebacks python 
    gunicorn-registry stdout | 2023-09-11 10:16:07,423 [289] [ERROR] [gunicorn.error] Error handling request /v2/atos_gsoprod/xbip-ingester-svl/manifests/1.15.3
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/app/lib/python3.6/site-packages/peewee.py", line 3057, in execute_sql
gunicorn-registry stdout |     cursor.execute(sql, params or ())
...
gunicorn-registry stdout | peewee.InternalError: cannot execute INSERT in a read-only transaction

Solution :

Supprimer le pod applicatif 

kswitch <ocb-example>
kubectl delete pods -n quay -l app=quay

Fix duplicate accounts#

Problem:

After changes with Keycloak, Quay may want to create new usernames even if the user has already one. Quay tries to create users with a 0 at the end. Example : johndoe0

Solution:

  • Edit the configmap to enable the user admin quay 
    kubectl edit configmap -n quay quay-config
# - AUTHENTICATION_TYPE: AppToken
# + AUTHENTICATION_TYPE: Database
# - FEATURE_DIRECT_LOGIN: false
# + FEATURE_DIRECT_LOGIN: true
  • Kill the pod quay and wait for it to restart (it takes a while). Some environments have 2 replicas, we have to kill both, but we can do it one after the other.
  • Once Quay has restarted, you can login with the user quay. You can find its password in the vaults infra-stg/prd 
    vault kv get -field password secret/concourse-infra/cicd/quay-admin
  • Once logged in, go to https://docker-registry.$ZONE_NAME.caascad.com/superuser/ -> Manage Users. And delete all usernames which end by a 0.
  • Clone the repo caascad/cicd and use the script quay-sync-with-keycloak.sh : 
    git clone git@git.corp.caascad.com:caascad/cicd.git
cd cicd/quay-sync-with-keycloak
./quay-sync-with-keycloak.sh $ZONE_NAME $OIDC_REALM
# EXEMPLE : ./quay-sync-with-keycloak.sh infra-prd infra-prd
  • OIDC login should works fine now.
  • Don't forget to restore the configmap and kill the pods again to apply it. 
    kubectl edit configmap -n quay quay-config
# + AUTHENTICATION_TYPE: AppToken
# - AUTHENTICATION_TYPE: Database
# + FEATURE_DIRECT_LOGIN: false
# - FEATURE_DIRECT_LOGIN: true

Fix Unable to scan images#

Problem:

After pushing an image into Quay, the security scan result says Unable to scan

Solution:

  • Make sure Clair is working correctly (Do not trust the "running" status of the pod Clair. Check its "health" URL instead) 
    ❯ curl -i https://clair.ZONE_NAME.caascad.com/health
HTTP/2 200
server: nginx/1.17.8
date: Wed, 17 Nov 2021 09:12:01 GMT
content-length: 0
strict-transport-security: max-age=15724800; includeSubDomains

Or look for this line in the logs: 

2021-11-17 09:02:28,499 INFO gave up: clair entered FATAL state, too many start retries too quickly

  • If the curl command does not return a 200 Code status, or if you find this line in the logs, you need to restart Clair. Just kill the pod and wait for it to come back.

  • Once you are sure Clair is up and running, you can force a new scan for all images: 

    ❯ kswitch "${ZONE_NAME}"

# Find the postgres leader
❯ kubectl exec -it -n quay quay-postgresql-cluster-0 -- bash -c "patronictl list"
Defaulted container "postgres" out of: postgres, prometheus-exporter
+ Cluster: quay-postgresql-cluster (7023717753131876421) -+---------+----+-----------+
|           Member          |     Host     |     Role     |  State  | TL | Lag in MB |
+---------------------------+--------------+--------------+---------+----+-----------+
| quay-postgresql-cluster-0 | 172.16.0.188 |    Leader    | running |  1 |           |
| quay-postgresql-cluster-1 | 172.16.0.209 | Sync Standby | running |  1 |         0 |
| quay-postgresql-cluster-2 | 172.16.0.198 |              | running |  1 |         0 |
+---------------------------+--------------+--------------+---------+----+-----------+

# Force a rescan
❯ kubectl exec -n quay quay-postgresql-cluster-0 -it -- psql -U quay -d quay -c 'UPDATE image SET security_indexed = false, security_indexed_engine = -1'