NGOT implementation#
Rancher is used in NGOT Bojana version. It allows the gitlab-ci from D&D to talk to NGOT k8s apis. Rancher acts as a proxy with rbac features, allowing gitlab ci to alter only a subsets of the kubernetes API.
Networking#
Gitlab ci talks to rancher via the internet. Rancher is deployed behind a whitelist letting D&D runners, vpn users, and whole line subnet to talk to him.
RBAC#
We need to ensure the rancher token used by Gitlab CI has limited rights over our kubernetes cluster. We need them to be able to CRUD probes, prometheusrules and configmaps over specific contexts. To achieve that goal we create:
- a unique project per cluster
- a project role template with correct permissions bound to the project
- a user for this cluster
- a role binding between the user and the role template
Now when an admin user adds namespaces to the project, the user automatically inherits permissions on the namespace.
Topology#
Cluster types#
We have 2 types of clusters in rancher:
- management clusters
- provisioning clusters
The management cluster is unique along the NGOT line. The proxy service zone is deployed on it. Precisely it is the deployment of the proxy service zone on it that give him this particular rancher cluster type.
Currently we have 2 proxy service zones:
- svc-proxy-stg (staging line)
- svc-proxy (prod line)
They are located on (zone.cluster_zone_name):
- kub-10008 # staging
- kub-8 # prod
Those clusters are rancher management clusters. Other clusters among the line are provisioning clusters in Rancher language.
# get rancher management clusters
sd get zones | jq '.[]|select(.product=="ngot")|select(.subtype=="proxy").cluster_zone_name'
--- infra zone name is: infra-stg
Deployment#
Proxy service zone#
To fully deploy the service zone:
cd "${ENVS_NG}/contexts/ngot"
# build its loadbalancer by talking to its cluster
trackbone apply -c fe_loadbalancer_v3 -c ingress_controller_proxy -c fe_whitelist -z <kub-8 or kub-10008>
# deploy the service zone
trackbone apply -z <svc-proxy-stg|svc-proxy> -t bootstrap=true
The service zones handles:
- provisioning a dedicated LB via zone parameters
- rancher helm deployment and initial admin password retrieve via rancher2-helm configuration
- configure basically rancher via rancher2-ngot configuration
Provisioning Cluster zones#
Cluster zones other than the rancher management cluster runs rancher-ngot-client envs-ng configuration. It is a very touchy configuration due to rancher2 random behaviours, regressions, bugs and other. It handles:
- rancher objects deployment
- agent deployment
- several secrets provisioning like gitops workflow tokens
- an exotic bootstraping mode
This configuration can add other users, roles and tokens vith rancher2_config terraform module and thus uses its schema.
Note the current configurations state allows only one proxy service zone per line. No improvement is planned as the whole rancher service will be unprovisioned.
To fully deploy rancher provisioning clusters:
cd "${ENVS_NG}/contexts/ngot"
# deploy a single cluster in rancher
trackbone apply -c rancher2-ngot-client -z <kub clusters other than kub-8 or kub-10008> -t bootstrap=true
# to deploy the full line
trackbone apply -c rancher2-ngot-client -t <line> -t bootstrap=true
Secrets#
Our configuration management stores all rancher secrets in one path:
INFRA_ZONE_NAME= infra-xxx # put here the infra zone name
vault list secret/zones/fe/${PROXY_SERVICE_ZONE_NAME}/rancher