Operation in Rancher NGOT#
Operational instructions to manage rancher in NGOT. Rancher is deployed through the proxy service zone. Currently we have
Standard operations#
Get service zones#
# get rancher management clusters
sd get zones | jq '.[]|select(.product=="ngot")|select(.subtype=="proxy").name'
--- infra zone name is: infra-stg
"svc-proxy-stg" # staging
"svc-proxy" # prod
Get management cluster zones#
# get rancher management clusters
sd get zones | jq '.[]|select(.product=="ngot")|select(.subtype=="proxy").cluster_zone_name'
--- infra zone name is: infra-stg
"kub-10008" # staging
"kub-8" # prod
Secrets location#
All secrets for ngot rancher are store in one path per line:
export VAULT_ADDR=<vault infra url>
vault list secret/zones/fe/${PROXY_SERVICE_ZONE_NAME}/rancher
Connect to Rancher with admin user#
Get admin password :
export VAULT_ADDR=<vault infra url>
vault read secret/zones/fe/${PROXY_SERVICE_ZONE_NAME}/rancher/api-token-svc-proxy
Connect to console prd/stg with this admin user.
Get a gitlab monitoring user token for gitops workflow#
export VAULT_ADDR=<vault infra url>
vault read secret/zones/fe/${PROXY_SERVICE_ZONE_NAME}/rancher/token-gitlab-monitoring-${CLUSTER_ZONE_NAME}
Retrieve gitlab monitoring user password#
You will need this password to login on rancher Web UI.
cd "${ENVS_NG}/contexts/ngot"
trackbone outputs -c rancher2-ngot-client -z "${CLUSTER_ZONE_NAME}"
...
"local_users": {
"value": {
"gitlab-monitoring-kub-10009": {
"password": "XXXXXXXXXXXXXXXXX",
"username": "u-23e1f"
}
},
...
Increase log level#
Inside a rancher server or agent pod:
PODS=($(kubectl -n cattle-system get pods -o name))
for p in ${PODS[@]};do
kubectl -n cattle-system exec $p -- bash -c "loglevel --set debug"
done
kubectl -n cattle-system logs <some rancher or agent pod>
for p in ${PODS[@]};do
kubectl -n cattle-system exec $p -- bash -c "loglevel --set info"
done
ReBootstrap#
Rancher edits the manifest file of a given cluster when its agent connects for the first time. The manifest url becomes unusable. It explains why we are deploying the agent with a kubectl command in pre apply scripts, we want this done only once. Moreover, it the scripts detects the deployment already exists, it won't try to deploy the agent. So if you need to fully rebootstrap a cluster you need to first delete agent deployment
cd "${ENVS_NG}/contexts/ngot"
ZONE_NAME=kub-xxxxx
trackbone destroy-c rancher2-ngot-client -z "{ZONE_NAME}" -t bootstrap=true
trackbone apply -c rancher2-ngot-client -z "{ZONE_NAME}" -t bootstrap=true
Deploy the whole line#
trackbone apply -t ${LINE} -c rancher-helm -c rancher2-ngot -t bootstrap=true
trackbone applu -t ${LINE} -c rancher2-ngot-client -t bootstrap=true
Known bugs#
Several bugs hits the current deployment:
Agent deployment done only at bootstrap#
When a cluster object is created, rancher generates a manifest file on a specific url. When an agent is connected with the credentials provided by the manifest file, the manifest file changes and its new version is not functional. It is maybe a security concern in rancher, but it prevents us to control its effective deployment with terraform workflow. If you try to re apply the manifest, the agent will stop talking with rancher.
Incorrect permissions#
If such errors appears while using kubectl with a rancher local user:
kubectl get probes -A
E0804 10:53:06.787643 29259 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0804 10:53:06.807263 29259 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0804 10:53:06.826717 29259 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
trackbone apply -c rancher2-ngot-client -z ${CLUSTER_ZONE} -t bootstrap=true
Since Rancher2.6, users needs to login once to be fully functional with projects objects. It could be simulated by patching the userattributes.management.cattle.io objects of the user, but for some reason, that object may not exist and appear later in the api. It is thus not possible to correctly handle that issue automatically. We have fixed this issue by adding a selenium based script issueing a login operation during bootstrap only.
No permissions on a specific namespace#
If you encounter this error message:
kubectl get probes -n test
E0804 14:30:40.194801 44391 memcache.go:265] couldn't get current server API group list: {"Code":{"Code":"Forbidden","Status":403},"Message":"clusters.management.cattle.io \"c-5rnkn\" is forbidden: User \"u-66dxx\" cannot get resource \"clusters\" in API group \"management.cattle.io\" at the cluster scope","Cause":null,"FieldName":""}