Aller au contenu

CCE v1.21? Network Problems Work Around#

Introduction#

After an upgrade from CCE v1.19 to v1.21 we started noticing network network timeouts among different pods. It turns out that the old (currently non-existing) nodes were still present in the kubernetes routing configuration. In order to mitigrate the issue we can apply the procedure described below or restart the canal service on the master nodes (operation which has to be performed by the FE support team).

Procedure#

Canalctl#

For this step we need to get the kubeconfig file which can be retrieved either from vault (see below) or from the FE WebUI. The kubeconfig file should be copied to one of the cluster nodes in /tmp/kubeconfig.

In order to retrieve the kube config file from vault infra-stg or infra-prd: 

vault read secret/zones/fe/<zone_name>/kubeconfig

From within one of the cluster nodes accessible via the bastion server, we have to create the canalctl command which is an alias for kubectl connecting to port 9443. We need two pieces of information: * the kubeconfig file from above * the IP of the k8s service endpoint. This IP can be found using netstat -antpl|grep canal-agent. The IP we are looking for is the Foreign address of the ESTABLISHED connection (XXX.XXX.XXX.XXX:9443) 

alias canalctl="kubectl -s https://XXX.XXX.XXX.XXX:9443 --kubeconfig /tmp/kubeconfig"

Work around procedure (non-destructive)#

  1. Run the canalctl get nodes command to check the data of the old master node.
  2. Run the canalctl get endpoint command to check the endpoint data of the old master node
  3. Run the canalctl get distnodesubnet command to check the distnodesubnet data corresponding to the endpoint of the old master node.
  4. Run the canalctl delete XXX command to delete residual data about nodes, endpoints, and distnodesubnets. Compare the cluster creation time with the creation time of resources such as nodes to determine which resources are residual and which were newly created during the upgrade.
  5. Run the systemctl restart canal command to restart the Canal agent on each master node and worker node.

In case too much has been deleted: If you delete normal records by mistake, run the systemctl restart canal command on master nodes and worker nodes to restore the records.

FE support has to be contacted for all operations on the master nodes.

Alternate work around (destructive)#

We can always hibernate and wake up the cluster. This will impact all applications. A 1h downtime windows is required.