How to Rebalance k8s Clusters#
Our toolbox contains two utilities which allow us to rebalance the workload within a k8s cluster. These tools are kube-rebalancer and k9s.
kube-rebalancer#
Installation#
The toolbox should be updated to its latest version: toolbox update
$ toolbox install kube-rebalancer
Using kube-rebalancer for rebalancing the workload#
We will be using ocb-test06 as an example. The state of the cluster before running the tool is:
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
10.0.108.46 5205m 65% 26004Mi 91%
10.0.196.214 192m 9% 5886Mi 43%
10.0.197.141 2773m 35% 24499Mi 85%
10.0.252.205 6492m 82% 27908Mi 97%
10.0.90.224 5242m 66% 25713Mi 90%
kube-rebalancer work autonomously and decides which pods to move to which nodes:
$rebalancer-move-pods
=====================need_memory=====================
name: 10.0.252.205
mem_percent: 97.71
cpu_percent 86.55
pods_to_move ['dashboard-api-proxy-788964f6d-k5r56', 'dashboard-metrics-api-58766c899c-wwbn8', 'redis-6d7d95f894-sk7cf']
===================================================
name: 10.0.108.46
mem_percent: 91.34
cpu_percent 78.02
pods_to_move ['rancher-6cf979799-zdlnx', 'cert-manager-cainjector-594b44dbf4-6n6qt', 'concourse-web-67c4f8cd9-4gf8v']
===================================================
name: 10.0.90.224
mem_percent: 90.95
cpu_percent 62.97
pods_to_move ['eventrouter-67fb645b44-pfwnf', 'rancher-6cf979799-dcwgt', 'rancher-webhook-9bfdc4cf7-7pddw']
===================================================
=====================need_cpu=====================
Apply ? (y/N)y
Moving pod dashboard-api-proxy-788964f6d-k5r56 to node 10.0.197.141...
Moving pod dashboard-metrics-api-58766c899c-wwbn8 to node 10.0.197.141...
Moving pod redis-6d7d95f894-sk7cf to node 10.0.197.141...
Moving pod ingester-6bfc459968-wcht8 to node 10.0.197.141...
Moving pod ingress-nginx-private-nginx-ingress-controller-7dbbf46b56-kw26n to node 10.0.197.141...
Moving pod ingester-6bfc459968-m2xmw to node 10.0.197.141...
After kube-rebalancer finished its work, the cluster is left in the following state:
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
10.0.108.46 4996m 63% 25361Mi 88%
10.0.196.214 204m 10% 5890Mi 43%
10.0.197.141 2796m 35% 25528Mi 89%
10.0.252.205 4850m 61% 17305Mi 60%
10.0.90.224 4937m 62% 25367Mi 88%
The cluster is well balanced. 10.0.196.214 is the node hosting quay that is why pods aren't scheduled on this node (other than quay obviously).
When things fail#
In case of an error, check that the nodes are not cordoned and manually uncordon them before re-running the tool.
k9s#
Installation#
The toolbox should be updated to its latest version: toolbox update
$ toolbox install k9s
Using k9s for rebalancing the workload#
This process is rather manual. First we start up k9s in node mode by writing :node once k9s is running. The screen should look somehting like the image below:
We can observe that the node with the highest memory comsumption is: 10.0.90.224
In order to rebalance the cluster, 10.0.90.224 will be cordoned while all the other nodes will be left uncordoned. To cordon a node, we position our cursor on the target node and hit the c key. In order to uncordon a node, we position the cursor on the target node and hit the u key. Once ready, the output shoudl look like:
The top consumers can be seen by viewing the node's details (press enter while positioned on the node) and sorting either my memory (shift-m) or cpu (shift-c):
Using our judgment we will start moving pods one at a time until the overall resource consumption decreases below the threshold desired. Pods can be deleted in k9s by hidding ctrl-d with the cursor positioned on the target pod.
This process is very manual and requires constant checking of resource consumption on all nodes but in the end the cluster should be fairly balanced.
