Deleting a Node from a Kubernetes cluster

This guide outlines the steps to delete worker-nodes from a cluster, specifically how to proceed with rook-ceph and other KubeOps Container Manager applications
Required license: CM Enterprise orDCM Enterprise
Tags:

  5 minute read  

Deleting a Node from a Kubernetes cluster

In rare cases, it may be necessary to remove nodes from a Kubernetes cluster. This guide explains the prerequisites and the key considerations to keep in mind before starting the node removal process.

You can use the following steps to delete nodes from a Kubernetes cluster.

Prerequisites

  • To check which MON(monitor) and OSD (object storage daemon) are running on the node you want to delete you can use the command:
kubectl get po -n rook-ceph -o wide | grep worker02 | grep "mon\|osd" | grep -v "osd-prepare" | awk '{print $1}'
  • To check which mon and osd is running on the node you want to delete you can use the command kubectl get pods -n rook-ceph -o wide | grep <node-name> | grep "mon\|osd" | grep -v "osd-prepare" | awk '{print $1}'. As an output you get the mon and the osd running on that node. If you don’t get an output, you don’t have to delete the ressource and can skip to the “delete the node”-section

Worker Node Removal

In this example, worker01 (zone1) is removed from the cluster. Worker01 hosts osd.0 and mon-c.

1. Scale down the rook-ceph-operator deployment

Scale the operator down to 0 replicas to prevent new MONs or OSDs from being created during the removal process.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=0

2. Check Which Hosts and OSDs Belong to Each Zone

Run the following command:

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

Example output:

ID   CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
 -1         0.21478  root default
 -9         0.04880      zone zone1
 -7         0.04880          host worker01                              # worker01 is being removed
  0    ssd  0.04880              osd.0          up   1.00000  1.00000   # osd.0 is being removed
-15         0.04880          host worker04
  3    ssd  0.04880              osd.3          up   1.00000  1.00000
-11         0.10739      zone zone2
 -3         0.05859          host worker02
  1    ssd  0.05859              osd.1          up   0.95001  1.00000
-13         0.05859      zone zone3
 -5         0.05859          host worker03
  2    ssd  0.05859              osd.2          up   0.95001  1.00000

From this output, you can see that osd.0 belongs to worker01.

3. Scale down the OSD deployment

Scale down the OSD deployment running on the node to be removed:

kubectl scale deploy -n rook-ceph rook-ceph-osd-<x> --replicas=0
Example 
kubectl scale deploy -n rook-ceph rook-ceph-osd-0 --replicas=0

4. Remove the OSD via ceph-tools

Open a shell in the rook-ceph-tools pod and remove the OSD:

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- bash
# show OSD tree
ceph osd tree
# mark OSD out
ceph osd out <x>
# Example: ceph osd out 0
ceph osd purge <x> --yes-i-really-mean-it
# Example: ceph osd purge 0 --yes-i-really-mean-it
ceph auth del osd.<x>
# adjust CRUSH map
ceph osd crush remove <nodename>
# exit from ceph-tools
exit
# show OSD tree (now without the deleted node)
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

Check the MON running on the node

Before deleting a MON deployment, check which MON is running on the node you want to remove.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon -o wide

Check the MON running on the node

Before deleting a MON deployment, check which MON is running on the node you want to remove.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon -o wide

After this step, verify that the deleted OSD and node no longer appear in the OSD tree.

5. Delete OSD and MON deployments

Delete OSD and MON deployments

After checking the OSD and MON running on the node, delete the corresponding deployments.

kubectl delete deploy -n rook-ceph rook-ceph-osd-<x> rook-ceph-mon-<y>
Example 
kubectl delete deploy -n rook-ceph rook-ceph-osd-0 rook-ceph-mon-c

6. Remove the deleted mon from the ceph tools

Run the following commands to remove the deleted MON entry:

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm <y>
# verify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
Example 
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm c
# verify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

Example output before removal:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b
2: [v2:192.168.196.110:3300/0,v1:192.168.196.110:6789/0] mon.c

Example output after removal::

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b

7. Delete the node from the Kubernetes cluster

  • Remove the node you want to delete from the cluster-values.yaml file.
  • After updating the file, apply the deletion with the following command:
kubeopsctl apply --delete -f cluster-values.yaml
Example

Example cluster-values.yaml with worker01 removed and worker04 present:

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6      
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker04
    iPAddress: 10.2.10.214
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6  
  - name: worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

After updating the cluster-values.yaml, execute the delete command:

kubeopsctl apply --delete -f cluster-values.yaml

8. Scale the rook-ceph-operator Deployment Back to 1

After the node has been removed, scale the operator back to 1 replica:

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=1

This allows a new MON to be created automatically in zone2.

9. Check Timing and Cluster Health

The total duration depends on cluster size and node performance. Before continuing, verify that Ceph is healthy and that placement groups are clean.

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph status
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph pg stat

Typical duration ranges from 15 to 120 minutes.

If you want to rejoin the same node, reset it to a time prior to joining the cluster. Only this way you can be sure, that no leftovers from the deletion process remain!