KubeOpsDoku – How to Guides

Kubeopsctl-2.0: Documentation for production

Mon, 01 Jan 0001 00:00:00 +0000

Kubeopsctl-2.0: Joining a Node in a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

To increase performance or add additional resource capacity to your cluster, adding a node to the cluster is the correct process. This process with kubeopsctl is very easy.
You can use the following steps to join control-plane nodes or worker-nodes to a Kubernetes cluster.

Join Node Process:

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

ETCD Backup Recommendation

Before performing changes on the control planes, it is recommended to create an ETCD backup. Refer to the official Kubernetes documentation for details here

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull a specific Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-controlplaneXX
    iPAddress: 10.2.10.XXX
    type: controlplane
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone1
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone1.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6       # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true             # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-controlplaneXX   # -> has to be changed
    iPAddress: 10.2.10.XXX      # -> has to be changed
    type: controlplane
    kubeVersion: 1.31.6         # -> check with actual version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. If you are ready just start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

Example 1: Joining a Worker Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion x.xx.x you can pull other Kubernetes versions.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-workerXX
    iPAddress: 10.2.10.XX
    type: worker
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone2
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone2.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6        # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true              # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
  - name: demo-workerXX          # -> has to be changed
    iPAddress: 10.2.10.XX        # -> has to be changed
    type: worker
    kubeVersion: 1.31.6          # -> check with actual version     
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. Before joining the node, note the following:

Note

It is recommended if you join worker nodes to a cluster and rook-ceph is installed, to adjust your rook-ceph CRUSH map. Refer to the official rook-ceph documentation for details here. for updating the crush map, you should set in your enterprise-values.yaml:

...
packages:
- name: rook-ceph
  enabled: true
...

If you are ready, start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

If you need to apply the CRUSH map adjustment from above, use the following command instead:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Deleting a Node from a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Deleting a Node from a Kubernetes cluster

Note This guide requires a Compliance enterprise license.

In rare cases, it may be necessary to remove nodes from a Kubernetes cluster. This how-to guide explains the prerequisites and the key considerations to keep in mind before starting the node removal process.

You can use the following steps to delete nodes from a Kubernetes cluster.

Prerequisites

In order to run rook-ceph stable for a longer period your cluster needs at least 3 zones with each zone containing at least 1 worker-node
To check which mon and osd is running on the node you want to delete you can use the command kubectl get pods -n rook-ceph -o wide | grep <node-name> | grep "mon\|osd" | grep -v "osd-prepare" | awk '{print $1}'. As an output you get the mon and the osd running on that node. If you don’t get an output, you don’t have to delete the ressource and can skip to the “delete the node”-section

Worker

Important: Due to rook-ceph, a worker node must not be removed without following the steps below. In this example, worker01 (zone1) is removed from the cluster. Worker01 contains osd.0 and mon-c.

Scale down the rook-ceph-operator deployment to 0

This prevents new MONs or OSDs from being created.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=0

Check which hosts and OSDs belong to each zone

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

ID   CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
 -1         0.21478  root default
 -9         0.04880      zone zone1
 -7         0.04880          host worker01                              # worker01 is being removed
  0    ssd  0.04880              osd.0          up   1.00000  1.00000   # osd.0 is being removed
-15         0.04880          host worker04
  3    ssd  0.04880              osd.3          up   1.00000  1.00000
-11         0.10739      zone zone2
 -3         0.05859          host worker02
  1    ssd  0.05859              osd.1          up   0.95001  1.00000
-13         0.05859      zone zone3
 -5         0.05859          host worker03
  2    ssd  0.05859              osd.2          up   0.95001  1.00000

From this output you can see that osd.0 is part of worker01.

Scale down the OSD deployment

kubectl scale deploy -n rook-ceph rook-ceph-osd-<x> --replicas=0
# Example: kubectl scale deploy -n rook-ceph rook-ceph-osd-0 --replicas=0

Remove the OSD via ceph-tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- bash
# show OSD tree
ceph osd tree
# mark OSD out
ceph osd out <x>
# Example: ceph osd out 0
ceph osd purge <x> --yes-i-really-mean-it
# Example: ceph osd purge 0 --yes-i-really-mean-it
ceph auth del osd.<x>
# adjust CRUSH map
ceph osd crush remove <nodename>
# exit from ceph-tools
exit
# show OSD tree (now without the deleted node)
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

Check the MON running on the node

Before deleting a MON deployment, check which MON is running on the node you want to remove.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon -o wide

Delete OSD and MON deployments

After checking the OSD and MON running on the node, delete the corresponding deployments.

kubectl delete deploy -n rook-ceph rook-ceph-osd-<x> rook-ceph-mon-<y>

Example

kubectl delete deploy -n rook-ceph rook-ceph-osd-0 rook-ceph-mon-c

Remove the deleted mon from the ceph tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm <y>
# verify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

Example

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm c
# verfify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

This is the dump before executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b
2: [v2:192.168.196.110:3300/0,v1:192.168.196.110:6789/0] mon.c

This is the dump after executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b

Delete the node from the kubernetes cluster

Remove the node you want to delete from the cluster-values.yaml file.
After updating the file, apply the deletion with the following command:

kubeopsctl apply --delete -f cluster-values.yaml

Example

The cluster-values.yaml without node1 but with node4

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6      
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker04
    iPAddress: 10.2.10.214
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6  
  - name: worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

After updating the cluster-values.yaml, execute the delete command:

kubeopsctl apply --delete -f cluster-values.yaml

Scale the rook-ceph-operator deployment back to 1

This allows a new MON to be created automatically in zone2.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=1

Timing and health checks

The total duration depends on cluster size and node performance. Before proceeding, verify Ceph health and placement groups are clean.

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph status
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph pg stat

Typical duration ranges from 15 to 120 minutes.

If you want to rejoin the same node, reset it to a time prior to joining the cluster. Only this way you can be sure, that no leftovers from the deletion process remain!

Kubeopsctl-2.0: Single Sign-On with Keycloak

Mon, 01 Jan 0001 00:00:00 +0000

In this guide, you will learn how to implement Single Sign-On (SSO) using Keycloak. We will walk through the complete flow—from understanding SSO for platforms and services such as Rook Ceph, Harbor, OpenSearch, and other Kubernetes applications, to configuring Keycloak, exposing it securely, and integrating it with kubeops.

By the end of this guide, you will be able to:

Understand how Keycloak enables centralized authentication
Configure Keycloak for SSO
Securely expose Keycloak using Kubernetes Ingress and TLS
Integrate Keycloak with kubeops for authentication and authorization
Validate and troubleshoot the SSO login flow

Let’s get started on enabling secure and seamless authentication with Keycloak.

Kubeopsctl-2.0: Upgrading a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

Upgrading a Kubernetes cluster is essential to maintain security, stability, and compatibility.Like Kubernetes itself, we adhere to the version skew policy and only allow upgrades between releases that differ by a single minor version. This ensures compatibility between components, reduces the risk of instability, and keeps the cluster in a supported and secure state.

For more information about the Version Skew Policy, see the official Kubernetes documentation:
Click here

Use the following steps to upgrade the Kubernetes version of a cluster.

Kubernetes Version Upgrade Process:

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull an older Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull --kubernetesVersion <x.xx.x>

2. Change your target version inside the cluster-values

Note The actual version is the current Kubernetes version in your cluster. The target version in the nodes, is version we want to do the upgrade (e.g., see this yaml file)

Note To upgrade your cluster, the parameter changeCluster must be set to true (e.g., see this yaml file)

3. Start the upgrade

kubeopsctl apply -f cluster-values.yaml

Important It is important to start the upgrade with the initial controlplane node. It is the first node with which the cluster was created.

Example 1 - Upgrade all nodes in the cluster to a specific version

In this example, a cluster is upgraded from Kubernetes version v1.33.5 to v1.34.1.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.34.1 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.34.1

2. Update the target version in `cluster-values.yaml`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version in the kubeVersion field of each node.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.33.5     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true           # -> important! Needs to be set for an upgrade
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.34.1       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.34.1       # -> target version

2. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully.
When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

Example 2 - Tranche upgrade of zones to a specific version

In this example, the cluster is upgraded in tranches. The order is:

zone1 (because it contains the initial control plane node),
then zone3,
and finally zone2.

Important It is important to start your tranche upgrade with zone1, because zone1 contains the initial control plane node.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.33.5 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.33.5

2. Adjust your cluster-values in `zone1`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version only for the nodes in zone1. In the snippet below, only zone1 is configured with the target version.

Note Pay close attention to the format and syntax in cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

3. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

4. Adjust your `cluster-values.yaml` in `zone2`

Next, change the target version of zone2:

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

5. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

6. Adjust your `cluster-values.yaml` `in zone3`

Finally, change the target version of zone 3.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.33.5       # -> target version

7. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Kubeopsctl-2.0: Installing KubeOps Compliance applications

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be individually configured as needed.

By separating the cluster values from the application values, the application values can be modified independently and installed at a later stage, providing greater flexibility and maintainability.

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

⚠ Warning To ensure proper DNS resolution, all components of the logging stack — Filebeat, Logstash, OpenSearch, and OpenSearch Dashboards—must be deployed within the same Kubernetes namespace (or network domain).

Example 1: Installing Applications in a non-airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. For more information about available packages as well as parameters for each package check here.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2      
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and install the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the installation process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Installing Applications in an airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                        # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2      
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Updating KubeOps Compliance applications

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be configured individually as needed.

By separating cluster values from application values, application values can be modified independently and installed later, providing greater flexibility and maintainability.

kubeopsctl automatically detects whether an application is already deployed and updates it accordingly. de]

Important If you have made changes to your deployments, reapply them afterward or secure them using the advancedValues configuration.

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

Updated KubeOpsctl

If you have an older kubeopsctl version installed, update it before starting with updating Compliance appliactions.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

Example 1: Updating Applications in a non-airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application update process
Important for only update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and update the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the update process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Updating Applications in an airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Update kubeopsctl

If you have an older kubeopsctl version installed, update it using the following commands.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application update process
Important for only the update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                          # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Harbor Deployment with CloudNativePG

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to deploy Harbor on Kubernetes using a CloudNativePG (CNPG) PostgreSQL cluster managed by the CloudNativePG operator in an air‑gap and non-airgap environments.

Harbor Deployment with CloudNativePG in Non-airgap enviroment

Prerequisites

A running Kubernetes cluster.
kubectl, kosi, and kubeopsctl installed and configured.
You need to login with kosi . Refer to the official KOSI documentation for details here.

Step 1 — Install CloudNativePG operator

Deploy the operator with Kosi:

kosi install --hub kubeops kubeops/cloudnative-pg-operator:<kubeopsctl_version> --dname cnpg-operator

With this step, it Installs the CloudNativePG operator into the cluster and the operator manages PostgreSQL clusters and their lifecycle.

Step 2 — Create PostgreSQL cluster for Harbor

1. Apply the following Cluster manifest to create a Postgres cluster with 2 instances and 1Gi storage:

cat <<EOF | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloudnative-pg
  namespace: harbor
spec:
  instances: 2
  imagePullSecrets:
  - name: registry-pullsecret
  storage:
    size: 1Gi
EOF

2. Services and pods created for the cluster `cloudnative-pg`:

cloudnative-pg-rw → primary (read/write) cloudnative-pg-ro → replicas (read-only) cloudnative-pg-r → all pods

3. Verify pods are Running:

kubectl get pods -n harbor

Step 3 — Retrieve application user credentials

CNPG automatically creates a Secret named cloudnative-pg-app in the harbor namespace.

1. Verify the Secret exists:

kubectl get secret cloudnative-pg-app -n harbor

2. Decode the base64-encoded fields:

kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.username}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.password}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.dbname}" | base64 -d

Example values (for illustration only):

username: app
password: Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2
database: app

Important:
Use the non-superuser application credentials from this Secret in Harbor’s configuration.

Step 4 — Update Harbor tools.yaml for an external database

Edit your tools.yaml and set Harbor values under the helm chart configuration.
Example snippet:

- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "<DB_PASSWORD>"
      redisPassword: "Redis_Password"
      externalURL: <your_domain_name>
      nodePort: 30002
      hostname: <your_domain_name>
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
      database:
        type: external
        external:
          host: "cloudnative-pg-rw.harbor.svc.cluster.local"
          port: "5432"
          username: "app"
          password: "Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2"
          coreDatabase: "app"

Important: Use the -rw service host (cloudnative-pg-rw…) for write operations.
Do not use a superuser account.
Ensure the password matches the CNPG Secret.

Step 5 — Install Harbor with Kosi

1. Deploy Harbor using the updated tools.yaml:

kosi install --hub kubeops kubeops/harbor:2.0.3 -f tools.yaml --dname harbor

2. Verify Harbor pods:

kubectl get pods -n harbor

3. Access Harbor at: <your_domain_name>:30002 (or as configured)

Harbor Deployment with CloudNativePG in Airgap envoirnment

Prerequisites

A running Kubernetes cluster.
podman, kubectl, kosi, and kubeopsctl installed and configured.
You need to login with kosi . Refer to the official KOSI documentation for details here.

First, log in to your Harbor registry and pull the CloudNativePG operator package using kosi.

podman login <ip_address>:<nodePort> -u <user_name> -p <password> --tls-verify=false   
kosi pull --hub kubeops kubeops/cloudnative-pg-operator:<kubeopsctl_version> -o cloudnative-pg.tgz -r <ip_address>:<NodePort>/kubeops -t localhost:<NodePort>/kubeops

kosi install -p cloudnative-pg.tgz --dname cld-pg

Step 2: Manually Mirror the PostgreSQL Image to the KubeOps Project

In an air‑gapped environment, you must manually pull the PostgreSQL image from the external registry and push it to the KubeOps project in Harbor.

podman pull ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie

podman tag ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie <ip_address>:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie

podman push <ip_address>:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie --tls-verify=false

Step 3: Create the PostgreSQL Cluster for Harbor

Create a CNPG PostgreSQL cluster with two instances and 1Gi of storage in the harbor namespace:

cat <<EOF | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloudnative-pg
  namespace: harbor
spec:
  instances: 2
  storage:
    size: 1Gi
EOF

Step 4: Configure the Cluster to Use the Mirrored Image

Update the CNPG cluster to use the image you pushed to Harbor in Step 2.

1.List the cluster:

kubectl get cluster -n harbor

2.Edit the cluster:

kubectl edit cluster -n harbor cloudnative-pg

3.Replace the image name in the editor:

From:

ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie

To:

localhost:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie

Step 5: Retrieve CNPG Application User Credentials

CNPG automatically creates a Secret named cloudnative-pg-app in the harbor namespace.

1. Verify the Secret exists:

kubectl get secret cloudnative-pg-app -n harbor

2. Decode the base64-encoded fields:

kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.username}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.password}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.dbname}" | base64 -d

Example values (for illustration only):

username: app
password: Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2
database: app

Important:
Use the non-superuser application credentials from this Secret in Harbor’s configuration.

Step 6: Configure Harbor in enterprise.yaml

Update your enterprise-values.yaml to configure Harbor with the external CNPG database.
Example snippet:

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
  - name: harbor
    enabled: true
    values:
      standard:
        namespace: harbor
        harborpass: "password"
        databasePassword: "<DB_PASSWORD>"
        redisPassword: "Redis_Password"
        externalURL: <your_domain_name>
        nodePort: 30002
        hostname: <your_domain_name>
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi
              storageClass: "rook-cephfs"
            jobservice:
              jobLog:
                size: 1Gi
                storageClass: "rook-cephfs"
            database:
              size: 1Gi
              storageClass: "rook-cephfs"
            redis:
              size: 1Gi
              storageClass: "rook-cephfs"
            trivy: 
              size: 5Gi
              storageClass: "rook-cephfs"
      advanced:
        database:
          type: external
          external:
            host: "cloudnative-pg-rw.harbor.svc.cluster.local"
            port: "5432"
            username: "app"
            password: "Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2"
            coreDatabase: "app"

Important: Use the -rw service host (cloudnative-pg-rw…) for write operations.
Do not use a superuser account.
Ensure the password matches the CNPG Secret.

Apply the updated configuration:

kubeopsctl apply -f enterprise-values.yaml

Step 6 — Verify the Harbor Deployment

Check that Harbor pods are running in the harbor namespace:

kubectl get pods -n harbor

All Harbor pods should eventually reach a Running or Ready state

General Notes and best practices

Always decode the CNPG app Secret to obtain the correct username, password, and database name used by Harbor.
Use the cloudnative-pg-rw service for write traffic; use -ro services for reads if required.
Production recommendations (follow your organizational security policies):
Create a dedicated DB user (for example, harbor) rather than reusing the default app user.
Enable TLS for Postgres connections.
Implement backups (for example, CNPG Barman or object storage like S3).
To scale the CNPG cluster, update the instances field in the Cluster spec and reapply.

Kubeopsctl-2.0: Ingress Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Manual configuration of the Nginx-Ingress-Controller

Right now the Ingress Controller Package is not fully configured. To make complete use of the Ingress capabilities of the cluster, the user needs to manually update some of the settings of the corresponding service.

Choosing an ingress controller. KubeOps ships two ingress controllers: ingress-nginx and traefik. NGINX has reached end of support, so new clusters should prefer Traefik and existing clusters should plan a migration. To replace nginx-ingress with Traefik (including how SSO, certificates and dashboards keep working), follow How to migrate from nginx to traefik ingress. The rest of this page covers the manual nginx service configuration.

Locating the service

The service in question is called “ingress-nginx-controller” and can be found in the same namespace as the ingress package itself. To locate the service across all namespaces, you could use the following command.

kubectl get service -A | grep ingress-nginx-controller

This command should return two entries of services, “ingress-nginx-controller” and “ingress-nginx-controller-admission”, though only the first one needs to be further adjusted.

Setting the Ingress-Controller service to type NodePort

To edit the service, you can use the following command, although the actual namespace may be different. This will change the service type to NodePort.

kubectl patch service ingress-nginx-controller -n ingress -p '{"spec":{"type":"NodePort"}}'

Kubernetes will now automatically assign unused portnumbers for the nodePort to allow http and https connections to the service. These can be retrieved by running the same command, used to locate the service. Alternatively, you can use the following command, which adds the portnumbers 30080 and 30443 for the respective protocols. By doing so, you have to make sure, that these portnumbers are not being used by any other NodePort service.

kubectl patch service ingress-nginx-controller -n ingress --type=json -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}, {"op":"add","path":"/spec/ports/0/nodePort","value":30080}, {"op":"add","path":"/spec/ports/1/nodePort","value":30443}]'

Configuring external IPs

If you have access to external IPs that route to one or more cluster nodes, you can expose your Kubernetes-Services of any type through these addresses. The command below shows how to add an external IP-Adress to the service with the example value of “192.168.0.1”. Keep in mind that this value has to be changed in order to fit your networking settings.

kubectl patch service ingress-nginx-controller -n ingress -p '{"spec":{"externalIPs":["192.168.0.1"]}}'

Kubeopsctl-2.0: Accessing Dashboards installed with KubeOps

Mon, 01 Jan 0001 00:00:00 +0000

To access an application dashboard, an SSH tunnel to one of the control planes is needed. The following dashboards are available and configured with the following NodePorts by default:

⚠ Warning Change default credentials before going to production.

Note The NodePorts and hostnames shown on this page reflect the default values. Both can be configured in the enterprise-values.yaml. See Fileformats for details.

NodePort

32090 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

To establish a tunnel, forward the NodePort from one of the control planes to your local machine. You can use the command line, PuTTY, or MobaXterm for this.
After that, the dashboard can be accessed at localhost:32090/prometheus/.

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/prometheus/.

No credentials are necessary for login.

NodePort

30211 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/grafana/.

username: the username set in the enterprise-values.yaml of Prometheus (default: user)
password: the password set in the enterprise-values.yaml of Prometheus (default: password)

NodePort

30050 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/opensearch/.

OpenSearch Dashboards supports Single Sign-On (SSO) via Keycloak as the primary login, with username/password as a fallback.

SSO (recommended): open https://<your-domain>/opensearch/ and choose Log in with single sign-on. You are redirected to Keycloak. See the SSO for OpenSearch guide for setup and how to grant users access.
Username / password (fallback):
- username: admin
- password: the value of adminPassword in the enterprise-values.yaml (default: Password@@123456)

Note

SSO is only wired up when the OpenSearch hostname points to a reachable Keycloak (non-.local). For *.local hostnames only the username/password login is available. See Changing the OpenSearch password to change admin.

NodePort

HTTP: 30002
HTTPS: 30003

Connecting via SSH Tunnel

To establish a tunnel, forward port 30002 from one of the control planes to your local machine. You can use the command line, PuTTY, or MobaXterm for this.
After that, the dashboard can be accessed at localhost:30002/.

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/.

username: admin
password: the password set in the enterprise-values.yaml (default: password)

NodePort

The Rook/Ceph dashboard has no fixed NodePort. To find the NodePort in use, run one of the following:

kubectl get svc -n rook-ceph

Look for the rook-ceph-mgr-dashboard-external-http service:

NAME                                      TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                                     AGE
rook-ceph-mgr-dashboard-external-http     NodePort    192.168.197.13    <none>        7000:31268/TCP                              21h

Or retrieve the NodePort directly:

echo $(kubectl get -n rook-ceph -o jsonpath="{.spec.ports[0].nodePort}" services rook-ceph-mgr-dashboard-external-http)

In the example above, the NodePort is 31268.

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/ceph-dashboard/.

echo Username: admin
echo Password: $(kubectl get secret rook-ceph-dashboard-password -n rook-ceph --template={{.data.password}} | base64 -d)

NodePort

30007 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/kubeops-dashboard/.

kubectl -n monitoring create token headlamp-admin

NodePort

30180

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/keycloak/.

echo Username: $(kubectl get secret --namespace keycloak keycloak-kubeops -o jsonpath="{.data.ADMIN_USER}" | base64 -d)
echo Password: $(kubectl get secret --namespace keycloak keycloak-kubeops -o jsonpath="{.data.ADMIN_PASSWORD}" | base64 -d)

Kubeopsctl-2.0: Accessing Dashboards with OIDC

Mon, 01 Jan 0001 00:00:00 +0000

This how-to guide explains how to enable OIDC-based access to your Kubernetes application dashboards using kubeopsctl. You will configure a single enterprise-values.yaml file that defines the required components. Once the configuration is applied, the dashboards for Harbor, Grafana, Rook-Ceph, OpenSearch, Keycloak, the KubeOps dashboard, and Prometheus become accessible via their configured hostnames and paths. Following the steps in this guide, you will set up a consistent, centralized OIDC integration and make your dashboards securely available through a browser.

Prerequisites

yq should be installed on all master nodes. For more infomration refer yq installation guide.
Keycloak should be installed before other packages.
Pull all the required packages with ‘kubectl pull’ command. For more information refer Pull Packages with Kubeopsctl

Step 1: Configure enterprise-values.yaml

In order to connect to dashboad with OIDC, you need to configure your enterprise-values.yaml file as below

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: rook-ceph
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
            mon:
              requests:
                cpu: "1"
                memory: "1Gi"
            osd:
              requests:
                cpu: "1"
                memory: "1Gi"
          dashboard:
            enabled: "true"
        operator:
          data:
            rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: <domain_name>          # required for OIDC dashboard access
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: cert-manager 
  enabled: true
  values:
    standard:
      namespace: cert-manager
      replicaCount: 3
      logLevel: 2
    advanced:
    ca:      
      emailLetsEncrypt: <email>
      ingressName: nginx
- name: ingress-nginx
  enabled: true
  values:
    standard:
      namespace: ingress-nginx
      externalIPs: 
        - <>                 
      advanced:
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: "keycloak"
      storageClass: "rook-cephfs"
      nodePort: "30180"
      hostname: <domain_name>          # required for OIDC dashboard access
      keycloak:
        auth:
          adminUser: admin
          adminPassword: topsecret
      postgresql:
        auth:
          postgresUserPassword: "changeme"
          username: bn_keycloak
          password: "changeme"
          database: bitnami_keycloak
        volumeSize: "8Gi"
    advanced:
- name: filebeat-os
  enabled: true
  values:
    standard:
      namespace: logging
    advanced:
- name: logstash-os
  enabled: true
  values:
    standard:
      namespace: logging
      volumeClaimTemplate:
        accessModes: 
          - ReadWriteMany
        resources:
          requests:
            storage: 1Gi
        storageClass: "rook-cephfs"
    advanced:
- name: opensearch-dashboards
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access 
      namespace: logging
      nodePort: 30050
    advanced:
- name: opensearch-os
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access
      namespace: logging
      opensearchJavaOpts: "-Xmx512M -Xms512M" # optional, default is -Xmx512M -Xms512M
      resources:
        requests:
          cpu: "250m" # optional, default is 250m
          memory: "1024Mi" # optional, default is 1024Mi
        limits:
           cpu: "300m" # optional, default is 300m
           memory: "3072Mi" # optional, default is 3072Mi
      persistence:
        size: 4Gi # mandatory
        enabled: "true" # optional, default is true
        enableInitChown: "false" # optional, default is false
        labels:
          enabled: "false" # optional, default is false
        storageClass: "rook-cephfs" # optional, default is rook-cephfs
        accessModes:
          - "ReadWriteMany" # optional, default is {ReadWriteMany}
        securityConfig:
          enabled: "false" # optional, default value: false
        replicas: "3" # optional, default is 3
    advanced:
- name: kube-prometheus-stack
  enabled: true
  values:
    standard:
      namespace: monitoring
      grafanaUsername: admin
      grafanaPassword: topsecret
      retentionSize: "24GB"
      grafanaResources:
        hostname: <domain_name>          # required for OIDC dashboard access
        nodePort: 30211
        retention: 10d
        retentionSize: "24GB"
        storageClass: "rook-cephfs"
        storage: 25Gi
      prometheusResources:
        hostname: <domain_name>          # required for OIDC dashboard access
        retentionSize: "24GB"
        storageClass: "rook-cephfs"
        storage: 25Gi
    advanced:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: opa-gatekeeper
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: <domain_name>          # required for OIDC dashboard access
      service:
        nodePort: 30007
    advanced:
- name: velero
  enabled: true
  values:
    standard:
      namespace: "velero"
      accessKeyId: "your_s3_storage_username"
      secretAccessKey: "your_s3_storage_password"
      useNodeAgent: false
      defaultVolumesToFsBackup: false
      provider: "aws"
      bucket: "velero"
      useVolumeSnapshots: false
      backupLocationConfig:
        region: "minio"
        s3ForcePathStyle: true
        s3Url: "http://minio.velero.svc:9000"
    advanced:

step 2: Apply the Configuration with ‘KubeOpctl’

Apply the enterprise-values.yaml configuration using the following command:

kubeopsctl apply -f enterprise-values.yaml

Wait until all components are deployed and running before proceeding.

Step 3: Connect to Dashboards via Hostname

To access the dashboards via OIDC, use the hostnames configured in your enterprise-values.yaml (uservalues). Harbor is an exception: you can access it directly using the base domain name.

Use the following paths, replacing <domain_name> with your configured domain:

<domain_name>/ for Harbor
<domain_name>/grafana for grafana
<domain_name>/ceph-dashboard for rook-ceph
<domain_name>/opensearch for opensearch
<domain_name>/keycloak for keycloak
<domain_name>/kubeops-dashboard for Kubeops Dashboard
<domain_name>/prometheus for Prometheus

Kubeopsctl-2.0: Changing a User Password in OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains how to change passwords in OpenSearch.

Recommended: Change the `admin` password via the package value

The OpenSearch admin password is managed through the adminPassword package value. Set it for the opensearch-os package (and the matching opensearch-dashboards package) in your enterprise-values.yaml and run an update:

packages:
  - name: opensearch-os
    enabled: true
    values:
      standard:
        adminPassword: "<your-new-password>" # must match opensearch-dashboards
  - name: opensearch-dashboards
    enabled: true
    values:
      standard:
        adminPassword: "<your-new-password>" # must match opensearch-os

On the next kubeopsctl apply/update the new password is enforced automatically after the deployment. This also works on existing clusters and does not remove any other internal users.

Note Internal users created in the OpenSearch Dashboards Security UI persist across package updates. The update only enforces the admin password and reapplies the authentication/authorization configuration; it never overwrites your internal users.

Advanced: Change any user’s password manually via `securityadmin`

Use this method to change the password of any internal user (for example testuser or kibanaro) without a package update.

This process reads the live internal users from the security index (via securityadmin -backup), so every other user — including users you created in the OpenSearch Dashboards Security UI — is preserved. Do not build the file from the internal-users-config-secret: that secret only holds the packaged baseline users and would remove UI-created users when applied.

Prerequisites

Access to the Kubernetes cluster and kubectl.
The admin certificate is available in the pod at /usr/share/opensearch/config/certs/ (root-ca.pem, admin.pem, admin-key.pem) in a KubeOps deployment.

Set a shell variable for convenience (adjust the namespace and pod name as needed):

NS=logging; POD=opensearch-cluster-master-0

Step 1: Back up the current internal users (live state)

kubectl exec -it $POD -n $NS -- bash -c "\
    /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -backup /tmp/os-sec \
    -icl -nhnv \
    -cacert /usr/share/opensearch/config/certs/root-ca.pem \
    -cert /usr/share/opensearch/config/certs/admin.pem \
    -key /usr/share/opensearch/config/certs/admin-key.pem"

This writes the current configuration — including all internal users — to /tmp/os-sec/ in the pod.

Note securityadmin -backup may end with Configuration for 'audit' failed because of empty source and a non-zero exit code. This is harmless: internal_users.yml is written before that step, so the file you need is already present.

Step 2: Generate a new password hash for each user

Run this once per user and note the resulting hash:

kubectl exec -it $POD -n $NS -- bash -c "sh /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p <new_password>"

Step 3: Update the hashes in the backup file

Copy the file out, edit it locally, and copy it back:

kubectl cp $NS/$POD:/tmp/os-sec/internal_users.yml ./internal_users.yml
# edit ./internal_users.yml: replace the hash: value under each user you want to change
kubectl cp ./internal_users.yml $NS/$POD:/tmp/os-sec/internal_users.yml

Only change the hash: line of the target users; leave everyone else untouched:

testuser:
  hash: "<new-hash-for-testuser>"
  reserved: false
  backend_roles:
  - "testrole"
kibanaro:
  hash: "<new-hash-for-kibanaro>"
  reserved: false
  backend_roles:
  - "kibanauser"

Step 4: Apply only the internal users configuration

Note

The DN of the certificate specified with -cert must be listed under plugins.security.authcz.admin_dn in opensearch.yml. In a KubeOps deployment the admin certificate subject is C=DE,L=test,O=client,OU=client,CN=admin.

More information: Opensearch Documentation.

Apply only the internalusers config type. Do not use -cd (the whole directory) here: -cd requires the complete security config set (including nodes_dn.yml, allowlist.yml, audit.yml), which a KubeOps deployment does not ship, so it would fail with nodes_dn.yml ... No such file or directory.

kubectl exec -it $POD -n $NS -- bash -c "\
    /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -f /tmp/os-sec/internal_users.yml -t internalusers \
    -icl -nhnv \
    -cacert /usr/share/opensearch/config/certs/root-ca.pem \
    -cert /usr/share/opensearch/config/certs/admin.pem \
    -key /usr/share/opensearch/config/certs/admin-key.pem"

The change is written to the security index and takes effect immediately — no pod restart required.

Important

Once applied with securityadmin.sh, credentials are stored in the OpenSearch security index, which lives on the persistent volume and therefore survives pod and container restarts as well as package updates. Editing files inside the container without running securityadmin.sh only changes the local file and is not applied to the cluster. For the admin user, prefer the adminPassword package value described above — it is enforced again on every update and keeps all other internal users intact.

Kubeopsctl-2.0: Backup and restoring artifacts

Mon, 01 Jan 0001 00:00:00 +0000

What is Velero?

Velero uses object storage to store backups and associated artifacts. It also optionally integrates supported block storage systems to snapshot your persistent volumes. Before beginning the installation process, you should identify the object storage provider and optional block storage provider(s) you’ll be using from the list of compatible providers.

Velero supports storage providers for both cloud-provider environments and on-premises environments.

Velero prerequisites:

Access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled.
kubectl installed locally
Object Storage (S3, Cloud Provider Environment, On-Premises Environment)

Note : Compatible providers and on-premises documentation can be read on https://velero.io/docs

Install Velero

This command is an example on how you can install velero into your cluster:

velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.2.1 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000

NOTE:

s3Url has to be the url of your s3 storage login.

example for credentials-velero file:

[default]
aws_access_key_id = your_s3_storage_username
aws_secret_access_key = your_s3_storage_password

Backup the cluster

Scheduled Backups

This command creates a backup for the cluster every 6 hours:

velero schedule create cluster --schedule "0 */6 * * *"

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete cluster

Restore Scheduled Backup

This command restores the backup according to a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the cluster

velero backup create cluster

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Backup a specific deployment

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create filebeat --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “filebeat” every 6 hours:

velero schedule create filebeat --schedule "0 */6 * * *" --include-namespaces logging --selector app=filebeat-filebeat,release=filebeat --include-resources serviceaccount,deployment,daemonset,configmap,clusterrolebinding,clusterrole --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete filebeat

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create filebeat --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “filebeat”:

velero backup create filebeat --include-namespaces logging --selector app=filebeat-filebeat,release=filebeat --include-resources serviceaccount,deployment,daemonset,configmap,clusterrolebinding,clusterrole --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create logstash --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “logstash” every 6 hours:

velero schedule create logstash --schedule "0 */6 * * *" --include-namespaces logging --selector app=logstash-logstash,chart=logstash,release=logstash --include-resources StatefulSet,ServiceAccount,Service,Secret,RoleBinding,Role,PodSecurityPolicy,PodDisruptionBudget,Ingress,ConfigMap --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete logstash

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create logstash --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “logstash”:

velero backup create logstash --include-namespaces logging --selector app=logstash-logstash,chart=logstash,release=logstash --include-resources StatefulSet,ServiceAccount,Service,Secret,RoleBinding,Role,PodSecurityPolicy,PodDisruptionBudget,Ingress,ConfigMap --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create opensearch --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “opensearch” every 6 hours:

velero schedule create opensearch --schedule "0 */6 * * *" --include-namespaces logging --selector app.kubernetes.io/instance=opensearch,app.kubernetes.io/name=opensearch --include-resources ConfigMap,Ingress,NetworkPolicy,PodDisruptionBudget,PodSecurityPolicy,Role,RoleBinding,Secret,Service,ServiceAccount,StatefulSet --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete opensearch

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create opensearch --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “opensearch”:

velero backup create opensearch --include-namespaces logging --selector app.kubernetes.io/instance=opensearch,app.kubernetes.io/name=opensearch --include-resources ConfigMap,Ingress,NetworkPolicy,PodDisruptionBudget,PodSecurityPolicy,Role,RoleBinding,Secret,Service,ServiceAccount,StatefulSet --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “monitoring” every 6 hours:

velero schedule create prometheus --schedule "0 */6 * * *" --include-namespaces monitoring --include-cluster-resources=true

This command creates a backup for the deployment “prometheus” every 6 hours:

velero schedule create prometheus --schedule "0 */6 * * *" --include-namespaces monitoring --include-resources Alertmanager,Secret,Ingress,List,PodDisruptionBudget,Role,RoleBinding,PodSecurityPolicy,Service,ServiceAccount,ServiceMonitor,Endpoints,ConfigMap,ConfigMapList,ClusterRole,ClusterRoleBinding,SecretProviderClass,PodMonitor,Prometheus,Job,NetworkPolicy,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Issuer,Deployment,VerticalPodAutoscaler,ThanosRuler --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete prometheus

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “monitoring”:

velero backup create prometheus --include-namespaces monitoring --include-cluster-resources=true

This command creates a backup for the deployment “prometheus”:

velero backup create prometheus --include-namespaces monitoring --include-resources Alertmanager,Secret,Ingress,List,PodDisruptionBudget,Role,RoleBinding,PodSecurityPolicy,Service,ServiceAccount,ServiceMonitor,Endpoints,ConfigMap,ConfigMapList,ClusterRole,ClusterRoleBinding,SecretProviderClass,PodMonitor,Prometheus,Job,NetworkPolicy,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Issuer,Deployment,VerticalPodAutoscaler,ThanosRuler --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “harbor” every 6 hours:

velero schedule create harbor --schedule "0 */6 * * *" --include-namespaces harbor --include-cluster-resources=true

This command creates a backup for the deployment “harbor” every 6 hours:

velero schedule create harbor --schedule "0 */6 * * *" --include-namespaces harbor --include-resources ConfigMap,Deployment,PersistentVolumeClaim,Secret,Service,StatefulSet,Ingress,ServiceMonitor --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete harbor

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “harbor”:

velero backup create harbor --include-namespaces harbor --include-cluster-resources=true

This command creates a backup for the deployment “harbor”:

velero backup create harbor --include-namespaces harbor --include-resources ConfigMap,Deployment,PersistentVolumeClaim,Secret,Service,StatefulSet,Ingress,ServiceMonitor --include-cluster-resources=true --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “gatekeeper-system” every 6 hours:

velero schedule create gatekeeper --schedule "0 */6 * * *" --include-namespaces gatekeeper-system --include-cluster-resources=true

This command creates a backup for the deployment “gatekeeper” every 6 hours:

velero schedule create gatekeeper --schedule "0 */6 * * *" --include-namespaces gatekeeper-system --include-resources PodSecurityPolicy,ServiceAccount,Deployment,PodDisruptionBudget,ResourceQuota,ClusterRole,Role,ClusterRoleBinding,RoleBinding,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Secret,Service,Job --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete gatekeeper

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “gatekeeper-system”:

velero backup create gatekeeper --include-namespaces gatekeeper-system --include-cluster-resources=true

This command creates a backup for the deployment “gatekeeper-system”:

velero backup create gatekeeper --include-namespaces gatekeeper-system --include-resources PodSecurityPolicy,ServiceAccount,Deployment,PodDisruptionBudget,ResourceQuota,ClusterRole,Role,ClusterRoleBinding,RoleBinding,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Secret,Service,Job --include-cluster-resources=true --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “rook-ceph” every 6 hours:

velero schedule create rook-ceph --schedule "0 */6 * * *" --include-namespaces rook-ceph --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete rook-ceph

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “rook-ceph”:

velero backup create rook-ceph --include-namespaces rook-ceph --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

restore databases

keycloak

create backup of keycloak namespace, in this example the backup is called keycloak1.

This command creates a backup for the namespace “rook-ceph”:

velero backup create keycloak1 --include-namespaces keycloak --include-cluster-resources=true

restore backup, in this example keycloak1

velero restore create keycloak1 --from-backup keycloak1

restore the database dump:

kubectl -n <keycloak-namespace> exec keycloak-postgres-0 -- pg_restore -v --jobs=4 --clean --if-exists -d bitnami_keycloak /backup/keycloak-db.dump

Kubeopsctl-2.0: Add certificate as trusted

Mon, 01 Jan 0001 00:00:00 +0000

1. Download the certificate

As soon as Chrome issues a certificate warning, click on Not secure to the left of the address bar.
Show the certificate (Click on Certificate is not valid).
Go to Details tab.
Click Export... at the bottom and save the certificate.

As soon as Firefox issues a certificate warning, click on Advanced....
View the certificate (Click on View Certificate).
Scroll down to Miscellaneous and save the certificate.

2. Install the certificate

Press Windows + R.
Enter mmc and click OK.
Click on File > Add/Remove snap-in....
Select Certificates in the Available snap-ins list and click on Add >, then on OK. Add the snap-in.
In the tree pane, open Certificates - Current user > Trusted Root Certification Authorities, then right-click Certificates and select All tasks > Import....
The Certificate Import Wizard opens here. Click on Next.
Select the previously saved certificate and click Next.
Click Next again in the next window.
Click on Finish. If a warning pops up, click on Yes.
The program can now be closed. Console settings do not need to be saved.
Clear browser cache and restart browser.

The procedures for using a browser to import a certificate as trusted (on Linux systems) vary depending on the browser and Linux distribution used. To manually cause a self-signed certificate to be trusted by a browser on a Linux system:

Distribution	Copy certificate here	Run following command to trust certificate
RedHat	`/etc/pki/ca-trust/source/anchors/`	`update-ca-trust extract`

Note

If the directory does not exist, create it.

Note

If you do not have the ca-certificates package, install it with your package manager.

Kubeopsctl-2.0: Deploy Package on a cluster

Mon, 01 Jan 0001 00:00:00 +0000

You can install artifacts in your cluster in several ways. When creating a package, you can use one of the following plugins:

helm
kubectl
cmd
kosi

This guide demonstrates a complete, reproducible deployment of nginx-ingress using Helm and kosi.

Using the Helm Plugin

Pre-requisites

Before you begin, make sure the following tools are available in your environment:

helm
kosi

This example uses the official NGINX Helm repository and deploys nginx-ingress with Helm and KOSI.

Prepare the Files

In order to install an artifact with the Helm plugin, the Helm chart must first be downloaded.

To keep the environment clean and reproducible, first create a dedicated working directory and change into it:

mkdir nginx-ingress-kosi && cd nginx-ingress-kosi

The example uses the official NGINX Helm repository. Add the repository before performing any Helm operations:

helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update

To verify which chart versions are available, run:

helm search repo nginx-stable/nginx-ingress --versions

This guide uses chart version 0.16.1. Download the chart as a packaged Helm artifact:

helm pull nginx-stable/nginx-ingress --version 0.16.1

This command creates the following file in your working directory:

nginx-ingress-0.16.1.tgz

Create KOSI package

Once the chart is available locally, initialize the kosi package structure. The following command creates the necessary files in the current directory:

kosi create

The Helm chart remains the central input artifact for deployment, while configuration is handled via values.yaml. Helm already includes default values, which can be inspected or exported if needed:

helm show values nginx-stable/nginx-ingress --version 0.16.1 > values.yaml

Review and update values.yaml before packaging and deploying the chart. For example, in environments without a cloud-based LoadBalancer, you may need to change the service type in values.yaml.

controller:
  service:
    type: LoadBalancer

Only required overrides should be defined in the customized values.yaml file. To customize the deployment of the Helm chart, a new values.yaml file in the current directory must be created and edited with this content:

controller:
  service:
    type: NodePort

This override is merged with the chart defaults at runtime.

Helm charts do not directly contain the full list of container images required for deployment. To identify them, render the chart into Kubernetes manifests:

helm template nginx-ingress nginx-stable/nginx-ingress --version 0.16.1 > manifests.yaml

Then extract all referenced images from the rendered manifest:

grep image: manifests.yaml

For this example, the required image is:

nginx/nginx-ingress:3.0.1

With all artifacts identified, the package.kosi file defines what is required for deployment. It explicitly references:

- the Helm chart archive
- the custom values.yaml
- all required container images

nginx-ingress-0.16.1.tgz
values.yaml
package.kosi

Note

The downloaded Helm chart must also be located in the current directory. To customize the deployment of the Helm chart, the values.yaml file must be edited.

All files required by a task in the package must be named in the package.kosi file under files. The container images required by the Helm chart must also be listed in the package.kosi under containers.

In the example below, only two files are required for the installation: the Helm Chart for the nginx-ingress and the values.yaml to configure the deployment. To install nginx-ingress you will also need the nginx/nginx-ingress image with the tag 3.0.1.

To install nginx-ingress with the Helm plugin, call the plugin as shown in the example under install. The deployment configuration file is listed under values and the packed Helm chart is specified with the key tgz. Furthermore, it is also possible to specify the namespace in which the artifact should be deployed and the name of the deployment. The full documentation for the Helm plugin can be found here.

Use the following configuration:

languageversion = "1.0.0";
apiversion = "kubernative/kubeops/sina/user/v4";
name = "deployexample1";
description = "It shows how to deploy an artifact to your cluster using the helm plugin.";
version = "0.1.0";
docs = "docs.tgz";
logo = "logo.png";

files =
{
        valuesFile = "values.yaml";
        nginxHelmChart="nginx-ingress-0.16.1.tgz";
}

containers =
{
        nginx = ["docker.io", "nginx/nginx-ingress", "3.0.1"];
}

install
{
        helm
        (
            command = "install";
            tgz = "nginx-ingress-0.16.1.tgz";
            values = "['values.yaml']";
            namespace = "dev";
            deploymentName = "nginx-ingress"
        );
}

Build the Package

Once the package.kosi file has been fully configured, all files must be combined into a kosi package. To do this, execute the following command in the directory where the package.kosi file is located.

kosi build

Push the Package

To make the generated kosi package available on other machines, it is pushed to the user’s private KubeOps Hub. To do this, the user must first log in to the hub and then push it to hub.

$ kosi login -u <username>
2023-02-04 11:19:43 Info:      KOSI version: 2.13.0
2023-02-04 11:19:43 Info:      Please enter password
****************
2023-02-04 11:19:26 Info:      Login Succeeded to Hub.
$ kosi push --hub kosi
2023-02-04 11:23:18 Info:      KOSI version: 2.13.0
2023-02-04 11:23:19 Info:      Push to Private Registry registry.kubeops.net/<username>/

Deploy the Package

Once the KOSI package has been created and published, it needs to be installed on the admin node. The following command will download and execute the package. The package name and version refer to the values defined in package.kosi with the keys name and version.

kosi install --hub <username> <username>/<packagename>:<version>

For the example package, the command would be: kosi install --hub <username> <username>/deployExample:0.1.0.

Using the Kubectl Plugin

Prerequisites

Before you begin, make sure the following tools and components are available:

kosi
the kubeops-kubernetes-plugins package installed on the admin node

This example deploys nginx-ingress from a Kubernetes manifest using the Kubectl plugin.

Create KOSI package

First you need to create a KOSI package. The following command creates the necessary files in the current directory:

kosi create

The NGINX ingress controller YAML manifest can either be automatically downloaded and applied directly with kubectl apply or it can be downloaded manually if you want to customize the deployment. The YAML manifest can be downloaded from the NGINX GitHub Repo and must be placed in the same directory as the files for the KOSI package.

All files required by a task in the package must be named in the package.kosi file under files. The container images required by the YAML manifest must also be listed in the package.kosi under containers. In the example below, only one file is required for the installation: the YAML manifest for the nginx-ingress controller. To install nginx-ingress you will also need the registry.k8s.io/ingress-nginx/controller image with the tag v1.5.1 and the image registry.k8s.io/ingress-nginx/kube-webhook-certgen with tag v20220916-gd32f8c343.

To install nginx-ingress with the Kubectl plugin, call the plugin as shown in the example under installs. The full documentation for the Kubectl plugin can be found here.

languageversion = "1.0.0";
apiversion = "kubernative/kubeops/sina/user/v4";
name = "deployexample2";
description = "It shows how to deploy an artifact to your cluster using the helm plugin.";
version = "0.1.0";
docs = "docs.tgz";
logo = "logo.png";

files =
{
     manifest: "deploy.yaml"
}

containers =
{
    nginx = ["registry.k8s.io", "ingress-nginx/controller", "v1.5.1"];
    certgen= ["registry.k8s.io","ingress-nginx/kube-webhook-certgen","v20220916-gd32f8c343"];
}

install
{
    kubectl
    (
      operation="apply",
      flags="-f deploy.yaml";
      sudo = true;
      sudoPassword="toor"
    );
}

Build the Package

Once the package.kosi file has been fully configured, all files must be combined into a KOSI package. To do this, execute the following command in the directory where the package.kosi file is located.

kosi build

Publish the Package

To make the generated KOSI package available on other machines, it is pushed to the user’s private KubeOps Hub. To do this, the user must first log in to the hub.

$ kosi login -u <username>
2023-02-04 11:19:43 Info:      kosi version: 2.13.0
2023-02-04 11:19:43 Info:      Please enter password
****************
2023-02-04 11:19:26 Info:      Login Succeeded to Hub.
$ kosi push --hub kosi
2023-02-04 11:23:18 Info:      kosi version: 2.13.0
2023-02-04 11:23:19 Info:      Push to Private Registry registry.kubeops.net/<username>/

Deploy the Package

kosi install --hub <username> <username>/<packagename>:<version>

For the example package, the command would be: kosi install --hub <username> <username>/deployExample:0.1.0.

Kubeopsctl-2.0: How to migrate from nginx to traefik ingress

Mon, 01 Jan 0001 00:00:00 +0000

Installation

Kubeops supports deploying Traefik as a dynamic ingress controller and reverse proxy. This guide describes a concise, safe migration from an existing nginx-ingress controller to Traefik and explains how to install Traefik and replace a deprecated nginx-ingress deployment. The migration from nginx to Traefik is straightforward; the steps below show the process in order.

Prerequisites

A running Kubernetes cluster with an existing nginx-ingress controller.
It is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

1.Create Values file

Create a values.yaml file for the Traefik installation:

# values.yaml
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik
      externalIPs: []
    advanced: {}

Note: Update externalIPs and any other values as required for your environment.

2.Install Traefik

non-airgap-environment

After creating values.yaml, install Traefik:

# get your desired version/s
kosi search --hub kubeops --ps traefik
# install traefik
kosi install --hub kubeops kubeops/traefik:<desired_version> -f values.yaml --dname traefik

Example Non-airgap

kosi install --hub kubeops kubeops/traefik:2.1.0_Beta0 -f values.yaml --dname traefik

Airgap-environment

# log in to Harbor
podman login <ip_address>:<NodePort> -u <username> -p <password> --tls-verify=false
# pull your desired version
kosi pull --hub kubeops kubeops/<desired_version> -o traefik-<desired_version>.tgz -r <ip_address>:<NodePort>/kubeops -t 127.0.0.1:<NodePort>/kubeops
 # install Traefik
kosi install --dname traefik -p traefik-:<desired_version>.tgz -f values.yaml

Example Airgap

kosi pull --hub kubeops kubeops/traefik:2.1.0_Beta0 -o traefik-2.1.0-Beta0.tgz -r 10.2.10.11:30002/kubeops -t 127.0.0.1:30002/kubeops
 
kosi install --dname traefik -p traefik-2.1.0-Beta0.tgz -f values.yaml

3.Verify Deployment

Verify that the Traefik pods and services are running in the traefik namespace:

kubectl get pods -n traefik
kubectl get svc -n traefik

3b. Switch applications to the Traefik ingress class

This is the load-bearing step of the migration. Traefik ships a backwards-compatibility shim (an IngressClass named nginx plus the experimental providers.kubernetesIngressNginx provider) so that existing ingressClassName: nginx ingresses should keep working unchanged. This shim is experimental and does not work reliably for all workloads — Keycloak in particular has been observed to be unreachable through the nginx class while Traefik is the controller. Do not rely on the shim as the primary mechanism; switch each application’s ingress to ingressClassName: traefik.

Every ingress package exposes a uniform standard.ingressClassName key (defaulting to nginx). Set it to traefik on each package, and set the cert-manager solver class:

- name: keycloak
  values:
    standard:
      ingressClassName: traefik
- name: kubeOpsDashboard
  values:
    standard:
      ingressClassName: traefik
- name: kube-prometheus-stack
  values:
    standard:
      ingressClassName: traefik
- name: harbor
  values:
    standard:
      ingressClassName: traefik
- name: rook-ceph
  values:
    standard:
      ingressClassName: traefik
- name: opensearch-dashboards
  values:
    standard:
      ingressClassName: traefik
- name: kubevirt-manager
  values:
    standard:
      ingressClassName: traefik
- name: cert-manager
  values:
    ca:
      ingressName: traefik         # ACME HTTP01 solver ingress class

cert-manager is the only exception to the standard.ingressClassName key: its solver class lives under values.ca.ingressName. Certificates and the cert-manager.io/cluster-issuer annotations are otherwise controller-agnostic.

Note: the plain OpenSearch data package (opensearch-os) and logstash ship their ingress disabled by default, so they need no change. If you enable them, set the class via that package’s advanced block (ingress.ingressClassName / ingress.className).

Apply the updated values and confirm every ingress now uses the Traefik class:

kubectl get ingress -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName

SSO / certificates / dashboards: SSO is app-level OIDC (Keycloak reachable at https://<host>/keycloak), so it is not tied to the ingress controller and keeps working once the class is switched. The same is true for cert-manager and dashboard access.

nginx-only annotations do not migrate. Annotations of the form nginx.ingress.kubernetes.io/* are ignored by Traefik. OpenSearch Dashboards ships buffer/ header-size annotations (proxy-buffer-size, large-client-header-buffers, …) that matter for large OIDC headers; if login fails with 4xx/431/502, add an equivalent Traefik buffering middleware / entrypoint transport setting via the package’s advanced block. Harbor auto-injects nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" when internalTLS.enabled — verify Harbor over TLS after the switch.

4.Remove old nginx-ingress deployment and service

# get version of installed nginx-ingress and its deployment name (--dname)
kosi list
# delete old nginx-ingress
kosi delete --hub kubeops kubeops/ingress-nginx:<installed_version> -f enterprise-values.yaml --dname <kosi_deployment_name>

Example


kosi delete --hub kubeops ingress-nginx:2.1.0_Beta0 -f values.yaml --dname ingress-nginx

Clean up the leftover nginx IngressClass. The ingress-nginx chart annotates its IngressClass with helm.sh/resource-policy: keep, so it survives kosi delete. Both ingress-nginx and the Traefik package define an IngressClass named nginx for controller k8s.io/ingress-nginx, so they must not run at the same time. After removing ingress-nginx, delete any leftover class if it is no longer wanted:
kubectl get ingressclass
kubectl delete ingressclass nginx   # only if you no longer need the nginx compat class

Edit Traefik service

If nginx-ingress used specific NodePorts that you need to reuse, update the Traefik Service:

kubectl edit svc traefik -n traefik

Note: Default NodePorts (for example, 31080 / 31443) might not be reachable in your environment. If these ports are not accessible, determine the NodePorts previously used by nginx-ingress (for example, 30080 / 30443) and configure Traefik to use the same ports.

Update the ports:

Adjust the ports section to match the previous nginx NodePorts if required:

ports:
- name: web
  nodePort: 30080
  port: 80
  targetPort: web
- name: websecure
  nodePort: 30443
  port: 443
  targetPort: websecure

Verify Port Change

kubectl get svc -n traefik

Note: Ensure that the nginx Service is removed or that its NodePorts are freed before reusing those NodePorts on the Traefik Service.

KubeOpsDoku – How to Guides

Kubeopsctl-2.0: Documentation for production

Kubeopsctl-2.0: Joining a Node in a Kubernetes cluster

Join Node Process:

Prerequisits

KOSI Login Recommendation

ETCD Backup Recommendation

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

Example 1: Joining a Worker Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

Kubeopsctl-2.0: Deleting a Node from a Kubernetes cluster

Deleting a Node from a Kubernetes cluster

Prerequisites

Worker

Scale down the rook-ceph-operator deployment to 0

Check which hosts and OSDs belong to each zone

Scale down the OSD deployment

Remove the OSD via ceph-tools

Check the MON running on the node

Delete OSD and MON deployments

Remove the deleted mon from the ceph tools

Delete the node from the kubernetes cluster

Scale the rook-ceph-operator deployment back to 1

Timing and health checks

Kubeopsctl-2.0: Single Sign-On with Keycloak

Kubeopsctl-2.0: Upgrading a Kubernetes cluster

Kubernetes Version Upgrade Process:

Prerequisites

KOSI Login Recommendation

1. Pull required KOSI packages on your ADMIN

2. Change your target version inside the cluster-values

3. Start the upgrade

Example 1 - Upgrade all nodes in the cluster to a specific version

1. Pull required KOSI packages on your ADMIN

2. Update the target version in cluster-values.yaml

2. Validate your values and upgrade the cluster

Example 2 - Tranche upgrade of zones to a specific version

1. Pull required KOSI packages on your ADMIN

2. Adjust your cluster-values in zone1

3. Validate your values and upgrade the cluster

4. Adjust your cluster-values.yaml in zone2

5. Validate your values and upgrade the cluster

6. Adjust your cluster-values.yaml in zone3

7. Validate your values and upgrade the cluster

Kubeopsctl-2.0: Installing KubeOps Compliance applications

Prerequisits

KOSI Login Recommendation

Example 1: Installing Applications in a non-airgap-environment

Example 2: Installing Applications in an airgap-environment

Kubeopsctl-2.0: Updating KubeOps Compliance applications

Prerequisites

KOSI Login Recommendation

Updated KubeOpsctl

Example 1: Updating Applications in a non-airgap-environment

Example 2: Updating Applications in an airgap-environment

Kubeopsctl-2.0: Harbor Deployment with CloudNativePG

Harbor Deployment with CloudNativePG in Non-airgap enviroment

Prerequisites

Step 1 — Install CloudNativePG operator

Step 2 — Create PostgreSQL cluster for Harbor

1. Apply the following Cluster manifest to create a Postgres cluster with 2 instances and 1Gi storage:

2. Services and pods created for the cluster cloudnative-pg:

3. Verify pods are Running:

Step 3 — Retrieve application user credentials

1. Verify the Secret exists:

2. Decode the base64-encoded fields:

Step 4 — Update Harbor tools.yaml for an external database

Step 5 — Install Harbor with Kosi

1. Deploy Harbor using the updated tools.yaml:

2. Verify Harbor pods:

3. Access Harbor at: <your_domain_name>:30002 (or as configured)

Harbor Deployment with CloudNativePG in Airgap envoirnment

Prerequisites

Step 1: Login into Harbor

Step 2: Manually Mirror the PostgreSQL Image to the KubeOps Project

Step 3: Create the PostgreSQL Cluster for Harbor

Step 4: Configure the Cluster to Use the Mirrored Image

1.List the cluster:

2.Edit the cluster:

2. Update the target version in `cluster-values.yaml`

2. Adjust your cluster-values in `zone1`

4. Adjust your `cluster-values.yaml` in `zone2`

6. Adjust your `cluster-values.yaml` `in zone3`

2. Services and pods created for the cluster `cloudnative-pg`:

Recommended: Change the `admin` password via the package value

Advanced: Change any user’s password manually via `securityadmin`