KubeOpsDoku – guide

Kubeopsctl-2.0: Joining a Node in a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

To increase performance or add additional resource capacity to your cluster, adding a node to the cluster is the correct process. This process with kubeopsctl is very easy.
You can use the following steps to join control-plane nodes or worker-nodes to a Kubernetes cluster.

Join Node Process:

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

ETCD Backup Recommendation

Before performing changes on the control planes, it is recommended to create an ETCD backup. Refer to the official Kubernetes documentation for details here

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull a specific Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-controlplaneXX
    iPAddress: 10.2.10.XXX
    type: controlplane
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone1
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone1.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6       # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true             # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-controlplaneXX   # -> has to be changed
    iPAddress: 10.2.10.XXX      # -> has to be changed
    type: controlplane
    kubeVersion: 1.31.6         # -> check with actual version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. If you are ready just start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

Example 1: Joining a Worker Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion x.xx.x you can pull other Kubernetes versions.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-workerXX
    iPAddress: 10.2.10.XX
    type: worker
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone2
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone2.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6        # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true              # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
  - name: demo-workerXX          # -> has to be changed
    iPAddress: 10.2.10.XX        # -> has to be changed
    type: worker
    kubeVersion: 1.31.6          # -> check with actual version     
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. Before joining the node, note the following:

Note

It is recommended if you join worker nodes to a cluster and rook-ceph is installed, to adjust your rook-ceph CRUSH map. Refer to the official rook-ceph documentation for details here. for updating the crush map, you should set in your enterprise-values.yaml:

...
packages:
- name: rook-ceph
  enabled: true
...

If you are ready, start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

If you need to apply the CRUSH map adjustment from above, use the following command instead:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Deleting a Node from a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Deleting a Node from a Kubernetes cluster

Note This guide requires a Compliance enterprise license.

In rare cases, it may be necessary to remove nodes from a Kubernetes cluster. This how-to guide explains the prerequisites and the key considerations to keep in mind before starting the node removal process.

You can use the following steps to delete nodes from a Kubernetes cluster.

Prerequisites

In order to run rook-ceph stable for a longer period your cluster needs at least 3 zones with each zone containing at least 1 worker-node
To check which mon and osd is running on the node you want to delete you can use the command kubectl get pods -n rook-ceph -o wide | grep <node-name> | grep "mon\|osd" | grep -v "osd-prepare" | awk '{print $1}'. As an output you get the mon and the osd running on that node. If you don’t get an output, you don’t have to delete the ressource and can skip to the “delete the node”-section

Worker

Important: Due to rook-ceph, a worker node must not be removed without following the steps below. In this example, worker01 (zone1) is removed from the cluster. Worker01 contains osd.0 and mon-c.

Scale down the rook-ceph-operator deployment to 0

This prevents new MONs or OSDs from being created.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=0

Check which hosts and OSDs belong to each zone

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

ID   CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
 -1         0.21478  root default
 -9         0.04880      zone zone1
 -7         0.04880          host worker01                              # worker01 is being removed
  0    ssd  0.04880              osd.0          up   1.00000  1.00000   # osd.0 is being removed
-15         0.04880          host worker04
  3    ssd  0.04880              osd.3          up   1.00000  1.00000
-11         0.10739      zone zone2
 -3         0.05859          host worker02
  1    ssd  0.05859              osd.1          up   0.95001  1.00000
-13         0.05859      zone zone3
 -5         0.05859          host worker03
  2    ssd  0.05859              osd.2          up   0.95001  1.00000

From this output you can see that osd.0 is part of worker01.

Scale down the OSD deployment

kubectl scale deploy -n rook-ceph rook-ceph-osd-<x> --replicas=0
# Example: kubectl scale deploy -n rook-ceph rook-ceph-osd-0 --replicas=0

Remove the OSD via ceph-tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- bash
# show OSD tree
ceph osd tree
# mark OSD out
ceph osd out <x>
# Example: ceph osd out 0
ceph osd purge <x> --yes-i-really-mean-it
# Example: ceph osd purge 0 --yes-i-really-mean-it
ceph auth del osd.<x>
# adjust CRUSH map
ceph osd crush remove <nodename>
# exit from ceph-tools
exit
# show OSD tree (now without the deleted node)
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

Check the MON running on the node

Before deleting a MON deployment, check which MON is running on the node you want to remove.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon -o wide

Delete OSD and MON deployments

After checking the OSD and MON running on the node, delete the corresponding deployments.

kubectl delete deploy -n rook-ceph rook-ceph-osd-<x> rook-ceph-mon-<y>

Example

kubectl delete deploy -n rook-ceph rook-ceph-osd-0 rook-ceph-mon-c

Remove the deleted mon from the ceph tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm <y>
# verify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

Example

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm c
# verfify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

This is the dump before executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b
2: [v2:192.168.196.110:3300/0,v1:192.168.196.110:6789/0] mon.c

This is the dump after executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b

Delete the node from the kubernetes cluster

Remove the node you want to delete from the cluster-values.yaml file.
After updating the file, apply the deletion with the following command:

kubeopsctl apply --delete -f cluster-values.yaml

Example

The cluster-values.yaml without node1 but with node4

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6      
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker04
    iPAddress: 10.2.10.214
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6  
  - name: worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

After updating the cluster-values.yaml, execute the delete command:

kubeopsctl apply --delete -f cluster-values.yaml

Scale the rook-ceph-operator deployment back to 1

This allows a new MON to be created automatically in zone2.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=1

Timing and health checks

The total duration depends on cluster size and node performance. Before proceeding, verify Ceph health and placement groups are clean.

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph status
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph pg stat

Typical duration ranges from 15 to 120 minutes.

If you want to rejoin the same node, reset it to a time prior to joining the cluster. Only this way you can be sure, that no leftovers from the deletion process remain!

Kubeopsctl-2.0: Upgrading a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

Upgrading a Kubernetes cluster is essential to maintain security, stability, and compatibility.Like Kubernetes itself, we adhere to the version skew policy and only allow upgrades between releases that differ by a single minor version. This ensures compatibility between components, reduces the risk of instability, and keeps the cluster in a supported and secure state.

For more information about the Version Skew Policy, see the official Kubernetes documentation:
Click here

Use the following steps to upgrade the Kubernetes version of a cluster.

Kubernetes Version Upgrade Process:

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull an older Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull --kubernetesVersion <x.xx.x>

2. Change your target version inside the cluster-values

Note The actual version is the current Kubernetes version in your cluster. The target version in the nodes, is version we want to do the upgrade (e.g., see this yaml file)

Note To upgrade your cluster, the parameter changeCluster must be set to true (e.g., see this yaml file)

3. Start the upgrade

kubeopsctl apply -f cluster-values.yaml

Important It is important to start the upgrade with the initial controlplane node. It is the first node with which the cluster was created.

Example 1 - Upgrade all nodes in the cluster to a specific version

In this example, a cluster is upgraded from Kubernetes version v1.33.5 to v1.34.1.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.34.1 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.34.1

2. Update the target version in `cluster-values.yaml`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version in the kubeVersion field of each node.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.33.5     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true           # -> important! Needs to be set for an upgrade
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.34.1       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.34.1       # -> target version

2. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully.
When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

Example 2 - Tranche upgrade of zones to a specific version

In this example, the cluster is upgraded in tranches. The order is:

zone1 (because it contains the initial control plane node),
then zone3,
and finally zone2.

Important It is important to start your tranche upgrade with zone1, because zone1 contains the initial control plane node.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.33.5 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.33.5

2. Adjust your cluster-values in `zone1`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version only for the nodes in zone1. In the snippet below, only zone1 is configured with the target version.

Note Pay close attention to the format and syntax in cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

3. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

4. Adjust your `cluster-values.yaml` in `zone2`

Next, change the target version of zone2:

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

5. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

6. Adjust your `cluster-values.yaml` `in zone3`

Finally, change the target version of zone 3.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.33.5       # -> target version

7. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Kubeopsctl-2.0: Installing KubeOps Compliance applications

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be individually configured as needed.

By separating the cluster values from the application values, the application values can be modified independently and installed at a later stage, providing greater flexibility and maintainability.

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

⚠ Warning To ensure proper DNS resolution, all components of the logging stack — Filebeat, Logstash, OpenSearch, and OpenSearch Dashboards—must be deployed within the same Kubernetes namespace (or network domain).

Example 1: Installing Applications in a non-airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. For more information about available packages as well as parameters for each package check here.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2      
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and install the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the installation process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Installing Applications in an airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                        # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2      
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Updating KubeOps Compliance applications

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be configured individually as needed.

By separating cluster values from application values, application values can be modified independently and installed later, providing greater flexibility and maintainability.

kubeopsctl automatically detects whether an application is already deployed and updates it accordingly. de]

Important If you have made changes to your deployments, reapply them afterward or secure them using the advancedValues configuration.

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

Updated KubeOpsctl

If you have an older kubeopsctl version installed, update it before starting with updating Compliance appliactions.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

Example 1: Updating Applications in a non-airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application update process
Important for only update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and update the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the update process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Updating Applications in an airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Update kubeopsctl

If you have an older kubeopsctl version installed, update it using the following commands.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application update process
Important for only the update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                          # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Harbor Deployment with CloudNativePG

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to deploy Harbor on Kubernetes using a CloudNativePG (CNPG) PostgreSQL cluster managed by the CloudNativePG operator in an air‑gap and non-airgap environments.

Harbor Deployment with CloudNativePG in Non-airgap enviroment

Prerequisites

A running Kubernetes cluster.
kubectl, kosi, and kubeopsctl installed and configured.
You need to login with kosi . Refer to the official KOSI documentation for details here.

Step 1 — Install CloudNativePG operator

Deploy the operator with Kosi:

kosi install --hub kubeops kubeops/cloudnative-pg-operator:<kubeopsctl_version> --dname cnpg-operator

With this step, it Installs the CloudNativePG operator into the cluster and the operator manages PostgreSQL clusters and their lifecycle.

Step 2 — Create PostgreSQL cluster for Harbor

1. Apply the following Cluster manifest to create a Postgres cluster with 2 instances and 1Gi storage:

cat <<EOF | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloudnative-pg
  namespace: harbor
spec:
  instances: 2
  imagePullSecrets:
  - name: registry-pullsecret
  storage:
    size: 1Gi
EOF

2. Services and pods created for the cluster `cloudnative-pg`:

cloudnative-pg-rw → primary (read/write) cloudnative-pg-ro → replicas (read-only) cloudnative-pg-r → all pods

3. Verify pods are Running:

kubectl get pods -n harbor

Step 3 — Retrieve application user credentials

CNPG automatically creates a Secret named cloudnative-pg-app in the harbor namespace.

1. Verify the Secret exists:

kubectl get secret cloudnative-pg-app -n harbor

2. Decode the base64-encoded fields:

kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.username}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.password}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.dbname}" | base64 -d

Example values (for illustration only):

username: app
password: Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2
database: app

Important:
Use the non-superuser application credentials from this Secret in Harbor’s configuration.

Step 4 — Update Harbor tools.yaml for an external database

Edit your tools.yaml and set Harbor values under the helm chart configuration.
Example snippet:

- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "<DB_PASSWORD>"
      redisPassword: "Redis_Password"
      externalURL: <your_domain_name>
      nodePort: 30002
      hostname: <your_domain_name>
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
      database:
        type: external
        external:
          host: "cloudnative-pg-rw.harbor.svc.cluster.local"
          port: "5432"
          username: "app"
          password: "Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2"
          coreDatabase: "app"

Important: Use the -rw service host (cloudnative-pg-rw…) for write operations.
Do not use a superuser account.
Ensure the password matches the CNPG Secret.

Step 5 — Install Harbor with Kosi

1. Deploy Harbor using the updated tools.yaml:

kosi install --hub kubeops kubeops/harbor:2.0.3 -f tools.yaml --dname harbor

2. Verify Harbor pods:

kubectl get pods -n harbor

3. Access Harbor at: <your_domain_name>:30002 (or as configured)

Harbor Deployment with CloudNativePG in Airgap envoirnment

Prerequisites

A running Kubernetes cluster.
podman, kubectl, kosi, and kubeopsctl installed and configured.
You need to login with kosi . Refer to the official KOSI documentation for details here.

First, log in to your Harbor registry and pull the CloudNativePG operator package using kosi.

podman login <ip_address>:<nodePort> -u <user_name> -p <password> --tls-verify=false   
kosi pull --hub kubeops kubeops/cloudnative-pg-operator:<kubeopsctl_version> -o cloudnative-pg.tgz -r <ip_address>:<NodePort>/kubeops -t localhost:<NodePort>/kubeops

kosi install -p cloudnative-pg.tgz --dname cld-pg

Step 2: Manually Mirror the PostgreSQL Image to the KubeOps Project

In an air‑gapped environment, you must manually pull the PostgreSQL image from the external registry and push it to the KubeOps project in Harbor.

podman pull ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie

podman tag ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie <ip_address>:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie

podman push <ip_address>:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie --tls-verify=false

Step 3: Create the PostgreSQL Cluster for Harbor

Create a CNPG PostgreSQL cluster with two instances and 1Gi of storage in the harbor namespace:

cat <<EOF | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloudnative-pg
  namespace: harbor
spec:
  instances: 2
  storage:
    size: 1Gi
EOF

Step 4: Configure the Cluster to Use the Mirrored Image

Update the CNPG cluster to use the image you pushed to Harbor in Step 2.

1.List the cluster:

kubectl get cluster -n harbor

2.Edit the cluster:

kubectl edit cluster -n harbor cloudnative-pg

3.Replace the image name in the editor:

From:

ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie

To:

localhost:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie

Step 5: Retrieve CNPG Application User Credentials

CNPG automatically creates a Secret named cloudnative-pg-app in the harbor namespace.

1. Verify the Secret exists:

kubectl get secret cloudnative-pg-app -n harbor

2. Decode the base64-encoded fields:

kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.username}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.password}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.dbname}" | base64 -d

Example values (for illustration only):

username: app
password: Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2
database: app

Important:
Use the non-superuser application credentials from this Secret in Harbor’s configuration.

Step 6: Configure Harbor in enterprise.yaml

Update your enterprise-values.yaml to configure Harbor with the external CNPG database.
Example snippet:

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
  - name: harbor
    enabled: true
    values:
      standard:
        namespace: harbor
        harborpass: "password"
        databasePassword: "<DB_PASSWORD>"
        redisPassword: "Redis_Password"
        externalURL: <your_domain_name>
        nodePort: 30002
        hostname: <your_domain_name>
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi
              storageClass: "rook-cephfs"
            jobservice:
              jobLog:
                size: 1Gi
                storageClass: "rook-cephfs"
            database:
              size: 1Gi
              storageClass: "rook-cephfs"
            redis:
              size: 1Gi
              storageClass: "rook-cephfs"
            trivy: 
              size: 5Gi
              storageClass: "rook-cephfs"
      advanced:
        database:
          type: external
          external:
            host: "cloudnative-pg-rw.harbor.svc.cluster.local"
            port: "5432"
            username: "app"
            password: "Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2"
            coreDatabase: "app"

Important: Use the -rw service host (cloudnative-pg-rw…) for write operations.
Do not use a superuser account.
Ensure the password matches the CNPG Secret.

Apply the updated configuration:

kubeopsctl apply -f enterprise-values.yaml

Step 6 — Verify the Harbor Deployment

Check that Harbor pods are running in the harbor namespace:

kubectl get pods -n harbor

All Harbor pods should eventually reach a Running or Ready state

General Notes and best practices

Always decode the CNPG app Secret to obtain the correct username, password, and database name used by Harbor.
Use the cloudnative-pg-rw service for write traffic; use -ro services for reads if required.
Production recommendations (follow your organizational security policies):
Create a dedicated DB user (for example, harbor) rather than reusing the default app user.
Enable TLS for Postgres connections.
Implement backups (for example, CNPG Barman or object storage like S3).
To scale the CNPG cluster, update the instances field in the Cluster spec and reapply.

Kubeopsctl-2.0: Ingress Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Manual configuration of the Nginx-Ingress-Controller

Right now the Ingress Controller Package is not fully configured. To make complete use of the Ingress capabilities of the cluster, the user needs to manually update some of the settings of the corresponding service.

Choosing an ingress controller. KubeOps ships two ingress controllers: ingress-nginx and traefik. NGINX has reached end of support, so new clusters should prefer Traefik and existing clusters should plan a migration. To replace nginx-ingress with Traefik (including how SSO, certificates and dashboards keep working), follow How to migrate from nginx to traefik ingress. The rest of this page covers the manual nginx service configuration.

Locating the service

The service in question is called “ingress-nginx-controller” and can be found in the same namespace as the ingress package itself. To locate the service across all namespaces, you could use the following command.

kubectl get service -A | grep ingress-nginx-controller

This command should return two entries of services, “ingress-nginx-controller” and “ingress-nginx-controller-admission”, though only the first one needs to be further adjusted.

Setting the Ingress-Controller service to type NodePort

To edit the service, you can use the following command, although the actual namespace may be different. This will change the service type to NodePort.

kubectl patch service ingress-nginx-controller -n ingress -p '{"spec":{"type":"NodePort"}}'

Kubernetes will now automatically assign unused portnumbers for the nodePort to allow http and https connections to the service. These can be retrieved by running the same command, used to locate the service. Alternatively, you can use the following command, which adds the portnumbers 30080 and 30443 for the respective protocols. By doing so, you have to make sure, that these portnumbers are not being used by any other NodePort service.

kubectl patch service ingress-nginx-controller -n ingress --type=json -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}, {"op":"add","path":"/spec/ports/0/nodePort","value":30080}, {"op":"add","path":"/spec/ports/1/nodePort","value":30443}]'

Configuring external IPs

If you have access to external IPs that route to one or more cluster nodes, you can expose your Kubernetes-Services of any type through these addresses. The command below shows how to add an external IP-Adress to the service with the example value of “192.168.0.1”. Keep in mind that this value has to be changed in order to fit your networking settings.

kubectl patch service ingress-nginx-controller -n ingress -p '{"spec":{"externalIPs":["192.168.0.1"]}}'

Kubeopsctl-2.0: Accessing Dashboards installed with KubeOps

Mon, 01 Jan 0001 00:00:00 +0000

To access an application dashboard, an SSH tunnel to one of the control planes is needed. The following dashboards are available and configured with the following NodePorts by default:

⚠ Warning Change default credentials before going to production.

Note The NodePorts and hostnames shown on this page reflect the default values. Both can be configured in the enterprise-values.yaml. See Fileformats for details.

NodePort

32090 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

To establish a tunnel, forward the NodePort from one of the control planes to your local machine. You can use the command line, PuTTY, or MobaXterm for this.
After that, the dashboard can be accessed at localhost:32090/prometheus/.

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/prometheus/.

No credentials are necessary for login.

NodePort

30211 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/grafana/.

username: the username set in the enterprise-values.yaml of Prometheus (default: user)
password: the password set in the enterprise-values.yaml of Prometheus (default: password)

NodePort

30050 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/opensearch/.

OpenSearch Dashboards supports Single Sign-On (SSO) via Keycloak as the primary login, with username/password as a fallback.

SSO (recommended): open https://<your-domain>/opensearch/ and choose Log in with single sign-on. You are redirected to Keycloak. See the SSO for OpenSearch guide for setup and how to grant users access.
Username / password (fallback):
- username: admin
- password: the value of adminPassword in the enterprise-values.yaml (default: Password@@123456)

Note

SSO is only wired up when the OpenSearch hostname points to a reachable Keycloak (non-.local). For *.local hostnames only the username/password login is available. See Changing the OpenSearch password to change admin.

NodePort

HTTP: 30002
HTTPS: 30003

Connecting via SSH Tunnel

To establish a tunnel, forward port 30002 from one of the control planes to your local machine. You can use the command line, PuTTY, or MobaXterm for this.
After that, the dashboard can be accessed at localhost:30002/.

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/.

username: admin
password: the password set in the enterprise-values.yaml (default: password)

NodePort

The Rook/Ceph dashboard has no fixed NodePort. To find the NodePort in use, run one of the following:

kubectl get svc -n rook-ceph

Look for the rook-ceph-mgr-dashboard-external-http service:

NAME                                      TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                                     AGE
rook-ceph-mgr-dashboard-external-http     NodePort    192.168.197.13    <none>        7000:31268/TCP                              21h

Or retrieve the NodePort directly:

echo $(kubectl get -n rook-ceph -o jsonpath="{.spec.ports[0].nodePort}" services rook-ceph-mgr-dashboard-external-http)

In the example above, the NodePort is 31268.

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/ceph-dashboard/.

echo Username: admin
echo Password: $(kubectl get secret rook-ceph-dashboard-password -n rook-ceph --template={{.data.password}} | base64 -d)

NodePort

30007 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/kubeops-dashboard/.

kubectl -n monitoring create token headlamp-admin

NodePort

30180

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/keycloak/.

echo Username: $(kubectl get secret --namespace keycloak keycloak-kubeops -o jsonpath="{.data.ADMIN_USER}" | base64 -d)
echo Password: $(kubectl get secret --namespace keycloak keycloak-kubeops -o jsonpath="{.data.ADMIN_PASSWORD}" | base64 -d)

Kubeopsctl-2.0: Accessing Dashboards with OIDC

Mon, 01 Jan 0001 00:00:00 +0000

This how-to guide explains how to enable OIDC-based access to your Kubernetes application dashboards using kubeopsctl. You will configure a single enterprise-values.yaml file that defines the required components. Once the configuration is applied, the dashboards for Harbor, Grafana, Rook-Ceph, OpenSearch, Keycloak, the KubeOps dashboard, and Prometheus become accessible via their configured hostnames and paths. Following the steps in this guide, you will set up a consistent, centralized OIDC integration and make your dashboards securely available through a browser.

Prerequisites

yq should be installed on all master nodes. For more infomration refer yq installation guide.
Keycloak should be installed before other packages.
Pull all the required packages with ‘kubectl pull’ command. For more information refer Pull Packages with Kubeopsctl

Step 1: Configure enterprise-values.yaml

In order to connect to dashboad with OIDC, you need to configure your enterprise-values.yaml file as below

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: rook-ceph
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
            mon:
              requests:
                cpu: "1"
                memory: "1Gi"
            osd:
              requests:
                cpu: "1"
                memory: "1Gi"
          dashboard:
            enabled: "true"
        operator:
          data:
            rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: <domain_name>          # required for OIDC dashboard access
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: cert-manager 
  enabled: true
  values:
    standard:
      namespace: cert-manager
      replicaCount: 3
      logLevel: 2
    advanced:
    ca:      
      emailLetsEncrypt: <email>
      ingressName: nginx
- name: ingress-nginx
  enabled: true
  values:
    standard:
      namespace: ingress-nginx
      externalIPs: 
        - <>                 
      advanced:
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: "keycloak"
      storageClass: "rook-cephfs"
      nodePort: "30180"
      hostname: <domain_name>          # required for OIDC dashboard access
      keycloak:
        auth:
          adminUser: admin
          adminPassword: topsecret
      postgresql:
        auth:
          postgresUserPassword: "changeme"
          username: bn_keycloak
          password: "changeme"
          database: bitnami_keycloak
        volumeSize: "8Gi"
    advanced:
- name: filebeat-os
  enabled: true
  values:
    standard:
      namespace: logging
    advanced:
- name: logstash-os
  enabled: true
  values:
    standard:
      namespace: logging
      volumeClaimTemplate:
        accessModes: 
          - ReadWriteMany
        resources:
          requests:
            storage: 1Gi
        storageClass: "rook-cephfs"
    advanced:
- name: opensearch-dashboards
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access 
      namespace: logging
      nodePort: 30050
    advanced:
- name: opensearch-os
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access
      namespace: logging
      opensearchJavaOpts: "-Xmx512M -Xms512M" # optional, default is -Xmx512M -Xms512M
      resources:
        requests:
          cpu: "250m" # optional, default is 250m
          memory: "1024Mi" # optional, default is 1024Mi
        limits:
           cpu: "300m" # optional, default is 300m
           memory: "3072Mi" # optional, default is 3072Mi
      persistence:
        size: 4Gi # mandatory
        enabled: "true" # optional, default is true
        enableInitChown: "false" # optional, default is false
        labels:
          enabled: "false" # optional, default is false
        storageClass: "rook-cephfs" # optional, default is rook-cephfs
        accessModes:
          - "ReadWriteMany" # optional, default is {ReadWriteMany}
        securityConfig:
          enabled: "false" # optional, default value: false
        replicas: "3" # optional, default is 3
    advanced:
- name: kube-prometheus-stack
  enabled: true
  values:
    standard:
      namespace: monitoring
      grafanaUsername: admin
      grafanaPassword: topsecret
      retentionSize: "24GB"
      grafanaResources:
        hostname: <domain_name>          # required for OIDC dashboard access
        nodePort: 30211
        retention: 10d
        retentionSize: "24GB"
        storageClass: "rook-cephfs"
        storage: 25Gi
      prometheusResources:
        hostname: <domain_name>          # required for OIDC dashboard access
        retentionSize: "24GB"
        storageClass: "rook-cephfs"
        storage: 25Gi
    advanced:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: opa-gatekeeper
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: <domain_name>          # required for OIDC dashboard access
      service:
        nodePort: 30007
    advanced:
- name: velero
  enabled: true
  values:
    standard:
      namespace: "velero"
      accessKeyId: "your_s3_storage_username"
      secretAccessKey: "your_s3_storage_password"
      useNodeAgent: false
      defaultVolumesToFsBackup: false
      provider: "aws"
      bucket: "velero"
      useVolumeSnapshots: false
      backupLocationConfig:
        region: "minio"
        s3ForcePathStyle: true
        s3Url: "http://minio.velero.svc:9000"
    advanced:

step 2: Apply the Configuration with ‘KubeOpctl’

Apply the enterprise-values.yaml configuration using the following command:

kubeopsctl apply -f enterprise-values.yaml

Wait until all components are deployed and running before proceeding.

Step 3: Connect to Dashboards via Hostname

To access the dashboards via OIDC, use the hostnames configured in your enterprise-values.yaml (uservalues). Harbor is an exception: you can access it directly using the base domain name.

Use the following paths, replacing <domain_name> with your configured domain:

<domain_name>/ for Harbor
<domain_name>/grafana for grafana
<domain_name>/ceph-dashboard for rook-ceph
<domain_name>/opensearch for opensearch
<domain_name>/keycloak for keycloak
<domain_name>/kubeops-dashboard for Kubeops Dashboard
<domain_name>/prometheus for Prometheus

Kubeopsctl-2.0: Changing a User Password in OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains how to change passwords in OpenSearch.

Recommended: Change the `admin` password via the package value

The OpenSearch admin password is managed through the adminPassword package value. Set it for the opensearch-os package (and the matching opensearch-dashboards package) in your enterprise-values.yaml and run an update:

packages:
  - name: opensearch-os
    enabled: true
    values:
      standard:
        adminPassword: "<your-new-password>" # must match opensearch-dashboards
  - name: opensearch-dashboards
    enabled: true
    values:
      standard:
        adminPassword: "<your-new-password>" # must match opensearch-os

On the next kubeopsctl apply/update the new password is enforced automatically after the deployment. This also works on existing clusters and does not remove any other internal users.

Note Internal users created in the OpenSearch Dashboards Security UI persist across package updates. The update only enforces the admin password and reapplies the authentication/authorization configuration; it never overwrites your internal users.

Advanced: Change any user’s password manually via `securityadmin`

Use this method to change the password of any internal user (for example testuser or kibanaro) without a package update.

This process reads the live internal users from the security index (via securityadmin -backup), so every other user — including users you created in the OpenSearch Dashboards Security UI — is preserved. Do not build the file from the internal-users-config-secret: that secret only holds the packaged baseline users and would remove UI-created users when applied.

Prerequisites

Access to the Kubernetes cluster and kubectl.
The admin certificate is available in the pod at /usr/share/opensearch/config/certs/ (root-ca.pem, admin.pem, admin-key.pem) in a KubeOps deployment.

Set a shell variable for convenience (adjust the namespace and pod name as needed):

NS=logging; POD=opensearch-cluster-master-0

Step 1: Back up the current internal users (live state)

kubectl exec -it $POD -n $NS -- bash -c "\
    /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -backup /tmp/os-sec \
    -icl -nhnv \
    -cacert /usr/share/opensearch/config/certs/root-ca.pem \
    -cert /usr/share/opensearch/config/certs/admin.pem \
    -key /usr/share/opensearch/config/certs/admin-key.pem"

This writes the current configuration — including all internal users — to /tmp/os-sec/ in the pod.

Note securityadmin -backup may end with Configuration for 'audit' failed because of empty source and a non-zero exit code. This is harmless: internal_users.yml is written before that step, so the file you need is already present.

Step 2: Generate a new password hash for each user

Run this once per user and note the resulting hash:

kubectl exec -it $POD -n $NS -- bash -c "sh /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p <new_password>"

Step 3: Update the hashes in the backup file

Copy the file out, edit it locally, and copy it back:

kubectl cp $NS/$POD:/tmp/os-sec/internal_users.yml ./internal_users.yml
# edit ./internal_users.yml: replace the hash: value under each user you want to change
kubectl cp ./internal_users.yml $NS/$POD:/tmp/os-sec/internal_users.yml

Only change the hash: line of the target users; leave everyone else untouched:

testuser:
  hash: "<new-hash-for-testuser>"
  reserved: false
  backend_roles:
  - "testrole"
kibanaro:
  hash: "<new-hash-for-kibanaro>"
  reserved: false
  backend_roles:
  - "kibanauser"

Step 4: Apply only the internal users configuration

Note

The DN of the certificate specified with -cert must be listed under plugins.security.authcz.admin_dn in opensearch.yml. In a KubeOps deployment the admin certificate subject is C=DE,L=test,O=client,OU=client,CN=admin.

More information: Opensearch Documentation.

Apply only the internalusers config type. Do not use -cd (the whole directory) here: -cd requires the complete security config set (including nodes_dn.yml, allowlist.yml, audit.yml), which a KubeOps deployment does not ship, so it would fail with nodes_dn.yml ... No such file or directory.

kubectl exec -it $POD -n $NS -- bash -c "\
    /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -f /tmp/os-sec/internal_users.yml -t internalusers \
    -icl -nhnv \
    -cacert /usr/share/opensearch/config/certs/root-ca.pem \
    -cert /usr/share/opensearch/config/certs/admin.pem \
    -key /usr/share/opensearch/config/certs/admin-key.pem"

The change is written to the security index and takes effect immediately — no pod restart required.

Important

Once applied with securityadmin.sh, credentials are stored in the OpenSearch security index, which lives on the persistent volume and therefore survives pod and container restarts as well as package updates. Editing files inside the container without running securityadmin.sh only changes the local file and is not applied to the cluster. For the admin user, prefer the adminPassword package value described above — it is enforced again on every update and keeps all other internal users intact.

Kubeopsctl-2.0: Backup and restoring artifacts

Mon, 01 Jan 0001 00:00:00 +0000

What is Velero?

Velero uses object storage to store backups and associated artifacts. It also optionally integrates supported block storage systems to snapshot your persistent volumes. Before beginning the installation process, you should identify the object storage provider and optional block storage provider(s) you’ll be using from the list of compatible providers.

Velero supports storage providers for both cloud-provider environments and on-premises environments.

Velero prerequisites:

Access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled.
kubectl installed locally
Object Storage (S3, Cloud Provider Environment, On-Premises Environment)

Note : Compatible providers and on-premises documentation can be read on https://velero.io/docs

Install Velero

This command is an example on how you can install velero into your cluster:

velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.2.1 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000

NOTE:

s3Url has to be the url of your s3 storage login.

example for credentials-velero file:

[default]
aws_access_key_id = your_s3_storage_username
aws_secret_access_key = your_s3_storage_password

Backup the cluster

Scheduled Backups

This command creates a backup for the cluster every 6 hours:

velero schedule create cluster --schedule "0 */6 * * *"

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete cluster

Restore Scheduled Backup

This command restores the backup according to a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the cluster

velero backup create cluster

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Backup a specific deployment

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create filebeat --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “filebeat” every 6 hours:

velero schedule create filebeat --schedule "0 */6 * * *" --include-namespaces logging --selector app=filebeat-filebeat,release=filebeat --include-resources serviceaccount,deployment,daemonset,configmap,clusterrolebinding,clusterrole --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete filebeat

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create filebeat --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “filebeat”:

velero backup create filebeat --include-namespaces logging --selector app=filebeat-filebeat,release=filebeat --include-resources serviceaccount,deployment,daemonset,configmap,clusterrolebinding,clusterrole --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create logstash --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “logstash” every 6 hours:

velero schedule create logstash --schedule "0 */6 * * *" --include-namespaces logging --selector app=logstash-logstash,chart=logstash,release=logstash --include-resources StatefulSet,ServiceAccount,Service,Secret,RoleBinding,Role,PodSecurityPolicy,PodDisruptionBudget,Ingress,ConfigMap --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete logstash

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create logstash --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “logstash”:

velero backup create logstash --include-namespaces logging --selector app=logstash-logstash,chart=logstash,release=logstash --include-resources StatefulSet,ServiceAccount,Service,Secret,RoleBinding,Role,PodSecurityPolicy,PodDisruptionBudget,Ingress,ConfigMap --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create opensearch --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “opensearch” every 6 hours:

velero schedule create opensearch --schedule "0 */6 * * *" --include-namespaces logging --selector app.kubernetes.io/instance=opensearch,app.kubernetes.io/name=opensearch --include-resources ConfigMap,Ingress,NetworkPolicy,PodDisruptionBudget,PodSecurityPolicy,Role,RoleBinding,Secret,Service,ServiceAccount,StatefulSet --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete opensearch

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create opensearch --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “opensearch”:

velero backup create opensearch --include-namespaces logging --selector app.kubernetes.io/instance=opensearch,app.kubernetes.io/name=opensearch --include-resources ConfigMap,Ingress,NetworkPolicy,PodDisruptionBudget,PodSecurityPolicy,Role,RoleBinding,Secret,Service,ServiceAccount,StatefulSet --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “monitoring” every 6 hours:

velero schedule create prometheus --schedule "0 */6 * * *" --include-namespaces monitoring --include-cluster-resources=true

This command creates a backup for the deployment “prometheus” every 6 hours:

velero schedule create prometheus --schedule "0 */6 * * *" --include-namespaces monitoring --include-resources Alertmanager,Secret,Ingress,List,PodDisruptionBudget,Role,RoleBinding,PodSecurityPolicy,Service,ServiceAccount,ServiceMonitor,Endpoints,ConfigMap,ConfigMapList,ClusterRole,ClusterRoleBinding,SecretProviderClass,PodMonitor,Prometheus,Job,NetworkPolicy,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Issuer,Deployment,VerticalPodAutoscaler,ThanosRuler --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete prometheus

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “monitoring”:

velero backup create prometheus --include-namespaces monitoring --include-cluster-resources=true

This command creates a backup for the deployment “prometheus”:

velero backup create prometheus --include-namespaces monitoring --include-resources Alertmanager,Secret,Ingress,List,PodDisruptionBudget,Role,RoleBinding,PodSecurityPolicy,Service,ServiceAccount,ServiceMonitor,Endpoints,ConfigMap,ConfigMapList,ClusterRole,ClusterRoleBinding,SecretProviderClass,PodMonitor,Prometheus,Job,NetworkPolicy,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Issuer,Deployment,VerticalPodAutoscaler,ThanosRuler --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “harbor” every 6 hours:

velero schedule create harbor --schedule "0 */6 * * *" --include-namespaces harbor --include-cluster-resources=true

This command creates a backup for the deployment “harbor” every 6 hours:

velero schedule create harbor --schedule "0 */6 * * *" --include-namespaces harbor --include-resources ConfigMap,Deployment,PersistentVolumeClaim,Secret,Service,StatefulSet,Ingress,ServiceMonitor --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete harbor

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “harbor”:

velero backup create harbor --include-namespaces harbor --include-cluster-resources=true

This command creates a backup for the deployment “harbor”:

velero backup create harbor --include-namespaces harbor --include-resources ConfigMap,Deployment,PersistentVolumeClaim,Secret,Service,StatefulSet,Ingress,ServiceMonitor --include-cluster-resources=true --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “gatekeeper-system” every 6 hours:

velero schedule create gatekeeper --schedule "0 */6 * * *" --include-namespaces gatekeeper-system --include-cluster-resources=true

This command creates a backup for the deployment “gatekeeper” every 6 hours:

velero schedule create gatekeeper --schedule "0 */6 * * *" --include-namespaces gatekeeper-system --include-resources PodSecurityPolicy,ServiceAccount,Deployment,PodDisruptionBudget,ResourceQuota,ClusterRole,Role,ClusterRoleBinding,RoleBinding,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Secret,Service,Job --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete gatekeeper

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “gatekeeper-system”:

velero backup create gatekeeper --include-namespaces gatekeeper-system --include-cluster-resources=true

This command creates a backup for the deployment “gatekeeper-system”:

velero backup create gatekeeper --include-namespaces gatekeeper-system --include-resources PodSecurityPolicy,ServiceAccount,Deployment,PodDisruptionBudget,ResourceQuota,ClusterRole,Role,ClusterRoleBinding,RoleBinding,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Secret,Service,Job --include-cluster-resources=true --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “rook-ceph” every 6 hours:

velero schedule create rook-ceph --schedule "0 */6 * * *" --include-namespaces rook-ceph --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete rook-ceph

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “rook-ceph”:

velero backup create rook-ceph --include-namespaces rook-ceph --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

restore databases

keycloak

create backup of keycloak namespace, in this example the backup is called keycloak1.

This command creates a backup for the namespace “rook-ceph”:

velero backup create keycloak1 --include-namespaces keycloak --include-cluster-resources=true

restore backup, in this example keycloak1

velero restore create keycloak1 --from-backup keycloak1

restore the database dump:

kubectl -n <keycloak-namespace> exec keycloak-postgres-0 -- pg_restore -v --jobs=4 --clean --if-exists -d bitnami_keycloak /backup/keycloak-db.dump

Kubeopsctl-2.0: Add certificate as trusted

Mon, 01 Jan 0001 00:00:00 +0000

1. Download the certificate

As soon as Chrome issues a certificate warning, click on Not secure to the left of the address bar.
Show the certificate (Click on Certificate is not valid).
Go to Details tab.
Click Export... at the bottom and save the certificate.

As soon as Firefox issues a certificate warning, click on Advanced....
View the certificate (Click on View Certificate).
Scroll down to Miscellaneous and save the certificate.

2. Install the certificate

Press Windows + R.
Enter mmc and click OK.
Click on File > Add/Remove snap-in....
Select Certificates in the Available snap-ins list and click on Add >, then on OK. Add the snap-in.
In the tree pane, open Certificates - Current user > Trusted Root Certification Authorities, then right-click Certificates and select All tasks > Import....
The Certificate Import Wizard opens here. Click on Next.
Select the previously saved certificate and click Next.
Click Next again in the next window.
Click on Finish. If a warning pops up, click on Yes.
The program can now be closed. Console settings do not need to be saved.
Clear browser cache and restart browser.

The procedures for using a browser to import a certificate as trusted (on Linux systems) vary depending on the browser and Linux distribution used. To manually cause a self-signed certificate to be trusted by a browser on a Linux system:

Distribution	Copy certificate here	Run following command to trust certificate
RedHat	`/etc/pki/ca-trust/source/anchors/`	`update-ca-trust extract`

Note

If the directory does not exist, create it.

Note

If you do not have the ca-certificates package, install it with your package manager.

Kubeopsctl-2.0: Deploy Package on a cluster

Mon, 01 Jan 0001 00:00:00 +0000

You can install artifacts in your cluster in several ways. When creating a package, you can use one of the following plugins:

helm
kubectl
cmd
kosi

This guide demonstrates a complete, reproducible deployment of nginx-ingress using Helm and kosi.

Using the Helm Plugin

Pre-requisites

Before you begin, make sure the following tools are available in your environment:

helm
kosi

This example uses the official NGINX Helm repository and deploys nginx-ingress with Helm and KOSI.

Prepare the Files

In order to install an artifact with the Helm plugin, the Helm chart must first be downloaded.

To keep the environment clean and reproducible, first create a dedicated working directory and change into it:

mkdir nginx-ingress-kosi && cd nginx-ingress-kosi

The example uses the official NGINX Helm repository. Add the repository before performing any Helm operations:

helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update

To verify which chart versions are available, run:

helm search repo nginx-stable/nginx-ingress --versions

This guide uses chart version 0.16.1. Download the chart as a packaged Helm artifact:

helm pull nginx-stable/nginx-ingress --version 0.16.1

This command creates the following file in your working directory:

nginx-ingress-0.16.1.tgz

Create KOSI package

Once the chart is available locally, initialize the kosi package structure. The following command creates the necessary files in the current directory:

kosi create

The Helm chart remains the central input artifact for deployment, while configuration is handled via values.yaml. Helm already includes default values, which can be inspected or exported if needed:

helm show values nginx-stable/nginx-ingress --version 0.16.1 > values.yaml

Review and update values.yaml before packaging and deploying the chart. For example, in environments without a cloud-based LoadBalancer, you may need to change the service type in values.yaml.

controller:
  service:
    type: LoadBalancer

Only required overrides should be defined in the customized values.yaml file. To customize the deployment of the Helm chart, a new values.yaml file in the current directory must be created and edited with this content:

controller:
  service:
    type: NodePort

This override is merged with the chart defaults at runtime.

Helm charts do not directly contain the full list of container images required for deployment. To identify them, render the chart into Kubernetes manifests:

helm template nginx-ingress nginx-stable/nginx-ingress --version 0.16.1 > manifests.yaml

Then extract all referenced images from the rendered manifest:

grep image: manifests.yaml

For this example, the required image is:

nginx/nginx-ingress:3.0.1

With all artifacts identified, the package.kosi file defines what is required for deployment. It explicitly references:

- the Helm chart archive
- the custom values.yaml
- all required container images

nginx-ingress-0.16.1.tgz
values.yaml
package.kosi

Note

The downloaded Helm chart must also be located in the current directory. To customize the deployment of the Helm chart, the values.yaml file must be edited.

All files required by a task in the package must be named in the package.kosi file under files. The container images required by the Helm chart must also be listed in the package.kosi under containers.

In the example below, only two files are required for the installation: the Helm Chart for the nginx-ingress and the values.yaml to configure the deployment. To install nginx-ingress you will also need the nginx/nginx-ingress image with the tag 3.0.1.

To install nginx-ingress with the Helm plugin, call the plugin as shown in the example under install. The deployment configuration file is listed under values and the packed Helm chart is specified with the key tgz. Furthermore, it is also possible to specify the namespace in which the artifact should be deployed and the name of the deployment. The full documentation for the Helm plugin can be found here.

Use the following configuration:

languageversion = "1.0.0";
apiversion = "kubernative/kubeops/sina/user/v4";
name = "deployexample1";
description = "It shows how to deploy an artifact to your cluster using the helm plugin.";
version = "0.1.0";
docs = "docs.tgz";
logo = "logo.png";

files =
{
        valuesFile = "values.yaml";
        nginxHelmChart="nginx-ingress-0.16.1.tgz";
}

containers =
{
        nginx = ["docker.io", "nginx/nginx-ingress", "3.0.1"];
}

install
{
        helm
        (
            command = "install";
            tgz = "nginx-ingress-0.16.1.tgz";
            values = "['values.yaml']";
            namespace = "dev";
            deploymentName = "nginx-ingress"
        );
}

Build the Package

Once the package.kosi file has been fully configured, all files must be combined into a kosi package. To do this, execute the following command in the directory where the package.kosi file is located.

kosi build

Push the Package

To make the generated kosi package available on other machines, it is pushed to the user’s private KubeOps Hub. To do this, the user must first log in to the hub and then push it to hub.

$ kosi login -u <username>
2023-02-04 11:19:43 Info:      KOSI version: 2.13.0
2023-02-04 11:19:43 Info:      Please enter password
****************
2023-02-04 11:19:26 Info:      Login Succeeded to Hub.
$ kosi push --hub kosi
2023-02-04 11:23:18 Info:      KOSI version: 2.13.0
2023-02-04 11:23:19 Info:      Push to Private Registry registry.kubeops.net/<username>/

Deploy the Package

Once the KOSI package has been created and published, it needs to be installed on the admin node. The following command will download and execute the package. The package name and version refer to the values defined in package.kosi with the keys name and version.

kosi install --hub <username> <username>/<packagename>:<version>

For the example package, the command would be: kosi install --hub <username> <username>/deployExample:0.1.0.

Using the Kubectl Plugin

Prerequisites

Before you begin, make sure the following tools and components are available:

kosi
the kubeops-kubernetes-plugins package installed on the admin node

This example deploys nginx-ingress from a Kubernetes manifest using the Kubectl plugin.

Create KOSI package

First you need to create a KOSI package. The following command creates the necessary files in the current directory:

kosi create

The NGINX ingress controller YAML manifest can either be automatically downloaded and applied directly with kubectl apply or it can be downloaded manually if you want to customize the deployment. The YAML manifest can be downloaded from the NGINX GitHub Repo and must be placed in the same directory as the files for the KOSI package.

All files required by a task in the package must be named in the package.kosi file under files. The container images required by the YAML manifest must also be listed in the package.kosi under containers. In the example below, only one file is required for the installation: the YAML manifest for the nginx-ingress controller. To install nginx-ingress you will also need the registry.k8s.io/ingress-nginx/controller image with the tag v1.5.1 and the image registry.k8s.io/ingress-nginx/kube-webhook-certgen with tag v20220916-gd32f8c343.

To install nginx-ingress with the Kubectl plugin, call the plugin as shown in the example under installs. The full documentation for the Kubectl plugin can be found here.

languageversion = "1.0.0";
apiversion = "kubernative/kubeops/sina/user/v4";
name = "deployexample2";
description = "It shows how to deploy an artifact to your cluster using the helm plugin.";
version = "0.1.0";
docs = "docs.tgz";
logo = "logo.png";

files =
{
     manifest: "deploy.yaml"
}

containers =
{
    nginx = ["registry.k8s.io", "ingress-nginx/controller", "v1.5.1"];
    certgen= ["registry.k8s.io","ingress-nginx/kube-webhook-certgen","v20220916-gd32f8c343"];
}

install
{
    kubectl
    (
      operation="apply",
      flags="-f deploy.yaml";
      sudo = true;
      sudoPassword="toor"
    );
}

Build the Package

Once the package.kosi file has been fully configured, all files must be combined into a KOSI package. To do this, execute the following command in the directory where the package.kosi file is located.

kosi build

Publish the Package

To make the generated KOSI package available on other machines, it is pushed to the user’s private KubeOps Hub. To do this, the user must first log in to the hub.

$ kosi login -u <username>
2023-02-04 11:19:43 Info:      kosi version: 2.13.0
2023-02-04 11:19:43 Info:      Please enter password
****************
2023-02-04 11:19:26 Info:      Login Succeeded to Hub.
$ kosi push --hub kosi
2023-02-04 11:23:18 Info:      kosi version: 2.13.0
2023-02-04 11:23:19 Info:      Push to Private Registry registry.kubeops.net/<username>/

Deploy the Package

kosi install --hub <username> <username>/<packagename>:<version>

For the example package, the command would be: kosi install --hub <username> <username>/deployExample:0.1.0.

Kubeopsctl: Set up a Basic Cluster

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

kubeopctl plattform requirements
best practices for machine setup
setup secure headless environment for communication between admin and all masters / workers
how to install required software
how to use the official KubeOps Website to download kubeopsctl
how to create a basic cluster

After the installation, kubeopctl is available as command line interface.

Notice The intent of this guide is to setup an example cluster with 7 machines. We do not recommend this guide for production environments. For more information about kubeopsctl production environments, see LINK.

Prerequisites

To get the most out of this guide, the following requirements should be met:

basic understanding of Linux environments, bash / shell
basic understanding of text editors, vi / nano
administrator privileges (root) are granted

A total of 7 machines (virtual or physical) are required and need to be set up:

one admin - control plane, this machine will manage all tasks on the cluster and integrated machines
three masters
three workers

The final cluster will have the following structure. Masters and workers are added to two clusters zones.

Important The platform setup using this guide can take up to 3 hours. Depending on the resources available.

Step 1 - Minimal Platform Requirements

kubeopsctl is designed to work with the latest versions of the following operating systems.

Notice Make sure all machines use the same operating system.

Supported Operating Systems


Operating system	Red Hat Enterprise Linux (Version 8.2 or newer)

System requirements Admin Nodes


CPU	2x
Memory	2 GB
Diskspace	50 GB
Internet Access	Yes to use default `KubeOps Registry`

Important: Choose the correct KubeOps Registry based on your version:
"registry1.kubernative.net/lima"
"registry.kubeops.net"

System requirements Master Nodes


CPU	4x
Memory	8 GB
Diskspace	50 GB

System requirements Worker Nodes


CPU	8x
Memory	16 GB
Diskspace	50 GB 50 GB unformatted non-partitioned disk storage for `ceph`

Important

By default Red Hat Enterprise Linux restricts package management such as installing or updating. Before installing any software, you may need to register and subscribe with the Red Hat Customer Portal.

To register and subscribe, run the command subscription-manager register.

For more information about rook-ceph, see the prerequisites in its official documentation.

If a node fails, the minimum requirements may not be sufficient to guarantee trouble-free operation.

Step 2 - Set up your Machines

You can setup the admin, master and worker nodes as virtual or as physical machines.

During the setup of the machines, make sure that you meet the following requirements:

heed the platform requirements as mentioned above
all machines need to be synchronized with the current time
all machines need to be within the same network environment

Important

During the setup, note the IP addresses of all machines.

Assign unique hostnames for each machine (lowercase letters and numbers only).

We recommended using self-explanatory hostnames (see example table below).

To get the most out of this guide, use the following hostnames for your basic cluster:

Hostnames

Machine / Purpose	Hostnames
Admin	admin
Masters	master1 master2 master3
Workers	worker1 worker2 worker3

Assigning Hostnames manually

If you need to assign hostnames manually, login to the machine and use the hostnamectl set-hostname command.

hostnamectl set-hostname master1

Repeat this process for all machines where necessary.

Remove firewalld on Red Hat Enterprise Linux 8

Important Repeat this process for all master and worker machines where necessary.

If you are using Red Hat Enterprise Linux 8, you must remove firewalld. Kubeopsctl installs nftables by default.
You can use the following commands to remove firewalld:

systemctl disable --now firewalld
systemctl mask firewalld
dnf remove -y firewalld
reboot

Step 3 - Set up Access for the Admin Machine

The admin machine needs secure and headless access to all other machines.

Set up IP Addresses for DNS

It is recommended, that a DNS service is running. If you do not have a DNS service, you need to edit the /etc/hosts file on the admin machine.

Add the following lines at the end of the /etc/hosts file. Replace the IP addresses with the actual addresses you noted during the setup of all machines. Replace the hostnames with the actual hostnames you assigned during the setup of all machines:

10.2.10.10 admin
10.2.10.11 master1
10.2.10.12 master2
10.2.10.13 master3
10.2.10.14 worker1
10.2.10.15 worker2
10.2.10.16 worker3

Set up Secure Access

To securely access the master and worker machines, you need to create a ssh-key-pair (private and public key) on the admin machine. Afterwards copy the public key onto each machine.

To learn more about ssh and key-pairs see our guide on How to set up SSH keys.

Step 4 - Install Podman on the Admin Machine

To ensure compatibility across different containerized environments, kubeopsctl requires the installation of Podman (latest version).

Notice There can be an issue with conflicts with containerd, so it is recommended that containerd.io is removed before installing the podman package.

Install Podman on the admin machine using the inbuilt package manager.

sudo dnf install -y podman

you need to install jq for rook-ceph:

dnf install -y jq

also you need to login to systemd with your os user:

loginctl enable-linger myuser

Notice

By default, kubeopsctl only supports secure registries.

To setup kubeopsctl with insecure registries, see here.

Step 5 - Install kubeopsctl on the Admin Machine

With everything prepared, the next step is to download and install kubeopsctl on the admin machine.

Downloading KOSI

Download your desired version of the kubeopsctl package file (.rpm) from the official download page onto the admin machine.

Installing KOSI
```
sudo dnf install -y <path>/<kosi_rpm>
```
Login to the kubeops hub using kosi

After you input the password, you will gain access to the kubeops hub. Use the following command to begin the login process:
```
  kosi login -u <user>
```

Installing kubeopsctl

Install kubeopsctl using the inbuilt package manager. Replace <path> and <kubeopsctl_rpm> with the respective path and file name of the kubeopsctl package file.

Notice

On installation, kubeopsctl will perform all necessary steps to setup the environment in dependence of your used operating system. This includes the automatic installation / deinstallation of additional software or packages.

Removes software:

runc

Installs Software:

tc, libnftnl
iproute2, libnftnl

To install kubeopsctl use the following command.

sudo dnf install -y <path>/<kubeopsctl_rpm>

Create Work Folders and Setup Environment Variables

After the setup, you need to create work folders where kubeopsctl can save and manage configurations and other settings.

mkdir -p ~/kubeops
mkdir -p ~/kubeops/lima

To work comfortably, you need to assign these folders to the predefined environment variables KUBEOPSROOT and LIMAROOT.

echo 'export KUBEOPSROOT="${HOME}/kubeops"' >> $HOME/.bashrc
echo 'export LIMAROOT="${HOME}/kubeops/lima"' >> $HOME/.bashrc
source $HOME/.bashrc

Verify your Installation

To verify the installation of kubeopsctl on your system, use the command kubeopsctl version.

kubeopsctl version

Step 6 - Configure the Basic Cluster

With everything ready to start, the next step is to configure the cluster.

For configurations, kubeopsctl uses the YAML format.

Use an editor to create and edit the configuration file:

nano ~/basicCluster.yml

Copy and paste all lines into the file. You need to edit specific parameters according to the assigned IP addresses, hostnames etc.:

master/name - set all master hostnames
master/ipAdress - set all master IP addresses
worker/name - set all worker hostnames
worker/ipAdress - set all worker IP addresses

apiVersion: kubeops/kubeopsctl/alpha/v5 # mandatory
imagePullRegistry: "registry1.kubernative.net/lima"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11
systemCpu: "200m"
systemMemory: "200Mi"

zones:
  - name: zone1
    nodes:
      master:
        - name: master1
          ipAdress: 10.2.10.11
          status: active
          kubeversion: 1.30.0
        - name: master2
          ipAdress: 10.2.10.12
          status: active
          kubeversion: 1.30.0
      worker:
        - name: worker1
          ipAdress: 10.2.10.14
          status: active
          kubeversion: 1.30.0
        - name: worker2
          ipAdress: 10.2.10.15
          status: active
          kubeversion: 1.30.0
  - name: zone2
    nodes:
      master:
        - name: master3
          ipAdress: 10.2.10.13
          status: active
          kubeversion: 1.30.0  
      worker:
        - name: worker3
          ipAdress: 10.2.10.16
          status: active
          kubeversion: 1.30.0


# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
kubeops-dashboard: true
certman: true
ingress: true 
keycloak: true
velero: true

harborValues: 
  harborpass: "password" # change to your desired password
  databasePassword: "Postgres_Password" # change to your desired password
  redisPassword: "Redis_Password" 
  externalURL: http://10.2.10.11:30002 # change to ip adress of master1

prometheusValues:
  grafanaUsername: "user"
  grafanaPassword: "password"

ingressValues:
  externalIPs: []

keycloakValues:
  keycloak:
    auth:
      adminUser: admin
      adminPassword: admin
  postgresql:
    auth:
      postgresPassword: ""
      username: bn_keycloak
      password: ""
      database: bitnami_keycloak
      existingSecret: ""

veleroValues:
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"

apiVersion: kubeops/kubeopsctl/alpha/v5 # mandatory
imagePullRegistry: "registry.kubeops.net"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11
systemCpu: "200m"
systemMemory: "200Mi"

zones:
  - name: zone1
    nodes:
      master:
        - name: master1
          ipAdress: 10.2.10.11
          status: active
          kubeversion: 1.30.0
        - name: master2
          ipAdress: 10.2.10.12
          status: active
          kubeversion: 1.30.0
      worker:
        - name: worker1
          ipAdress: 10.2.10.14
          status: active
          kubeversion: 1.30.0
        - name: worker2
          ipAdress: 10.2.10.15
          status: active
          kubeversion: 1.30.0
  - name: zone2
    nodes:
      master:
        - name: master3
          ipAdress: 10.2.10.13
          status: active
          kubeversion: 1.30.0  
      worker:
        - name: worker3
          ipAdress: 10.2.10.16
          status: active
          kubeversion: 1.30.0


# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
kubeops-dashboard: true
certman: true
ingress: true 
keycloak: true
velero: true

harborValues: 
  harborpass: "password" # change to your desired password
  databasePassword: "Postgres_Password" # change to your desired password
  redisPassword: "Redis_Password" 
  externalURL: http://10.2.10.11:30002 # change to ip adress of master1

prometheusValues:
  grafanaUsername: "user"
  grafanaPassword: "password"

ingressValues:
  externalIPs: []

keycloakValues:
  keycloak:
    auth:
      adminUser: admin
      adminPassword: admin
  postgresql:
    auth:
      postgresUserPassword: ""
      username: bn_keycloak
      password: ""
      database: bitnami_keycloak
    volumeSize: 8Gi

veleroValues:
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"

Step 7 - Start the Basic Cluster

After the configuration is setup correctly, you can start your basic cluster for the first time:

kubeopsctl apply -f ~/basicCluster.yml

Notice This automatic process may take a while. Do not turn off any machine.

Kubeopsctl: Commands, Cluster Health and Modifications

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

basic kubeopsctl command line operations
apply changes to cluster, e.g. add new worker

Prerequisites

To get the most out of this guide, the following requirements should be met:

kubeopsctl is installed, see kubeopsctl Installation Guide
a cluster is installed and running
basic understanding of Linux environments, bash / shell
basic understanding of text editors, vi / nano
administrator privileges (root) are granted

Overview of the kubeopsctl CLI

kubeopsctl provides a set of command line operations. For more information, see here.

The main kubeopsctl commands are:

Command	Description
`--version`	Shows the current version of kubeopsctl.
`--help`	Shows an overview of all available commands:
`apply`	Sets up the kubeops platform with a configuration file.
`change registry`	Changes the currently used registry to a different one with a given configuration file.
`drain`	Drains a cluster, zone or node.
`uncordon`	Uncordons a cluster, zone or node. This ensures that while existing pods will remain running on the node, no new pods can be scheduled.
`upgrade`	Upgrades the Kubernetes version of a cluster, zone or node.
`status`	Prints the state of a cluster.

Print kubeopsctl Version

kubeopsctl --version

Show kubeopsctl Help

kubeopsctl --help

Setup Kubernetes Cluster or Apply Changes

Sets up the kubeops platform with a configuration file. Use the flag -f and pass the configuration file, e.g. kubeopsctl.yaml.

kubeopsctl apply -f kubeopsctl.yaml

You can also set the log level to a specific value. Available log levels are:

kubeopsctl apply -f kubeopsctl.yaml -l Debug3

The default log level is Info.

Error
Warning
Info (default log level)
Debug1
Debug2
Debug3

The command kubeopsctl apply can also be used to modify a cluster. For more information see Apply Changes to Cluster within this document.

Change the Registry

Changes the currently used registry to a different one with a given configuration file. For example:

kubeopsctl change registry -f kubeopsctl.yaml -r 10.2.10.11/library -t localhost/library

The -f parameter is used to use yaml parameter file.
The parameter -r is used to pull the docker images which are included in the package to a given local docker registry.
The -t parameter is used to tag the images with localhost. For the szenario that the registry of the cluster is exposed to the admin via a network internal domain name, but this name cant be resolved by the nodes, the flag -t can be used, to use the cluster internal hostname of the registry.

Draining

For draining clusters, zones or nodes use kubeopsctl drain <type>/<name>.

To drain a cluster use and replace <clustername> with the desired cluster:

kubeopsctl drain cluster/<clustername>

To drain a zone use and replace <zonename> with the desired zone:

kubeopsctl drain zone/<zonename>

To drain a node use and replace <nodename> with the desired node:

kubeopsctl drain node/<nodename>

Uncordoning

Uncordoning is similar to draining but draining means remove all running tasks/pods so the node can be fixed or taken offline safely. Uncordoning is just opening it up again for new tasks/pods.

For uncordoning clusters, zones or nodes use kubeopsctl uncordon TYPE/NAME.

To uncordon a cluster use and replace <clustername> with the desired cluster:

kubeopsctl uncordon cluster/<clustername>

To uncordon a zone use and replace <zonename> with the desired zone:

kubeopsctl uncordon zone/<zonename>

To uncordon a node use and replace <nodename> with the desired node:

kubeopsctl uncordon node/<nodename>

Upgrading

Upgrade clusters, zones or nodes by using kubeopsctl upgrade -v <version>.

To upgrade a cluster use and replace <clustername> with the desired cluster:

kubeopsctl upgrade cluster/<clustername> -v 1.26.6

To upgrade a zone use and replace <zonename> with the desired zone:

kubeopsctl upgrade zone/<zonename> -v 1.26.6

To upgrade a node use and replace <nodename> with the desired node:

kubeopsctl upgrade node/<nodename> -v 1.26.6

Check on Health and State of Clusters

To check on health and state of a cluster use the command kubeopsctl status cluster/<clustername>.

For example:

kubeopsctl status cluster/basiccluster

Apply Changes to Cluster

If the cluster is already running, it may be necessary to make additional settings or carry out updates.

For example:

when IP addresses or host names of nodes change
if you want to add or remove nodes

In any case, you need to modify the base configuration and apply the necessary changes to the configuration file. Changes will be applied state wise - short hand: only the difference will be applied.

For more information, see How to Guides.

Example: Add new Worker

If you want to add a new worker to your cluster, edit the configuration file.

In this example we reuse the basic configuration basiccluster.yml from the previous chapter (see Set up a Basic Cluster). Use an editor to edit the configuration file:

nano ~/basiccluster.yml

Add the lines to the configuration file for an additional worker4 at zone2. Also edit the desired ipAdress for the new worker.

apiVersion: kubeops/kubeopsctl/alpha/v3 # mandatory
imagePullRegistry: "registry1.kubernative.net/lima"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11

zones:
	- name: zone1
		nodes:
			master:
				- name: master1
					ipAdress: 10.2.10.11
					status: active
					kubeversion: 1.30.0
				- name: master2
					ipAdress: 10.2.10.12
					status: active
					kubeversion: 1.30.0
			worker:
				- name: worker1
					ipAdress: 10.2.10.14
					status: active
					kubeversion: 1.30.0
				- name: worker2
					ipAdress: 10.2.10.15
					status: active
					kubeversion: 1.26.2
	- name: zone2
		nodes:
			master:
				- name: master3
					ipAdress: 10.2.10.13
					status: active
					kubeversion: 1.30.0	
			worker:
				- name: worker3
					ipAdress: 10.2.10.16
					status: active
					kubeversion: 1.30.0

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
headlamp: true
certman: true
ingress: true
keycloak: true
velero: true

rookValues:
	cluster:
		resources:
			mgr:
				requests:
					cpu: 500m
					memory: 1Gi
			mon:
				requests:
					cpu: 500m
					memory: 1Gi
			osd:
				requests:
					cpu: 500m
					memory: 1Gi

harborValues:
	harborpass: "password" # change to your desired password
	databasePassword: "Postgres_Password" # change to your desired password
	redisPassword: "Redis_Password"
	externalURL: http://10.2.10.11:30002 # change to ip adress of master1
	nodePort: 30002
	harborPersistence:
		persistentVolumeClaim:
			registry:
				size: 40Gi
			jobservice:
				jobLog:
					size: 1Gi
			database:
				size: 1Gi
			redis:
				size: 1Gi
			trivy:
				size: 5Gi

prometheusValues:
	grafanaUsername: "user"
	grafanaPassword: "password"

logstashValues:
	volumeClaimTemplate:
		resources:
			requests:
				storage: 1Gi

openSearchValues:
	persistence:
		size: 4Gi

keycloakValues:
	keycloak:
		auth:
			adminUser: admin
			adminPassword: admin
	postgresql:
		auth:
			postgresPassword: ""
			username: bn_keycloak
			password: ""
			database: bitnami_keycloak
			existingSecret: ""

veleroValues:
	namespace: "velero"
	accessKeyId: "your_s3_storage_username"
	secretAccessKey: "your_s3_storage_password"
	useNodeAgent: false
	defaultVolumesToFsBackup: false
	provider: "aws"
	bucket: "velero"
	useVolumeSnapshots: false
	backupLocationConfig:
		region: "minio"
		s3ForcePathStyle: true
		s3Url: "http://minio.velero.svc:9000"

apiVersion: kubeops/kubeopsctl/alpha/v3 # mandatory
imagePullRegistry: "registry.kubeops.net"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11

zones:
	- name: zone1
		nodes:
			master:
				- name: master1
					ipAdress: 10.2.10.11
					status: active
					kubeversion: 1.30.0
				- name: master2
					ipAdress: 10.2.10.12
					status: active
					kubeversion: 1.30.0
			worker:
				- name: worker1
					ipAdress: 10.2.10.14
					status: active
					kubeversion: 1.30.0
				- name: worker2
					ipAdress: 10.2.10.15
					status: active
					kubeversion: 1.26.2
	- name: zone2
		nodes:
			master:
				- name: master3
					ipAdress: 10.2.10.13
					status: active
					kubeversion: 1.30.0	
			worker:
				- name: worker3
					ipAdress: 10.2.10.16
					status: active
					kubeversion: 1.30.0

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
headlamp: true
certman: true
ingress: true
keycloak: true
velero: true

rookValues:
	cluster:
		resources:
			mgr:
				requests:
					cpu: 500m
					memory: 1Gi
			mon:
				requests:
					cpu: 500m
					memory: 1Gi
			osd:
				requests:
					cpu: 500m
					memory: 1Gi

harborValues:
	harborpass: "password" # change to your desired password
	databasePassword: "Postgres_Password" # change to your desired password
	redisPassword: "Redis_Password"
	externalURL: http://10.2.10.11:30002 # change to ip adress of master1
	nodePort: 30002
	harborPersistence:
		persistentVolumeClaim:
			registry:
				size: 40Gi
			jobservice:
				jobLog:
					size: 1Gi
			database:
				size: 1Gi
			redis:
				size: 1Gi
			trivy:
				size: 5Gi

prometheusValues:
	grafanaUsername: "user"
	grafanaPassword: "password"

logstashValues:
	volumeClaimTemplate:
		resources:
			requests:
				storage: 1Gi

openSearchValues:
	persistence:
		size: 4Gi

keycloakValues:
	keycloak:
		auth:
			adminUser: admin
			adminPassword: admin
	postgresql:
		auth:
			postgresUserPassword: ""
			username: bn_keycloak
			password: ""
			database: bitnami_keycloak
		volumeSize: "8Gi"

veleroValues:
	namespace: "velero"
	accessKeyId: "your_s3_storage_username"
	secretAccessKey: "your_s3_storage_password"
	useNodeAgent: false
	defaultVolumesToFsBackup: false
	provider: "aws"
	bucket: "velero"
	useVolumeSnapshots: false
	backupLocationConfig:
		region: "minio"
		s3ForcePathStyle: true
		s3Url: "http://minio.velero.svc:9000"

After the modification is set up corretly, you can apply it to your cluster:

kubeopsctl apply -f ~/basiccluster.yml

Best Practices for Changes and Modifications

When configuring the first setup for a cluster, keep your base configuration file, e.g. basiccluster.yml.

Since the modifications are carried out per status / change, we recommend naming the new configuration files and adding a timestamp, if necessary.

For Example:

basiccluster.yml
20240101-add-worker4.yml
20240101-drain-worker2.yml
20240101-update-kubeversion.yml

Kosi: Installation Guide

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

KOSI system requirements
how to install KOSI
how to use the official KubeOps Website to download KOSI
how to install required software

KOSI seamlessly integrates with other package management software such as Helm and Podman. To ensure compatibility across different containerized environments, KOSI requires the installation of this software.

After the installation, KOSI is available as command line interface.

Note We recommend to install KOSI onto the admin machine of a cluster.

Prerequisites

To get the most out of this guide, the following requirements should be met:

basic understanding of Linux environments, bash / shell
sudo privileges (root) to install packages

Step 1 - System Requirements

Make sure you have prepared the listed requirements before installing KOSI.

KOSI is designed to work with the latest versions of the following operating systems:

OS	Diskspace
Red Hat Enterprise Linux 9.6 and 9.7	500 MB
Ubuntu 24.04.02	500 MB
OpenSUSE MicroOS 20260630 (rolling release)	500 MB

⚠ Warning Only RHEL 9.6 and 9.7 are supported. Make sure the OS version is pinned to prevent accidental updates to an unsupported version, as this may break the cluster.

Important

To register and subscribe, run the command subscription-manager register.

⚠ Warning Only RHEL 9.6 and 9.7 are supported. Make sure the OS version is pinned to prevent accidental updates to an unsupported version, as this may break the cluster.

Step 2 - Include Package Repo

To easily install the kosi package you should add the kubeops package repo to your operating system’s package manager.

wget https://packagerepo.kubeops.net/pgp-key.public
cat pgp-key.public | sudo gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/kubeops.gpg] https://packagerepo.kubeops.net/deb stable main' | sudo tee /etc/apt/sources.list.d/kubeops.list
sudo apt update

sudo dnf config-manager --add-repo https://packagerepo.kubeops.net/rpm/kubeops.repo

# Only the standard OpenSUSE Microos repos in /etc/zypp/repos.d are used
# You have to download the KOSI package manually

Step 3 - Install KOSI

There are two ways to install kosi.

a) quick via the package repo of your operating system’s package manager.

sudo apt update
sudo apt install -y kosi=2.14*

sudo dnf install -y --disableexcludes=kubeops-repo kosi-2.14.0.0-0

# KOSI installation via package repo is not supported
# Please install it manually

b) manual download the kosi package from the official KubeOps Website and install it.

Downloading KOSI

Login into your KubeOps account. If you don‘t have an account, you can create one by following the instructions for Create a KOSI account (with SSO).

Download your desired version of the KOSI package file (.rpm or .deb) from the official download page.

# download kosi deb manually and install with
sudo dpkg --install kosi_2.14.0.0-1_amd64.deb

# download kosi rpm manually and install with
sudo rpm --install kosi-2.14.0.0-0.x86_64.rpm

# Download KOSI rpm manually and install with
sudo transactional-update pkg install --no-confirm --allow-unsigned-rpm --force kosi-2.15.0.4_Beta0-0.x86_64.rpm

# Need reboot after transactional update!
sudo reboot now 

# Copy plugins and KOSI config from last snapshot
sudo mkdir -p /var/kubeops/plugins
sudo cp -r /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/var/kubeops/plugins/ /var/kubeops/
sudo cp -r /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/var/kubeops/kosi/ /var/kubeops/

Step 4 - Set the KUBEOPSROOT env var

Set KUBEOPSROOT and XDG_RUNTIME_DIR in ~/.bashrc

⚠ Warning Adapt /home/<yourUser>/kubeops with your value before usage.

Remove existing KUBEOPSROOT and XDG_RUNTIME_DIR entries before setting your values.

sed -i '/^export KUBEOPSROOT=/d' ~/.bashrc
sed -i '/^export XDG_RUNTIME_DIR=/d' ~/.bashrc

Then add your values:

# file ~/.bashrc
# Add these values to the end of your ~/.bashrc file
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

Source .bashrc to apply the values

source ~/.bashrc
echo $KUBEOPSROOT
echo $XDG_RUNTIME_DIR

Both commands should return your KUBEOPSROOT path, for example:

/home/<yourUser>/kubeops
/home/<yourUser>/kubeops

Step 5 - Adjust KOSI Configuration

This creates a kubeops directory in your home directory and transfers all necessary files, e.g., the kosi-config and the plugins, to it.

mkdir ~/kubeops
cd ~/kubeops
cp -R /var/kubeops/kosi/ .
cp -R /var/kubeops/plugins/ .

The config.yaml is in your KUBEOPSROOT-path (typically in ~/kubeops/kosi)

Set hub in your kosi config to hub: https://dispatcher.kubeops.net/v4/dispatcher/

⚠ Warning This file entry has to be adapted to your values before usage.

Set the “plugins”-entry in your kosi config to plugins: /home/<yourUser>/kubeops/plugins, where is changed to your username

⚠ Warning This file has to be adapted to your values before usage.

# file $KUBEOPSROOT/kosi/config.yaml
apiversion: kubernative/sina/config/v3

spec:
  hub: https://dispatcher.kubeops.net/v4/dispatcher/ # <-- set hub url
  plugins: <your kubeopsroot>/plugins/ # <-- set the path to your plugin folder (~ for home or $KUBEOPSROOT don't work, it has to be the full path)
  workspace: /tmp/kosi/process/
  logging: info
  housekeeping: false
  proxy: false
  checkforupdates: false

Step 6 - Install KOSI enterprise plugins

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.0.0

Step 7 - Install Podman

To ensure compatibility across different containerized environments, KOSI requires the installation of Podman (latest version).

Install Podman using the inbuilt package manager.

sudo apt update
sudo apt install -y podman

sudo dnf install -y podman

# Podman is already installed by default

Step 8 - Install Helm

Many KOSI packages use helm. Therefore, it is recommended to install helm.

Install Helm using the KubeOps package repo.

sudo apt update
sudo apt install -y helm

sudo dnf install -y helm

sudo transactional-update pkg install --no-confirm helm
sudo reboot now

Verify your Installation

To verify the installation of KOSI on your system, use the command kosi version.

kosi version

The output displays the KOSI version along with relevant information. It confirms the successful installation of KOSI.

2026-01-23 16:09:55 Info:      KOSI version: 2.14.0.0
2026-01-23 16:09:55 Info:      Latest KOSI-package-apiversion: kubernative/kubeops/sina/user/v4
2026-01-23 16:09:55 Info:      Latest KOSI-lanugage-version is: 1.0.0
2026-01-23 16:09:55 Info:      This work is licensed under Creative Commons Attribution - NoDerivatives 4.0 International License(see https://creativecommons.org/licenses/by-nd/4.0/legalcode for more details).
2026-01-23 16:09:55 Info:      © KubeOps GmbH, Hinter Stöck 17, 72406 Bisingen - Germany, 2025

Check if you can log in with the command kosi login.

kosi login -u <yourUser>

The output confirms that you are successfully loged in.

2026-01-23 16:11:40 Info:      KOSI version: 2.14.0.0
2026-01-23 16:11:47 Info:      Login Succeeded to Hub.
2026-01-23 16:11:48 Info:      The login to registry was successful. You are now logged in as kubeops and can use the public and private registry.

Kosi: Creating a Basic KOSI-Package and Deployment

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

how to get to know KOSI-packages
add repositories and download helm charts
create a KOSI-package
configure a KOSI-package to deploy a demo WordPress page
optional: install and deploy the KOSI-package to a cluster

Notice

The intent of this guide is to setup a demo WordPress page on your cluster or your machine.

We do not recommend this guide for production environments.

Important To get the most out of this guide, we recommend to create your first demo KOSI-package on the admin machine of a basic cluster. For a helpful guide, see Set up a Basic cluster using kubeopsctl.

Prerequisites

To get the most out of this guide, the following requirements should be met:

KOSI is installed, see KOSI Installation Guide
basic understanding of Kubernetes, network clusters and tools such as Helm or Podman
basic understanding of Linux environments, bash / shell
administrator privileges (root) are granted

Step 1 - Create a KOSI-Package

After the installation, KOSI is available as command line interface.

Create a new folder and use the command kosi create to create a default KOSI-package with pre-built files.

mkdir ~/kosi-demo
cd ~/kosi-demo

kosi create

Example output:

2024-08-07 13:49:00 Info:      KOSI version: 2.14.0.0_XXXXXXXXXX
2024-08-07 13:49:00 Info:      Write package.kosi to /root/kosi-demo
2024-08-07 13:49:00 Info:      Write template.yaml to /root/kosi-demo
2024-08-07 13:49:00 Info:      Write logo.png to /root/kosi-demo
2024-08-07 13:49:00 Info:      Write docs.tgz to /root/kosi-demo

File	Description
`package.kosi`	Contains the „logic“ of the package.
`template.yml`	A template file for configurations and the templating engine.
`logo.png`	A logo for the package. replace this with your desired logo (png format).
`docs.tgz`	Contains your documentaion of your package.

Step 2 - Understand the Package Logic

The package.kosi file contains the „logic“ of the package. This file uses the kosi format which itself uses the highly expressive KOSI Language.

Among other things the package.kosi file defines:

a preambel for meta data and descriptions
a files section to - manages files, such as templates, to be included and target files
a set of commands - for example „install“ or „upgrade“ which work as scripts within the kosi file

Templates make it possible to use variantes for your packages. For example, you want to reuse a package for different environments. Templates also allow to parse additional „value files“ as command line arguments.

Advanced templating makes it possible to split one large value files into smaller ones. In this case we use some of the defaultvalues but all values will be transfered.

Step 3 - Modify the Template File

For the transfer of values into a package, a valuestemplate.yaml file is needed.

Use an editor to edit the file:

nano ~/kosi-demo/valuestemplate.yaml

To get the most out of this guide, add the following line:

{{values}}

This command will allow to parse the values of the file values.yaml. On installing, the file values.yaml will be automatically transcluded into the cluster. This is helpful to reuse a package for different environments.

Notice the exact same naming of the command with the file.

Step 4 - Add Necessary Repositories and Pull Charts

In this guide we want to create a KOSI-package containing a demo WordPress page. Therefore you need to add the necessary repository via helm. Afterwards you need to pull the latest version of the helm chart.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/wordpress --version X.Y.Z

Step 5 - Configure the Package

Configure the package to inlucde the valuestemplate.yaml file. Also add commands for install, update, and delete.

Use an editor to edit the file:

nano ~/kosi-demo/package.kosi

To get the most out of this guide, overwrite the file with the following lines:

languageversion = "1.0.0";
apiversion = "kubernative/kubeops/sina/user/v4";
name = "wordpressdemo";
description = "Deploys a wordpress helm chart";
version = "0.1.0";
docs = "docs.tgz";
logo = "logo.png";

files =
{
    wordpressvaluestemplate= "valuestemplate.yaml";
    chart = "wordpress-X.Y.Z.tgz";
}

containers = 
{
    wordpress =
    [
        "registry-1.docker.io",
        "bitnami/wordpress",
        "latest"
    ];

    mariadb =
    [
        "registry-1.docker.io",
        "bitnami/mariadb",
        "latest"
    ];
}

install
{
    template(tmpl="valuestemplate.yaml";target="wordpressvalues.yaml");
    helm(command="upgrade";tgz="wordpress-X.Y.Z.tgz";values="['wordpressvalues.yaml']";deploymentName="wordpressdemo";namespace="wordpressdemo";flags="['--create-namespace',' --install']");
}

update
{
    template(tmpl="valuestemplate.yaml";target="wordpressvalues.yaml");
    helm(command="upgrade";tgz="wordpress-X.Y.Z.tgz";values="['wordpressvalues.yaml']";deploymentName="wordpressdemo";namespace="wordpressdemo";flags="['--create-namespace',' --install']");
}

delete
{
    print(message="delete wordpressdemo helm deployment.");
    helm(command="delete";deploymentName="wordpressdemo";namespace="wordpressdemo";flags="['--wait']");
    print(message="echo delete wordpressdemo namespace.");
    print(message="kubectl delete namespace wordpressdemo");
}

Step 6 - Create Package Documentation

It is always a good practice to document your packages.

Within your package folder, create a docs folder:

mkdir -p ~/kosi-demo/docs

Create a simple README file (for example markdown format) and write some helpful descriptions. For example:

echo „My first KOSI-package. Contains a demo WordPress page.“ > ~/kosi-demo/docs/README.md

Afterwards you need to compress the whole docs folder using the tgs format.

cd ~/kosi-demo/
tar -cvzf docs.tgz docs/

These docs will be included when building the deployable package file.

Step 7 - Build the Package

When the package.kosi is finished and the documentation is done, the KOSI-package can be built. Navigate to the package folder and simply use the kosi build command.

cd ~/kosi-demo/
kosi build

During the build process, the files package.tgz and package.yaml are created.

Step 8 - Setting Values for the Package

To install the package.tgz, you need the package itself and some values. Among other things, you need to define the nodePorts for http at 30123. This node port will be needed to access and test your WordPress page.

In a previous step we made the package reuseable by using the valuestemplate.yaml file. This files expects a values.yaml file as command line argument.

Use an editor to edit the file:

nano ~/kosi-demo/values.yaml

To get the most out of this guide, add the following lines:

global:
    storageClass: rook-cephfs
service:
    type: NodePort
    nodePorts:
        http: 30123
mariadb:
    enabled: true
    auth:
        rootPassword: topsecretChangeMe
        password: secretChangeMe
    primary:
        persistence:
            enabled: true
            storageClass: rook-cephfs
            accessModes:
                - ReadWriteOnce
persistence:
    enabled: true
    storageClass: rook-cephfs

Optional Step 9 - Installing the Package and Deploy to Cluster

As recommended you need a running Kubernetes cluster to install and deploy your new KOSI-package. You must also be logged in to the Admin machine where you have created the KOSI-package. To set up a cluster easily and conveniently, you can use kubeopsctl (see kubeopsctl documentation).

Navigate to your built KOSI-package. Install the package and pass the prepared values.yaml file as command line argument.

cd ~/kosi-demo
kosi install -p package.tgz -f values.yaml

Important This command works with any running Kubernetes cluster.

Optional Step 10 - Test the WordPress Page

After you deployed the KOSI-package you can test the access via your cluster to the WordPress page.

Depending on your setup you need the following:

the IP or hostname of a master node
tunneling or expossing of ports according to the nodePorts as set in values.yaml

In this guides we set 30123 as nodePort. For example, in your browser navigate to:

http://master1:30123

Kubeopsctl-2.0: How to migrate from nginx to traefik ingress

Mon, 01 Jan 0001 00:00:00 +0000

Installation

Kubeops supports deploying Traefik as a dynamic ingress controller and reverse proxy. This guide describes a concise, safe migration from an existing nginx-ingress controller to Traefik and explains how to install Traefik and replace a deprecated nginx-ingress deployment. The migration from nginx to Traefik is straightforward; the steps below show the process in order.

Prerequisites

A running Kubernetes cluster with an existing nginx-ingress controller.
It is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

1.Create Values file

Create a values.yaml file for the Traefik installation:

# values.yaml
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik
      externalIPs: []
    advanced: {}

Note: Update externalIPs and any other values as required for your environment.

2.Install Traefik

non-airgap-environment

After creating values.yaml, install Traefik:

# get your desired version/s
kosi search --hub kubeops --ps traefik
# install traefik
kosi install --hub kubeops kubeops/traefik:<desired_version> -f values.yaml --dname traefik

Example Non-airgap

kosi install --hub kubeops kubeops/traefik:2.1.0_Beta0 -f values.yaml --dname traefik

Airgap-environment

# log in to Harbor
podman login <ip_address>:<NodePort> -u <username> -p <password> --tls-verify=false
# pull your desired version
kosi pull --hub kubeops kubeops/<desired_version> -o traefik-<desired_version>.tgz -r <ip_address>:<NodePort>/kubeops -t 127.0.0.1:<NodePort>/kubeops
 # install Traefik
kosi install --dname traefik -p traefik-:<desired_version>.tgz -f values.yaml

Example Airgap

kosi pull --hub kubeops kubeops/traefik:2.1.0_Beta0 -o traefik-2.1.0-Beta0.tgz -r 10.2.10.11:30002/kubeops -t 127.0.0.1:30002/kubeops
 
kosi install --dname traefik -p traefik-2.1.0-Beta0.tgz -f values.yaml

3.Verify Deployment

Verify that the Traefik pods and services are running in the traefik namespace:

kubectl get pods -n traefik
kubectl get svc -n traefik

3b. Switch applications to the Traefik ingress class

This is the load-bearing step of the migration. Traefik ships a backwards-compatibility shim (an IngressClass named nginx plus the experimental providers.kubernetesIngressNginx provider) so that existing ingressClassName: nginx ingresses should keep working unchanged. This shim is experimental and does not work reliably for all workloads — Keycloak in particular has been observed to be unreachable through the nginx class while Traefik is the controller. Do not rely on the shim as the primary mechanism; switch each application’s ingress to ingressClassName: traefik.

Every ingress package exposes a uniform standard.ingressClassName key (defaulting to nginx). Set it to traefik on each package, and set the cert-manager solver class:

- name: keycloak
  values:
    standard:
      ingressClassName: traefik
- name: kubeOpsDashboard
  values:
    standard:
      ingressClassName: traefik
- name: kube-prometheus-stack
  values:
    standard:
      ingressClassName: traefik
- name: harbor
  values:
    standard:
      ingressClassName: traefik
- name: rook-ceph
  values:
    standard:
      ingressClassName: traefik
- name: opensearch-dashboards
  values:
    standard:
      ingressClassName: traefik
- name: kubevirt-manager
  values:
    standard:
      ingressClassName: traefik
- name: cert-manager
  values:
    ca:
      ingressName: traefik         # ACME HTTP01 solver ingress class

cert-manager is the only exception to the standard.ingressClassName key: its solver class lives under values.ca.ingressName. Certificates and the cert-manager.io/cluster-issuer annotations are otherwise controller-agnostic.

Note: the plain OpenSearch data package (opensearch-os) and logstash ship their ingress disabled by default, so they need no change. If you enable them, set the class via that package’s advanced block (ingress.ingressClassName / ingress.className).

Apply the updated values and confirm every ingress now uses the Traefik class:

kubectl get ingress -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName

SSO / certificates / dashboards: SSO is app-level OIDC (Keycloak reachable at https://<host>/keycloak), so it is not tied to the ingress controller and keeps working once the class is switched. The same is true for cert-manager and dashboard access.

nginx-only annotations do not migrate. Annotations of the form nginx.ingress.kubernetes.io/* are ignored by Traefik. OpenSearch Dashboards ships buffer/ header-size annotations (proxy-buffer-size, large-client-header-buffers, …) that matter for large OIDC headers; if login fails with 4xx/431/502, add an equivalent Traefik buffering middleware / entrypoint transport setting via the package’s advanced block. Harbor auto-injects nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" when internalTLS.enabled — verify Harbor over TLS after the switch.

4.Remove old nginx-ingress deployment and service

# get version of installed nginx-ingress and its deployment name (--dname)
kosi list
# delete old nginx-ingress
kosi delete --hub kubeops kubeops/ingress-nginx:<installed_version> -f enterprise-values.yaml --dname <kosi_deployment_name>

Example


kosi delete --hub kubeops ingress-nginx:2.1.0_Beta0 -f values.yaml --dname ingress-nginx

Clean up the leftover nginx IngressClass. The ingress-nginx chart annotates its IngressClass with helm.sh/resource-policy: keep, so it survives kosi delete. Both ingress-nginx and the Traefik package define an IngressClass named nginx for controller k8s.io/ingress-nginx, so they must not run at the same time. After removing ingress-nginx, delete any leftover class if it is no longer wanted:
kubectl get ingressclass
kubectl delete ingressclass nginx   # only if you no longer need the nginx compat class

Edit Traefik service

If nginx-ingress used specific NodePorts that you need to reuse, update the Traefik Service:

kubectl edit svc traefik -n traefik

Note: Default NodePorts (for example, 31080 / 31443) might not be reachable in your environment. If these ports are not accessible, determine the NodePorts previously used by nginx-ingress (for example, 30080 / 30443) and configure Traefik to use the same ports.

Update the ports:

Adjust the ports section to match the previous nginx NodePorts if required:

ports:
- name: web
  nodePort: 30080
  port: 80
  targetPort: web
- name: websecure
  nodePort: 30443
  port: 443
  targetPort: websecure

Verify Port Change

kubectl get svc -n traefik

Note: Ensure that the nginx Service is removed or that its NodePorts are freed before reusing those NodePorts on the Traefik Service.

KubeOpsDoku – guide

Kubeopsctl-2.0: Joining a Node in a Kubernetes cluster

Join Node Process:

Prerequisits

KOSI Login Recommendation

ETCD Backup Recommendation

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

Example 1: Joining a Worker Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

Kubeopsctl-2.0: Deleting a Node from a Kubernetes cluster

Deleting a Node from a Kubernetes cluster

Prerequisites

Worker

Scale down the rook-ceph-operator deployment to 0

Check which hosts and OSDs belong to each zone

Scale down the OSD deployment

Remove the OSD via ceph-tools

Check the MON running on the node

Delete OSD and MON deployments

Remove the deleted mon from the ceph tools

Delete the node from the kubernetes cluster

Scale the rook-ceph-operator deployment back to 1

Timing and health checks

Kubeopsctl-2.0: Upgrading a Kubernetes cluster

Kubernetes Version Upgrade Process:

Prerequisites

KOSI Login Recommendation

1. Pull required KOSI packages on your ADMIN

2. Change your target version inside the cluster-values

3. Start the upgrade

Example 1 - Upgrade all nodes in the cluster to a specific version

1. Pull required KOSI packages on your ADMIN

2. Update the target version in cluster-values.yaml

2. Validate your values and upgrade the cluster

Example 2 - Tranche upgrade of zones to a specific version

1. Pull required KOSI packages on your ADMIN

2. Adjust your cluster-values in zone1

3. Validate your values and upgrade the cluster

4. Adjust your cluster-values.yaml in zone2

5. Validate your values and upgrade the cluster

6. Adjust your cluster-values.yaml in zone3

7. Validate your values and upgrade the cluster

Kubeopsctl-2.0: Installing KubeOps Compliance applications

Prerequisits

KOSI Login Recommendation

Example 1: Installing Applications in a non-airgap-environment

Example 2: Installing Applications in an airgap-environment

Kubeopsctl-2.0: Updating KubeOps Compliance applications

Prerequisites

KOSI Login Recommendation

Updated KubeOpsctl

Example 1: Updating Applications in a non-airgap-environment

Example 2: Updating Applications in an airgap-environment

Kubeopsctl-2.0: Harbor Deployment with CloudNativePG

Harbor Deployment with CloudNativePG in Non-airgap enviroment

Prerequisites

Step 1 — Install CloudNativePG operator

Step 2 — Create PostgreSQL cluster for Harbor

1. Apply the following Cluster manifest to create a Postgres cluster with 2 instances and 1Gi storage:

2. Services and pods created for the cluster cloudnative-pg:

3. Verify pods are Running:

Step 3 — Retrieve application user credentials

1. Verify the Secret exists:

2. Decode the base64-encoded fields:

Step 4 — Update Harbor tools.yaml for an external database

Step 5 — Install Harbor with Kosi

1. Deploy Harbor using the updated tools.yaml:

2. Verify Harbor pods:

3. Access Harbor at: <your_domain_name>:30002 (or as configured)

Harbor Deployment with CloudNativePG in Airgap envoirnment

Prerequisites

Step 1: Login into Harbor

Step 2: Manually Mirror the PostgreSQL Image to the KubeOps Project

Step 3: Create the PostgreSQL Cluster for Harbor

Step 4: Configure the Cluster to Use the Mirrored Image

1.List the cluster:

2.Edit the cluster:

3.Replace the image name in the editor:

Step 5: Retrieve CNPG Application User Credentials

2. Update the target version in `cluster-values.yaml`

2. Adjust your cluster-values in `zone1`

4. Adjust your `cluster-values.yaml` in `zone2`

6. Adjust your `cluster-values.yaml` `in zone3`

2. Services and pods created for the cluster `cloudnative-pg`:

Recommended: Change the `admin` password via the package value

Advanced: Change any user’s password manually via `securityadmin`