KubeOps Compliance 2.0

• 1: Release Notes
• 1.1: KubeOps Versions
• 1.2: KubeOps 2.1.1
• 1.3: KubeOps 2.2.0_Beta1
• 1.4: KubeOps 2.2.0_Beta0
• 1.5: KubeOps 2.1.0
• 1.6: KubeOps 2.1.0_Beta0
• 1.7: KubeOps 2.0.5
• 1.8: KubeOps 2.0.4
• 1.9: KubeOps 2.0.3
• 1.10: KubeOps 2.0.2
• 1.11: KubeOps 2.0.1
• 2: Getting-Started
• 2.1: About KubeOps Compliance
• 2.2: About KubeOps Compliance Licences
• 2.3: Prerequisites
• 2.4: Prepare Cluster
• 2.5: Setup Cluster
• 3: How to Guides
• 3.1: Documentation for production
• 3.1.1: Supported Package Versions — Ubuntu / Debian
• 3.1.2: Supported Package Versions — Red Hat Enterprise Linux (RHEL)
• 3.2: Joining a Node in a Kubernetes cluster
• 3.3: Deleting a Node from a Kubernetes cluster
• 3.4: Single Sign-On with Keycloak
• 3.4.1: Single Sign-On (SSO) with Keycloak for Harbor
• 3.4.2: Single Sign-On (SSO) with Keycloak for KubeOps Dashboard
• 3.4.3: Single Sign-On (SSO) with Keycloak for OpenSearch
• 3.5: Upgrading a Kubernetes cluster
• 3.6: Installing KubeOps Compliance applications
• 3.7: Updating KubeOps Compliance applications
• 3.8: Harbor Deployment with CloudNativePG
• 3.9: Ingress Configuration
• 3.10: Accessing Dashboards installed with KubeOps
• 3.11: Accessing Dashboards with OIDC
• 3.12: Changing a User Password in OpenSearch
• 3.13: Backup and restoring artifacts
• 3.14: Deploy Package on a cluster
• 3.15: How to migrate from nginx to traefik ingress
• 4: Reference
• 4.1: KubeOpsCtl CLI Commands
• 4.2: FAQs
• 4.3: Fileformats in kubeopsctl
• 4.4: Glossary
• 4.5: Software Bill of Materials (SBOM)

1 - Release Notes

1.1 - KubeOps Versions

Here is the KubeOps and it’s supported tools versions list. Make sure to install or upgrade according to supported versions only.

1.2 - KubeOps 2.1.1

KubeOps 2.1.1 - Release Date ??.08.2026

Changelog KubeOpsCtl 2.1.1

Fixes

- fixed missing opensearch-dashboards plugins and wrong version 

Versions KubeOps 2.1.1

1.3 - KubeOps 2.2.0_Beta1

KubeOps 2.2.0_Beta1 - Release Date 02.08.2026

Changelog KubeOpsCtl 2.2.0_Beta1

Whats new?

- Added Traefik support for kubeopsctl
- Added Kubevirt-Manager support for kubeopsctl
- Added Kubernetes 1.36 support
- Added Advanced Parameters for static Registry mount path, check out the file format [Cluster Values YAML](/kubeopsctl-2.0/reference/fileformats/#cluster-valuesyaml)

Fixes

- fixed a podman bug affecting pull packages
- fixed a harbor bug setting up cluster
- added creation of nginx ingressclass, when installing traefik for backwards compatabilty
- fixed default secret for kubeops-dashboard
- kubeopsctl pull command now verifies if the specified tool value file exists
- Added kubeops-harbor-secret for pulling images from the internal private harbor project 'kubeops'
- fixed a bug that prevented to create the kubeops harbor project as private 
- fixed a bug that prevented to change the kubeops harbor project to private 
- fixed a bug that prevented the image push of the kubernetes images to local-registry 
- changed the 'localhost' reference to '127.0.0.1' for enterprise-tools in preperation for the kubernetes 1.36 support
- fix image push in local registry
- fixed duplicated coredns in 1.36.1
- updated kosi versions for consistent installation
- updated Harbor ImagePullSecret with 127.0.0.1 instead of master1 IP
- remove hardcoded plugin versions in cluster prepare node
- fixed a bug in rook delete section
- fixed a bug in nginx ingressclass for traefik
- added default secret for grafana
- added role bindings for the kubevirt roles
- fixed delete ingresses in update process 
- OpenSearch users created in the Dashboards Security UI now persist across package updates 
- OpenSearch TLS certificates are no longer regenerated on every install/update, preventing certificate rotation on running clusters 
- Fixed OpenSearch SSO returning 401 Unauthorized by reliably applying the openid security configuration to the backend after deploy 
- Added a Keycloak protocol mapper so realm roles are propagated to OpenSearch for backend role mapping 
- Added Single Sign-On (SSO) with Keycloak (OpenID Connect) for OpenSearch and OpenSearch Dashboards 
- Added username/password fallback authentication for OpenSearch alongside SSO 
- Made the OpenSearch admin password configurable via the new `adminPassword` package value (opensearch-os & opensearch-dashboards) 
- Added `hostname` package value for opensearch-os to point OpenSearch at the Keycloak SSO endpoint 
- Refresh the Keycloak admin token before each step of the OpenSearch Dashboards OIDC join so long-running joins no longer fail with 401 once the 60s token lifespan is exceeded
- fixed an error in OpenSearch when it was not installed in the `logging` namespace
- fixed an issue in kubeopsctl where `tools-values.yaml` and `cluster-values.yaml` were not always stored as secrets in the cluster
- changed the netpols for traefik-nginx and ingress-nginx to make the dashboards available
- helm-plugin now parses the flags correctly in all possible actions (install, upgrade, delete)
- fixed missing kosi update on nodes, if packageRepository: local
- fixed missing opensearch-dashboards plugins and wrong version
- template network policies with default deny rules
- when installing the tools, apply cluster- and tools-values

Known Issues

Versions KubeOps 2.2.0_Beta1

1.4 - KubeOps 2.2.0_Beta0

KubeOps 2.2.0_Beta0 - Release Date 30.07.2026

Changelog KubeOpsCtl 2.2.0_Beta0

Whats new?

- Added Traefik support for kubeopsctl
- Added Kubevirt-Manager support for kubeopsctl
- Added Kubernetes 1.36 support
- Added Advanced Parameters for static Registry mount path, check out the file format [Cluster Values YAML](/kubeopsctl-2.0/reference/fileformats/#cluster-valuesyaml)

Fixes

- fixed a podman bug affecting pull packages
- fixed a harbor bug setting up cluster
- added creation of nginx ingressclass, when installing traefik for backwards compatabilty
- fixed default secret for kubeops-dashboard
- kubeopsctl pull command now verifies if the specified tool value file exists
- Added kubeops-harbor-secret for pulling images from the internal private harbor project 'kubeops'
- fixed a bug that prevented to create the kubeops harbor project as private 
- fixed a bug that prevented to change the kubeops harbor project to private 
- fixed a bug that prevented the image push of the kubernetes images to local-registry 
- changed the 'localhost' reference to '127.0.0.1' for enterprise-tools in preperation for the kubernetes 1.36 support
- fix image push in local registry
- fixed duplicated coredns in 1.36.1
- updated kosi versions for consistent installation
- updated Harbor ImagePullSecret with 127.0.0.1 instead of master1 IP
- remove hardcoded plugin versions in cluster prepare node
- fixed a bug in rook delete section
- fixed a bug in nginx ingressclass for traefik
- added default secret for grafana
- added role bindings for the kubevirt roles
- fixed delete ingresses in update process 
- OpenSearch users created in the Dashboards Security UI now persist across package updates 
- OpenSearch TLS certificates are no longer regenerated on every install/update, preventing certificate rotation on running clusters 
- Fixed OpenSearch SSO returning 401 Unauthorized by reliably applying the openid security configuration to the backend after deploy 
- Added a Keycloak protocol mapper so realm roles are propagated to OpenSearch for backend role mapping 
- Added Single Sign-On (SSO) with Keycloak (OpenID Connect) for OpenSearch and OpenSearch Dashboards 
- Added username/password fallback authentication for OpenSearch alongside SSO 
- Made the OpenSearch admin password configurable via the new `adminPassword` package value (opensearch-os & opensearch-dashboards) 
- Added `hostname` package value for opensearch-os to point OpenSearch at the Keycloak SSO endpoint 
- Refresh the Keycloak admin token before each step of the OpenSearch Dashboards OIDC join so long-running joins no longer fail with 401 once the 60s token lifespan is exceeded

Known Issues

Versions KubeOps 2.2.0_Beta0

1.5 - KubeOps 2.1.0

KubeOps 2.1.0 - Release Date 16.06.2026

Changelog KubeOpsCtl 2.1.0

Whats new?

- Added Single Sign-On (SSO) with Keycloak for Rook-Ceph
- Added Single Sign-On (SSO) with Keycloak for Harbor
- Added Single Sign-On (SSO) with Keycloak for Grafana
- Added Single Sign-On (SSO) with Keycloak for KubeOps Dashboard
- Added Single Sign-On (SSO) with Keycloak for OpenSearch Dashboard
- Added Traefik for manual installation
- Added CloudNativePG operator for manual installation
- Added command kubeopsctl check-connections -f cluster-values.yaml
- Added check-connections before kubeopsctl apply
- Improved cert-manager package to install lets-encrypt-staging as inital CA for SSO support
- Improved harbor backup by adding pre.hook.backup.velero.io in harbor package
- Improved keycloak backup by adding pre.hook.backup.velero.io in harbor package
- Improved kubeopsctl apply no longer throws an exception when cluster-values.yaml is missing.
- Added Kubernetes 1.35 support

Fixes

- Implemented fixes for the following CVEs affecting the `rook-ceph` package:
    - CVE-2025-68121
    - CVE-2026-33186
- Implemented fixes for the following CVEs affecting the `ingress-nginx` package:
    - CVE-2025-68121
- Implemented fixes for the `keycloak` package:
    - A Keycloak bug has been fixed where the database was not persistent
- Implemented fixes for the `opensearch-os` package:
    - A bug has been fixed that prevented usernames and passwords from being retained after the upgrade

Versions KubeOps 2.1.0

1.6 - KubeOps 2.1.0_Beta0

KubeOps 2.1.0_Beta0 - Release Date 13.04.2026

Changelog KubeOpsCtl 2.1.0_Beta0

Whats new?

- Added Single Sign-On (SSO) with Keycloak for Harbor
- Added Single Sign-On (SSO) with Keycloak for Grafana
- Added Single Sign-On (SSO) with Keycloak for KubeOps Dashboard
- Added Traefik for manual installation
- Added CloudNativePG operator for manual installation
- Added command kubeopsctl check-connections -f cluster-values.yaml
- Added check-connections before kubeopsctl apply
- Improved cert-manager package to install lets-encrypt-staging as inital CA for SSO support
- Improved harbor backup by adding pre.hook.backup.velero.io in harbor package
- Improved keycloak backup by adding pre.hook.backup.velero.io in harbor package
- Improved kubeopsctl apply no longer throws an exception when cluster-values.yaml is missing.

Versions KubeOps 2.1.0_Beta0

1.7 - KubeOps 2.0.5

KubeOps 2.0.5 - Release Date 08.05.2026

Changelog KubeOpsCtl 2.0.5

Fixes

- Implemented fixes for the following CVEs affecting the `rook-ceph` package:
    - CVE-2025-68121
    - CVE-2026-33186
- Implemented fixes for the following CVEs affecting the `ingress-nginx` package:
    - CVE-2025-68121
- Implemented fixes for the `keycloak` package:
    - A Keycloak bug has been fixed where the database was not persistent
- Implemented fixes for the `opensearch-os` package:
    - A bug has been fixed that prevented usernames and passwords from being retained after the upgrade
- Implemented fixes for RHEL:
    - A bug has been fixed that prevented users from using a repository other than “kubeops-repo”

Versions KubeOps 2.0.5

1.8 - KubeOps 2.0.4

KubeOps 2.0.4 - Release Date 02.03.2026

Changelog KubeOpsCtl 2.0.4

Fixes

- Implemented fixes for the following CVEs affecting the `harbor` package:
    - CVE-2025-27151
    - CVE-2025-49844
- Implemented fixes for the following CVEs affecting the `opensearch` package:
    - CVE-2025-68428
    - CVE-2025-9288
- Implemented fixes for the following CVEs affecting the `opensearch-dashboards` package:
    - App Version 3.4.0, Chart Version 3.4.0
    - CVE-2025-68428
    - CVE-2025-9288

Versions KubeOps 2.0.4

1.9 - KubeOps 2.0.3

KubeOps 2.0.3 - Release Date 10.02.2026

Changelog KubeOpsCtl 2.0.3

Fixes

- Implemented fixes for the following CVEs affecting the `ingress-nginx` package:
    - CVE-2025-15467
    - CVE-2025-49794
    - CVE-2025-49796

Versions KubeOps 2.0.3

1.10 - KubeOps 2.0.2

KubeOps 2.0.2 - Release Date 28.01.2026

Changelog KubeOpsCtl 2.0.2

Whats new?

- Added --delete-emptydir-data while draining nodes
- Added Repo support for the KOSI installation packages

Fixes

- Fixed a bug where imagePullSecrets were not created

Versions KubeOps 2.0.2

1.11 - KubeOps 2.0.1

KubeOps 2.0.1 - Release Date 22.12.2025

Changelog KubeOpsCtl 2.0.1

Whats new?

- Supports KubeVIP
- Supports Ubuntu 24.04 LTS
- Supports RHEL 9.6
- Supports Cilium CNI
- Reworked Cluster creation process
- Reworked Cluster upgrade process
- Reworked Compliance application installation process
- Reworked Compliance application update process
- Introduced new value format for cluster-values.yaml
- Introduced new value format for enterprise-values.yaml
- Introduced selfcontained air-gap packages
- Improve the Cluster creation process
- Improve the Cluster upgrade process
- Improve the Compliance application installation process
- Improve the Compliance application update process
- Improve the Cluster creation/upgrade speed
- Improve the Compliance application installation/update speed
- Updated opensearch package from 1.7.6 to 2.0.0
- Updated opensearch-dashboard package from 1.7.6 to 2.0.0
- Updated velero package from 1.7.6 to 2.0.0
- Enterprise packages are only pulled if they are marked as `enabled: true` in the enterprise-values.yaml

Fixes

- "Delete Node" now works over multiple zones
- An invalid cluster-values.yaml is now caught before cluster creation
- If no helm chart is installed the cluster creation process will still run through
- Harbor: fixed missing template for hostname
- nginx-ingress: updated netpols
- opensearch-dashboard: changed issuer to cluster-issuer
- airgap-packages are pulled from locally installed harbor registry instead of public registries

Known Issues

Versions KubeOps 2.0.1

2 - Getting-Started

2.1 - About KubeOps Compliance

What is KubeOps Compliance?

kubeopsctl serves as a versatile utility designed specifically to efficiently manage both the configuration and status of a cluster.

With its capabilities, users can articulate their desired cluster state in detail, outlining configurations and specifications.

Subsequently, kubeopsctl orchestrates the creation of a cluster that precisely matches the specified state, ensuring alignment between intentions and operational reality.

Why use KubeOps Compliance?

In kubeopsctl, configuration management involves defining, maintaining, and updating the desired state of a cluster, including configurations for nodes, pods, services, and other resources in the application environment.

The main goal of kubeopsctl is to match the clusters actual state with the desired state specified in the configuration files. With a declarative model, kubeopsctl enables administrators to express their desired system setup, focusing on „what“ they want rather than „how“ to achieve it.

This approach improves flexibility and automation in managing complex systems, making the management process smoother and allowing easy adjustment to changing needs.

kubeopsctl uses YAML files to store configuration data in a human-readable format. These files document important metadata about the objects managed by kubeopsctl, such as pods, services, and deployments.

Highlights

• creating a cluster
• adding nodes to your cluster
• drain nodes
• updating single nodes
• label nodes for zones
• adding platform software into your cluster

2.2 - About KubeOps Compliance Licences

KubeOps Compliance is available with the Compliance licence.

When you create an account, you receive a free 30 day trial version of our licences.

2.3 - Prerequisites

A total of at least seven machines is required:

• 1 Admin
• 3 Control-plane Nodes
• 3 Worker Nodes

Below are the supported operating systems with the associated minimum requirements for CPU, memory, and disk storage:

• For each worker node, an additional unformatted hard disk of 50 GB is required.
  For more information about the hard disks for Rook-Ceph, visit the rook-ceph prerequisites page

• Each machine in the cluster must be configured with the same username. Otherwise, the cluster creation and management process will fail.

• Prerequisites for admin
• Prerequisites for each node

sudo hostnamectl set-hostname <name of node>

sudo hostnamectl set-hostname admin

2.4 - Prepare Cluster

Before you start with KubeOps Compliance, all cluster nodes must be prepared with a consistent baseline configuration. This Prepare Cluster guide walks you through the required system-level steps on both control-plane and worker nodes, as well as the admin node used to manage the cluster. With these preparations done, you will ensure that all nodes behave predictably, can communicate securely with each other, and meet the prerequisites for the subsequent cluster setup procedure.

1. Include package repo on all nodes

To simplify the installation of the kosi and kubeopsctl packages, add the kubeops package repository to your operating system’s package manager on every node.

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6

wget https://packagerepo.kubeops.net/pgp-key.public
cat pgp-key.public | sudo gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/kubeops.gpg] https://packagerepo.kubeops.net/deb stable main' | sudo tee /etc/apt/sources.list.d/kubeops.list
sudo apt update

sudo dnf config-manager --add-repo https://packagerepo.kubeops.net/rpm/kubeops.repo

2. Special operating system adaptations on all nodes

Prepare all nodes with the following operating system adaptations.

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6

# remove unattended upgrades on all nodes!
sudo apt remove unattended-upgrades

# no special adaptions required 

3. Distribute ssh-keys on all nodes

The hostnames of all nodes must be resolvable via DNS.

If you do not run a DNS server, the simplest solution is to add the IP addresses and hostnames to /etc/hosts. Use the same host list on every node, so that each node can resolve the other nodes by its hostname.

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
<admin ip> <admin hostname>
<master1 ip> <master1 hostname>
<master2 ip> <master2 hostname>
<master3 ip> <master3 hostname>
...
<worker1 ip> <worker1 hostname>
<worker2 ip> <worker2 hostname>
<worker3 ip> <worker3 hostname>
...
EOL_ETC_HOSTS

sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
10.2.10.10 admin
10.2.10.110 master1
10.2.10.120 master2
10.2.10.130 master3
10.2.10.210 worker1
10.2.10.220 worker2
10.2.10.230 worker3
EOL_ETC_HOSTS

Create ssh-keys on all nodes

The ssh-keys must belong to the Linux user that will run kubeopsctl later. Run the following ssh commands as this user and without sudo. Only use root if kubeopsctl will also be run as root. This step is required because kubeopsctl needs passwordless SSH from each node to every node in the cluster.

mkdir -p ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 -N ""

Copy each node’s public SSH key to all nodes

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<admin hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<master1 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<master2 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<master3 hostname>
...
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<worker1 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<worker2 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<worker3 hostname>
...

ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@admin
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@master1
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@master2
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@master3
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@worker1
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@worker2
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@worker3

Scan host-keys with name and IP-address on all nodes

This test is important, because kubeopsctl may use either form.

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
ssh-keyscan <admin hostname> >> ~/.ssh/known_hosts
ssh-keyscan <admin ip> >> ~/.ssh/known_hosts
ssh-keyscan <master1 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <master1 ip> >> ~/.ssh/known_hosts
ssh-keyscan <master2 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <master2 ip> >> ~/.ssh/known_hosts
ssh-keyscan <master3 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <master3 ip> >> ~/.ssh/known_hosts
...
ssh-keyscan <worker1 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker1 ip> >> ~/.ssh/known_hosts
ssh-keyscan <worker2 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker2 ip> >> ~/.ssh/known_hosts
ssh-keyscan <worker3 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker3 ip> >> ~/.ssh/known_hosts
...

ssh-keyscan admin >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.10 >> ~/.ssh/known_hosts
ssh-keyscan master1 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.110 >> ~/.ssh/known_hosts
ssh-keyscan master2 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.120 >> ~/.ssh/known_hosts
ssh-keyscan master3 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.130 >> ~/.ssh/known_hosts
ssh-keyscan worker1 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.210 >> ~/.ssh/known_hosts
ssh-keyscan worker2 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.220 >> ~/.ssh/known_hosts
ssh-keyscan worker3 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.230 >> ~/.ssh/known_hosts

Test ssh login without password.

Passwordless SSH login must work from each node to all other nodes.

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
ssh <username>@<admin hostname> exit
ssh <username>@<admin ip> exit
ssh <username>@<master1 hostname> exit
ssh <username>@<master1 ip> exit
ssh <username>@<master2 hostname> exit
ssh <username>@<master2 ip> exit
ssh <username>@<master3 hostname> exit
ssh <username>@<master3 ip> exit
...
ssh <username>@<worker1 hostname> exit
ssh <username>@<worker1 ip> exit
ssh <username>@<worker2 hostname> exit
ssh <username>@<worker2 ip> exit
ssh <username>@<worker3 hostname> exit
ssh <username>@<worker3 ip> exit
...

ssh myuser@admin exit
ssh myuser@10.2.10.10 exit
ssh myuser@master1 exit
ssh myuser@10.2.10.110 exit
ssh myuser@master2 exit
ssh myuser@10.2.10.120 exit
ssh myuser@master3 exit
ssh myuser@10.2.10.130 exit
ssh myuser@worker1 exit
ssh myuser@10.2.10.210 exit
ssh myuser@worker2 exit
ssh myuser@10.2.10.220 exit
ssh myuser@worker3 exit
ssh myuser@10.2.10.230 exit

4. Distribute SUDOERS on all nodes

Replace the username “myuser” with your username, then copy the sudoers file to /etc/sudoers.d/<username> on all nodes.

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/apt/sources.list.d/kubeops.list
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove unattended-upgrades
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove *
myuser ALL=(root) NOPASSWD: /usr/bin/lsof /var/lib/dpkg/lock*

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kosi*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kubeopsctl*.deb


# kube-vip image import
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# Kubeadm init
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# enable UFW
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reset
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 6443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 2379\:2380/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10250/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10259/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10257/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10256/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 30000\:32767/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 179/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 4789/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5473/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 51820/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 22/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5000/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5001/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 7443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw logging low
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw enable
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reload
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw status
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart systemd-networkd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable --now ufw

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# kubeadm init phase and kubeadm token create commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /usr/bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# upgrade nodes
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin
myuser ALL=(root) NOPASSWD: /bin/rm -f /etc/cni/net.d/87-podman-bridge.conflist

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubeadm*
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubectl*
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubelet*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark hold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark unhold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubelet* kubeadm* kubectl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeadm*

# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install passt_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conmon_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install catatonit_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install netavark_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install aardvark-dns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-image_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-common_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containernetworking-plugins_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libsubid4_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install uidmap_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libslirp0_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install slirp4netns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libyajl2_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install crun_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install fuse-overlayfs_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install buildah_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install podman_*.deb

# Podman Remove of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove podman buildah fuse-overlayfs crun libyajl2 slirp4netns libslirp0 uidmap libsubid4 containernetworking-plugins golang-github-containers-common golang-github-containers-image aardvark-dns netavark catatonit conmon passt

# Podman Installation & Update with apt
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install */pia/kosi_*

# runtime setup
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conntrack_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install runc_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containerd_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y iproute-tc

# Remove packages
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove containerd

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard_*.deb wireguard-tools_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard-tools-*.deb systemd-resolved-*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove wireguard wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y wireguard-tools*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y systemd-resolved*

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/local/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf config-manager --add-repo https\://packagerepo.kubeops.net/rpm/kubeops.repo
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kosi*, !/usr/bin/dnf install -y kosi*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kosi*.rpm, !/usr/bin/rpm --install kosi*[[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kubeopsctl*.rpm, !/usr/bin/rpm --install kubeopsctl*[[\:space\:]]*.rpm

# kube-vip/create-manifest.sh.tmpl
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# delete-node/delete-node.sh.tmpl
# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# kubeadm-init/kubeadm-init.sh.tmpl
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/sysconfig/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm
# if Helm is in repository
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y helm* 

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s/package.kosi
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# ssh remote kubeadm commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# local execution of kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
# Adjustments sed -i * 
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# kubeadm upgrade node
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin, !/bin/tar xzf cni-*[[\:space\:]]* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin, !/bin/tar xzf crictl-*[[\:space\:]]* -C /usr/bin
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubeadm*, !/usr/bin/dnf install -y kubeadm*[[\:space\:]]*, !/usr/bin/dnf install -y kubeadm*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*, !/bin/mv /usr/bin/kubelet /usr/bin/kubelet_*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubectl*, !/usr/bin/dnf install -y kubectl*[[\:space\:]]*, !/usr/bin/dnf install -y kubectl*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubelet*, !/usr/bin/dnf install -y kubelet*[[\:space\:]]*, !/usr/bin/dnf install -y kubelet*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubelet-* kubeadm-* kubectl-*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeadm-*


# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install aardvark-dns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install netavark-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install container-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libnet-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-libs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libslirp-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install slirp4netns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install yajl-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install crun-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containers-common-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install fuse-overlayfs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install shadow-utils-subid-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conmon-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install podman-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-*.rpm iproute-tc-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containerd.io-*.rpm

# Podman Remove of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase podman shadow-utils-subid fuse-overlayfs crun containers-common yajl slirp4netns libslirp criu-libs criu libnet container-selinux netavark aardvark-dns passt passt-selinux

# Podman Installation & Update with dnf
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y podman*, !/usr/bin/dnf install -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf update -y podman*,!/usr/bin/dnf update -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install */pia/kosi-*

# Allow installation of container-runtime
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm, !/usr/bin/rpm --install conntrack-tools-*[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm libnetfilter_cthelper-*.rpm libnetfilter_cttimeout-*.rpm libnetfilter_queue-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-tc-*.rpm, !/usr/bin/rpm --install iproute-tc-*[\:space\:]]*.rpm

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y iproute-tc

# Remove RPM packages
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase containerd.io

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/systemd/system/containerd.service.d/
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/systemd/system/containerd.service.d/override.conf*

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install wireguard-tools-*.rpm systemd-resolved-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y wireguard-tools*, !/usr/bin/dnf install -y wireguard-tools*[\:space\:]]*, !/usr/bin/dnf install -y wireguard-tools*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y systemd-resolved*, !/usr/bin/dnf install -y systemd-resolved*[\:space\:]]*, !/usr/bin/dnf install -y systemd-resolved*.rpm

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

5. Configure time synchronization on all nodes

How to install and configure chrony (recommended)

1. Install chrony

Run on every cluster node:

• Ubuntu 24.04
• RHEL 9.6

sudo apt install -y chrony
sudo systemctl enable --now chrony

sudo dnf install -y chrony
sudo systemctl enable --now chronyd

2. Configure NTP servers

Edit the chrony configuration file and configure the NTP servers. Replace or remove existing entries if required by your setup:

server pool.ntp.org iburst
# or your internal NTP servers:
# server 10.2.10.10 iburst

• Ubuntu 24.04
• RHEL 9.6

sudo vi `/etc/chrony/chrony.conf`

sudo vi `/etc/chrony.conf`

Apply changes:

• Ubuntu 24.04
• RHEL 9.6

sudo systemctl restart chrony

sudo systemctl restart chronyd

3. Verify synchronization

chronyc tracking
chronyc sources -v

Expected:

Stratum ≤ 3
Leap Status = Normal
System time offset < 10 ms
One source marked with *

Example:

Reference ID    : 83BC03DC (ntp0.rrze.uni-erlangen.de)
Stratum         : 2
System time     : 0.000718 seconds slow
Last offset     : -0.000214 seconds
RMS offset      : 0.000093 seconds
Leap status     : Normal

Meaning:

• Reference ID → you are synchronizing with ntp0.rrze.uni-erlangen.de → good
• Stratum 2 → completely normal for public NTP servers
• System time slow: 0.0007s → deviation < 1 ms → excellent
• Load/RMS offset → very stable synchronization
• Leap status Normal → no time jumps, everything OK
  

6. Install curl

How to install and configure curl

Install curl

Run on every cluster node:

• Ubuntu 24.04
• RHEL 9.6

sudo apt install -y curl

sudo dnf install -y curl

Once all nodes are prepared, you can start setting up the cluster.

2.5 - Setup Cluster

After completing the cluster preparation steps, you are ready to configure the kubeops tooling and create the Kubernetes cluster from your cluster admin host. This Setup Cluster document describes how to install and configure KOSI and kubeopsctl, prepare the cluster configuration, and then trigger the automated cluster deployment.

Important: the following commands have to be executed on your cluster admin

1. Install KOSI

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6
• Ubuntu 24.04 without package repo
• RHEL 9.6 without package repo

# KOSI-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kosi=<kosi-version>

# KOSI-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kosi-version>

# KOSI-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kosi-version>.deb
sudo dpkg --install <kosi-version>.deb

# KOSI-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kosi
wget https://packagerepo.kubeops.net/rpm/<kosi-version>.rpm
sudo rpm --install <kosi-version>.rpm

2. Set the KUBEOPSROOT env var

Set KUBEOPSROOT and XDG_RUNTIME_DIR in ~/.bashrc

# file ~/.bashrc
# Append these values to the end of your ~/.bashrc file
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

Source .bashrc to apply the values

source ~/.bashrc
echo $KUBEOPSROOT
echo $XDG_RUNTIME_DIR

As a result you should see your KUBEOPSROOT-path two times.

3. Adjust KOSI Configuration

This creates a kubeops directory in your home directory and transfers all necessary files, e.g., the kosi-config and the plugins, to it.

mkdir ~/kubeops
cd ~/kubeops
cp -R /var/kubeops/kosi/ .
cp -R /var/kubeops/plugins/ .

The config.yaml is in your KUBEOPSROOT-path (typically in ~/kubeops/kosi)

• Set hub in your kosi config to hub: https://dispatcher.kubeops.net/v4/dispatcher/

• Set the “plugins”-entry in your kosi config to plugins: /home/<yourUser>/kubeops/plugins, where is changed to your username

# file $KUBEOPSROOT/kosi/config.yaml
apiversion: kubernative/sina/config/v3

spec:
  hub: https://dispatcher.kubeops.net/v4/dispatcher/ # <-- set hub url
  plugins: <your kubeopsroot>/plugins/ # <-- set the path to your plugin folder (~ for home or $KUBEOPSROOT don't work, it has to be the full path)
  workspace: /tmp/kosi/process/
  logging: info
  housekeeping: false
  proxy: false

4. Install KOSI enterprise plugins

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.2.0_Beta0

5. Login with your user

kosi login -u <yourUser>

At this point it is normal if you get the following error message:
Error: The login to registry is temporary not available. Please try again later.
The reason for this is that podman is not yet installed.

6. Install kubeopsctl

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6
• Ubuntu 24.04 without package repo
• RHEL 9.6 without package repo

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install -v <kubeopsctl-version>.rpm

7. Create a cluster-values.yaml configuration file

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local                    # If you have incorporated package repository, replace “local” with "kubeops-repo"
changeCluster: true
zones:
# IMPORTANT: The following part has to be adapted so that every one of your masternodes and workernodes is included
# This file only includes the minimum requirements for the amount of masters and workers and an example usage of zones
# You should adapt this part to your amount of masters and workers and cluster them into as many zones as you like
- name: zone1
  nodes:
  - name: <your master1 hostname>
    iPAddress: <your master1 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker1 hostname>
    iPAddress: <your worker1 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone2
  nodes:
  - name: <your master2 hostname>
    iPAddress: <your master2 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker2 hostname>
    iPAddress: <your worker2 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone3
  nodes:
  - name: <your master3 hostname>
    iPAddress: <your master3 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker3 hostname>
    iPAddress: <your worker3 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: myuser
kubernetesVersion: 1.32.2
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local                 # If you have incorporated package repository, replace “local” with "kubeops-repo"
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: dev07-master1-ubuntu2404
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker1-ubuntu2404
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2
- name: zone2
  nodes:
  - name: dev07-master2-ubuntu2404
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker2-ubuntu2404
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2
- name: zone3
  nodes:
  - name: dev07-master3-ubuntu2404
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker3-ubuntu2404
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

7.1 Using KubeVip in your Cluster (optional)

KubeVip provides a highly available virtual IP for the Kubernetes control plane, giving the cluster a stable API endpoint. In multi-control-plane clusters, the virtual IP can move to another control-plane node if the active one fails. For more information, refer to the official KubeVip documentation here.

To enable KubeVip, set kubeVipEnabled to true and configure virtualIP with an unused IP address in the same network as your control-plane nodes.

If KubeVip is disabled, set kubeVipEnabled to false and use the IP address of the first control-plane node as virtualIP in your cluster-values.yaml in the Setup.

Examples:

kubeVipEnabled: true
virtualIP: <IP in your cluster ip range which is not given yet>

or

kubeVipEnabled: false
virtualIP: <master1 ip>

For highly available clusters, enabling KubeVip is recommended. For test environments or single-control-plane clusters, disabling it is usually sufficient.

8. Pull required KOSI packages

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion <your wanted Kubernetesversion> you can pull an older Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

or

kubeopsctl pull --kubernetesVersion <x.xx.x>

9. Install podman

kosi install -p $KUBEOPSROOT/lima/podman_5.2.2.tgz -f cluster-values.yaml

10. Install helm

kosi install -p $KUBEOPSROOT/lima/helm_v3.16.4.tgz

11. Install kubernetes tools (kubectl)

Make sure the kubernetes version matches the one you pulled before.

kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_<your kubernetes version>.tgz -f cluster-values.yaml

This command also installs kubelet and kubeadm. You can either mask or delete them on your admin as they are not necessary for the cluster creation process. At the first installation there will be an error that shows that the package tries a backup from kublet which does not exists in the moment.

kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_1.32.2.tgz -f cluster-values.yaml

12. Cluster Setup

Make sure that you are logged in on hub and registry.

kosi login -u <your username>

Now the login for hub and registry should be successful!

Make sure that you changed the kosi config.yaml.

cat $KUBEOPSROOT/kosi/config.yaml

Make sure that you pulled all required packages.

ll -1 $KUBEOPSROOT/lima

Install Kubernetes Cluster with kubeopsctl. Cluster setup takes about 10 to 15 minutes.

kubeopsctl apply -f cluster-values.yaml

3 - How to Guides

In the following sections, you will find everything from initial setup and configuration, to advanced tips and tricks that will help you get the most out of the software. Our aim is to assist you in becoming proficient with kubeops, enhancing both your productivity and your user experience.

Lets get started on your journey to mastering kubeops!

3.1 - Documentation for production

3.1.1 - Supported Package Versions — Ubuntu / Debian

This page lists the packages and versions available from the KubeOps package repository for Ubuntu 24.04 (the .deb pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Kubernetes

KubeOps tools

Helm

Container runtime and etcd

Pre-release versions

Pin the installed versions

After the packages are installed, hold them at the installed version so a later apt upgrade cannot upgrade them and break the cluster:

sudo apt-mark hold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

To release a hold later (for a controlled upgrade):

sudo apt-mark unhold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

3.1.2 - Supported Package Versions — Red Hat Enterprise Linux (RHEL)

This page lists the packages and versions available from the KubeOps package repository for RHEL 9.6 / 9.7 (the .rpm pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Kubernetes

Container runtime and etcd

KubeOps tools

Helm

Pre-release versions

Pin the installed versions

On RHEL the KubeOps package repository is configured so that its packages are excluded from normal dnf transactions — the install commands have to pass --disableexcludes=kubeops-repo to override that exclusion. As a result a regular dnf update already leaves these packages untouched, so the cluster is protected out of the box.

To lock the installed versions explicitly, use the versionlock plugin after the packages are installed:

sudo dnf install -y python3-dnf-plugin-versionlock
sudo dnf versionlock add \
  containerd.io podman kubelet kubeadm kubectl kubernetes-cni cri-tools \
  etcdctl etcdutl helm kosi kubeopsctl conntrack-tools

Alternatively, exclude them explicitly on each update:

sudo dnf update --exclude=containerd.io,podman,kubelet,kubeadm,kubectl,kubernetes-cni,cri-tools,etcdctl,etcdutl,helm,kosi,kubeopsctl

Dependencies

The repository also provides the following compatibility/dependency packages (32-bit i686 runtime libraries), which are pulled in automatically: conntrack-tools (1.4.7), glibc (2.34), glibc-gconv-extra (2.34), libgcc (11.5.0), libmnl (1.0.4), libnetfilter_cthelper (1.0.0), libnetfilter_cttimeout (1.0.0), libnetfilter_queue (1.0.5), libnfnetlink (1.0.1).

3.2 - Joining a Node in a Kubernetes cluster

Join Node Process:

Prerequisits

KOSI Login Recommendation

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

ETCD Backup Recommendation

Before performing changes on the control planes, it is recommended to create an ETCD backup. Refer to the official Kubernetes documentation for details here

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull a specific Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

  - name: demo-controlplaneXX
    iPAddress: 10.2.10.XXX
    type: controlplane
    kubeVersion: 1.31.6 

3. Adjust your cluster-values in zone1
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone1.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6       # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true             # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-controlplaneXX   # -> has to be changed
    iPAddress: 10.2.10.XXX      # -> has to be changed
    type: controlplane
    kubeVersion: 1.31.6         # -> check with actual version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6      

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. If you are ready just start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

Example 1: Joining a Worker Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion x.xx.x you can pull other Kubernetes versions.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

  - name: demo-workerXX
    iPAddress: 10.2.10.XX
    type: worker
    kubeVersion: 1.31.6      

3. Adjust your cluster-values in zone2
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone2.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6        # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true              # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
  - name: demo-workerXX          # -> has to be changed
    iPAddress: 10.2.10.XX        # -> has to be changed
    type: worker
    kubeVersion: 1.31.6          # -> check with actual version     
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6      

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. Before joining the node, note the following:

...
packages:
- name: rook-ceph
  enabled: true
...

If you are ready, start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

If you need to apply the CRUSH map adjustment from above, use the following command instead:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

3.3 - Deleting a Node from a Kubernetes cluster

Deleting a Node from a Kubernetes cluster

In rare cases, it may be necessary to remove nodes from a Kubernetes cluster. This how-to guide explains the prerequisites and the key considerations to keep in mind before starting the node removal process.

You can use the following steps to delete nodes from a Kubernetes cluster.

Prerequisites

• In order to run rook-ceph stable for a longer period your cluster needs at least 3 zones with each zone containing at least 1 worker-node

• To check which mon and osd is running on the node you want to delete you can use the command kubectl get pods -n rook-ceph -o wide | grep <node-name> | grep "mon\|osd" | grep -v "osd-prepare" | awk '{print $1}'. As an output you get the mon and the osd running on that node. If you don’t get an output, you don’t have to delete the ressource and can skip to the “delete the node”-section

Worker

Important: Due to rook-ceph, a worker node must not be removed without following the steps below. In this example, worker01 (zone1) is removed from the cluster. Worker01 contains osd.0 and mon-c.

Scale down the rook-ceph-operator deployment to 0

This prevents new MONs or OSDs from being created.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=0

Check which hosts and OSDs belong to each zone

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

ID   CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
 -1         0.21478  root default
 -9         0.04880      zone zone1
 -7         0.04880          host worker01                              # worker01 is being removed
  0    ssd  0.04880              osd.0          up   1.00000  1.00000   # osd.0 is being removed
-15         0.04880          host worker04
  3    ssd  0.04880              osd.3          up   1.00000  1.00000
-11         0.10739      zone zone2
 -3         0.05859          host worker02
  1    ssd  0.05859              osd.1          up   0.95001  1.00000
-13         0.05859      zone zone3
 -5         0.05859          host worker03
  2    ssd  0.05859              osd.2          up   0.95001  1.00000

From this output you can see that osd.0 is part of worker01.

Scale down the OSD deployment

kubectl scale deploy -n rook-ceph rook-ceph-osd-<x> --replicas=0
# Example: kubectl scale deploy -n rook-ceph rook-ceph-osd-0 --replicas=0

Remove the OSD via ceph-tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- bash
# show OSD tree
ceph osd tree
# mark OSD out
ceph osd out <x>
# Example: ceph osd out 0
ceph osd purge <x> --yes-i-really-mean-it
# Example: ceph osd purge 0 --yes-i-really-mean-it
ceph auth del osd.<x>
# adjust CRUSH map
ceph osd crush remove <nodename>
# exit from ceph-tools
exit
# show OSD tree (now without the deleted node)
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

Check the MON running on the node

Before deleting a MON deployment, check which MON is running on the node you want to remove.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon -o wide

Delete OSD and MON deployments

After checking the OSD and MON running on the node, delete the corresponding deployments.

kubectl delete deploy -n rook-ceph rook-ceph-osd-<x> rook-ceph-mon-<y>

kubectl delete deploy -n rook-ceph rook-ceph-osd-0 rook-ceph-mon-c

Remove the deleted mon from the ceph tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm <y>
# verify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm c
# verfify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b
2: [v2:192.168.196.110:3300/0,v1:192.168.196.110:6789/0] mon.c

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b

Delete the node from the kubernetes cluster

• Remove the node you want to delete from the cluster-values.yaml file.
• After updating the file, apply the deletion with the following command:

kubeopsctl apply --delete -f cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6      
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker04
    iPAddress: 10.2.10.214
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6  
  - name: worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6     

kubeopsctl apply --delete -f cluster-values.yaml

Scale the rook-ceph-operator deployment back to 1

This allows a new MON to be created automatically in zone2.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=1

Timing and health checks

The total duration depends on cluster size and node performance. Before proceeding, verify Ceph health and placement groups are clean.

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph status
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph pg stat

Typical duration ranges from 15 to 120 minutes.

If you want to rejoin the same node, reset it to a time prior to joining the cluster. Only this way you can be sure, that no leftovers from the deletion process remain!

3.4 - Single Sign-On with Keycloak

In this guide, you will learn how to implement Single Sign-On (SSO) using Keycloak. We will walk through the complete flow—from understanding SSO for platforms and services such as Rook Ceph, Harbor, OpenSearch, and other Kubernetes applications, to configuring Keycloak, exposing it securely, and integrating it with kubeops.

By the end of this guide, you will be able to:

• Understand how Keycloak enables centralized authentication
• Configure Keycloak for SSO
• Securely expose Keycloak using Kubernetes Ingress and TLS
• Integrate Keycloak with kubeops for authentication and authorization
• Validate and troubleshoot the SSO login flow

Let’s get started on enabling secure and seamless authentication with Keycloak.

3.4.1 - Single Sign-On (SSO) with Keycloak for Harbor

This guide describes how to configure Harbor authentication using Keycloak (OIDC) in a kubeops-managed Kubernetes environment.

Prerequisites

Before proceeding, ensure the following requirements are met:

• A running Kubernetes cluster
• kubectl installed and configured.
• Keycloak is already installed and running
• Keycloak is exposed using Kubernetes Ingress
• A valid DNS record is configured for Keycloak and Harbor
• TLS is enabled with a trusted Certificate Authority (CA)

Step 1: Prepare Keycloak (Realm, User, and Client)

In this step, we configure Keycloak for Harbor SSO. Keycloak is assumed to be already installed, exposed via Ingress, and reachable over HTTPS.

Create Realm

Ensure a realm named kubeops-dashboards exists.
If it does not exist, create it in the Keycloak admin console.

• Realm name: kubeops-dashboards
• Enabled: true

Create User

Ensure a user named kubeops exists in the kubeops-dashboards realm.
If the user does not exist, create it and set credentials.

• Username: kubeops
• Enabled: true
• Set a permanent password

Create Client (Harbor)

Create a client for Harbor in the kubeops-dashboards realm.

• Client ID: harbor
• Client type: OpenID Connect
• Client authentication: Enabled
• Standard flow: Enabled
• Direct access grants: Disabled

Valid Redirect URIs

Add the following redirect URI:

https://<your_DNS_name>/c/oidc/callback

Web Origins

<your_DNS_name>

Client Secret

After creating the client, copy the client secret.
This value will be used in the Harbor configuration:

oidc_client_id: harbor
oidc_client_secret: <CLIENT_SECRET>

Create Secret

kubectl create secret generic oidc-harbor \
    --namespace <your_harbor_namespace> \
    --from-literal=client_id=<your_oidc_client_id> \
    --from-literal=client_secret=<your_oidc_client_secret>

Step 2: Prepare Harbor Values

The following kubeops package configuration enables Harbor and integrates it with Keycloak using OIDC authentication.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1

deleteNs: false
localRegistry: false

packages:
  - name: harbor
    enabled: true
    values:
      standard:
        namespace: <you_harbor_namespace>
        harborpass: "password"
        databasePassword: "password"
        redisPassword: "password"
        externalURL: https://<your_DNS_name>
        nodePort: 30002
        hostname: <your_DNS_name>
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi
              storageClass: "rook-cephfs"
            jobservice:
              jobLog:
                size: 1Gi
                storageClass: "rook-cephfs"
            database:
              size: 1Gi
              storageClass: "rook-cephfs"
            redis:
              size: 1Gi
              storageClass: "rook-cephfs"
            trivy:
              size: 5Gi
              storageClass: "rook-cephfs"

      advanced:
        core:
          extraEnvVars:
            - name: OIDC_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: oidc-harbor 
                  key: client_id
            - name: OIDC_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: oidc-harbor 
                  key: client_secret
            - name: CONFIG_OVERWRITE_JSON
              value: |
                {
                  "auth_mode": "oidc_auth",
                  "oidc_name": "keycloak",
                  "oidc_endpoint": "https://<your_DNS_name>/keycloak/realms/kubeops-dashboards",
                  "oidc_client_id": "$(OIDC_CLIENT_ID)",
                  "oidc_client_secret": "$(OIDC_CLIENT_SECRET)",
                  "oidc_scope": "openid,profile,email",
                  "oidc_verify_cert": true,
                  "oidc_auto_onboard": true
                }                

Notes

• Ensure the OIDC client in Keycloak matches the oidc_client_id and oidc_client_secret values.
• The externalURL and hostname must match the Harbor DNS name exactly.
• oidc_auto_onboard: true allows users to be created automatically in Harbor upon first login.

3.4.2 - Single Sign-On (SSO) with Keycloak for KubeOps Dashboard

This guide describes how to configure KubeOps Dashboard using Keycloak (OIDC) in a kubeops-managed Kubernetes environment.

Prerequisites

Before proceeding, ensure the following requirements are met:

• A running Kubernetes cluster
• kubectl installed and configured.
• Keycloak is already installed and running

Step 1: Extract Keycloak CA certificate

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

• On your admin host, run the OpenSSL command (kept exactly as provided):

  openssl s_client -showcerts -connect <your_DNS_name>:443 /dev/null | openssl x509 -outform PEM > keycloak-ca.crt

• Copy the CA certificate to each master

  scp @:/etc/kubernetes/pki/

Step 2: Update kube-apiserver yaml

On every master, edit the yaml : /etc/kubernetes/manifests/kube-apiserver.yaml

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

spec:

  containers:

  - command:


    - --oidc-issuer-url=https://<your_DNS_name>/keycloak/realms/master

    - --oidc-client-id=headlamp

    - --oidc-username-claim=preferred_username

    - --oidc-groups-claim=groups

    - "--oidc-username-prefix=oidc:"

    - "--oidc-groups-prefix=oidc:"

    - --oidc-ca-file=/etc/kubernetes/pki/keycloak-ca.crt

Step 3: Create a Keycloak client for Headlamp

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

• Create a client for headlamp

  • Client ID: headlamp
  • Client type: OpenID Connect
  • Access type: Confidential
  • Client authentication: Enabled
  • Standard flow: Enabled
  • Direct access grants: Disabled

• Valid Redirect URIs

  Add the following redirect URI:

  https://headlamp/<your_DNS_name>/*
  

• Web Origins

  <your_DNS_name>
  

Step 4: Create a client scope for Headlamp

• Create a client scope

  • Assigned Client Scope : headlamp-dedicated

• For groups, use the Group Mapper in Keycloak:

  • Mapper Type: groups
  • Name: groups
  • Token Claim Name: groups
  • Add to ID token: ON
  • Add to access token: ON
  • Add to user info: ON
  • Add to token introspection: ON

Step 5: Create a user Group and user in Keycloak

Create a group named headlamp (if doesn’t exist already) and user under the group.

Step 6: Create ClusterRoleBinding for Headlamp group

1.Use following yaml to create ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: headlamp-admin-user
subjects:
- kind: Group
  name: "oidc:headlamp" # Der 'sub' oder 'preferred_username' from the Keycloak-Token
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

The name “oidc:headlamp” needs to be the same as the group name.

2. Apply the ClusterRoleBinding file

    kubectl apply -f headlamp-clusterrolebinding.yaml

Step 7: Get client secret

After creating the client, copy the client secret.
This value will be used in the next step.

Step 8: Prepare Headlamp values (enterprise.yaml)

configure enterprise-yaml

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

packages:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      service:
        nodePort: 30007
      hostname: "<your_DNS_name>"
      path: "/"
    advanced:
      config:
        extraArgs:
          - "--in-cluster"
          - "--plugins-dir=/headlamp/plugins"
          - "--oidc-client-id=headlamp"
          - "--oidc-idp-issuer-url=https://<your_DNS_name>/keycloak/realms/master"
          - "--oidc-scopes=openid,profile,email"
          - "--insecure-ssl"
          - "--oidc-client-secret=<client-secret>"

Replace with the secret retrieved in Step 7.
-oidc-client-id must match the Keycloak client name (headlamp).

Step 9: Install Headlamp

Deploy Headlamp with the updated enterprise.yaml.

3.4.3 - Single Sign-On (SSO) with Keycloak for OpenSearch

This guide describes how to log in to OpenSearch Dashboards using Keycloak (OIDC) in a kubeops-managed Kubernetes environment.

Unlike some other integrations, you do not have to create the Keycloak realm, client, roles or mapper by hand — the opensearch-dashboards package provisions them automatically during install/update. Username/password login remains available as a fallback.

Prerequisites

• A running Kubernetes cluster with kubectl configured
• Keycloak is installed and running (the keycloak package), exposed via Ingress and reachable over HTTPS
• A valid DNS record and a trusted TLS certificate (e.g. Let’s Encrypt) for the ingress hostname
• The OpenSearch hostname is set to that public host and is identical for the opensearch-os and opensearch-dashboards packages

Step 1: Configure the packages

Set the shared hostname and (optionally) the adminPassword for both OpenSearch packages in your enterprise-values.yaml:

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1

deleteNs: false
localRegistry: false

packages:
  - name: opensearch-os
    enabled: true
    values:
      standard:
        namespace: logging
        hostname: <your_DNS_name>          # must match opensearch-dashboards and Keycloak host
        adminPassword: "<admin-password>"  # optional, default Password@@123456
  - name: opensearch-dashboards
    enabled: true
    values:
      standard:
        namespace: logging
        nodePort: 30050
        hostname: <your_DNS_name>          # must match opensearch-os
        adminPassword: "<admin-password>"  # must match opensearch-os