KubeOpsDoku – Getting-Started

Kubeopsctl-2.0: About KubeOps Compliance

Mon, 01 Jan 0001 00:00:00 +0000

What is KubeOps Compliance?

kubeopsctl serves as a versatile utility designed specifically to efficiently manage both the configuration and status of a cluster.

With its capabilities, users can articulate their desired cluster state in detail, outlining configurations and specifications.

Subsequently, kubeopsctl orchestrates the creation of a cluster that precisely matches the specified state, ensuring alignment between intentions and operational reality.

Why use KubeOps Compliance?

In kubeopsctl, configuration management involves defining, maintaining, and updating the desired state of a cluster, including configurations for nodes, pods, services, and other resources in the application environment.

The main goal of kubeopsctl is to match the clusters actual state with the desired state specified in the configuration files. With a declarative model, kubeopsctl enables administrators to express their desired system setup, focusing on „what“ they want rather than „how“ to achieve it.

This approach improves flexibility and automation in managing complex systems, making the management process smoother and allowing easy adjustment to changing needs.

kubeopsctl uses YAML files to store configuration data in a human-readable format. These files document important metadata about the objects managed by kubeopsctl, such as pods, services, and deployments.

Highlights

creating a cluster
adding nodes to your cluster
drain nodes
updating single nodes
label nodes for zones
adding platform software into your cluster

Kubeopsctl-2.0: About KubeOps Compliance Licences

Mon, 01 Jan 0001 00:00:00 +0000

KubeOps Compliance is available with the Compliance licence.

When you create an account, you receive a free 30 day trial version of our licences.

Note The free license includes only 1 control plane and 3 worker nodes.

Functions	Compliance
Cluster Lifecycle Management	✓
Build Cluster	✓
Build Airgap Cluster	✓
Security Hardening	✓
Price	Contact Sales

Kubeopsctl-2.0: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

A total of at least seven machines is required:

1 Admin
3 Control-plane Nodes
3 Worker Nodes

Below are the supported operating systems with the associated minimum requirements for CPU, memory, and disk storage:

OS	Minimum Requirements
Red Hat Enterprise Linux 9.6 and 9.7	8 CPU cores, 16 GB memory, 50GB disk storage
Ubuntu 24.04.02	8 CPU cores, 16 GB memory, 50GB disk storage

⚠ Warning Only RHEL 9.6 and 9.7 are supported. Make sure the OS version is pinned to prevent accidental updates to an unsupported version, as this may break the cluster.

For each worker node, an additional unformatted hard disk of 50 GB is required.
For more information about the hard disks for Rook-Ceph, visit the rook-ceph prerequisites page
Each machine in the cluster must be configured with the same username. Otherwise, the cluster creation and management process will fail.

Prerequisites for admin

The following requirements must be fulfilled on the admin machine.

You need an internet connection to use the default KubeOps Registry.

A local registry can be used in the Airgap environment. KubeOps supports only secure registries. If you use an insecure registry, it is important to configure it accordingly in the registry configuration file (/etc/containers/registries.conf for podman, /etc/docker/deamon.json for docker)

Prerequisites for each node

The following requirements must be fulfilled on each node.

You must assign a unique, lowercase hostname to every machine you use.

It is recommended to use self-explanatory hostnames.

To set the hostname on your machine, use the following command:

sudo hostnamectl set-hostname <name of node>

Example

Use the commands below to set the hostnames on each machine, for example to admin, master, node1 node2 (lowercase letters and numbers only).

sudo hostnamectl set-hostname admin

Optional: To use encrypted traffic inside the cluster, follow these steps:

For RHEL machines, you need to import the ELRepo Secure Boot key into your system. You can find a detailed explanation and comprehensive instructions in our how-to guide:Importing the Secure-Boot key

This is only necessary if your system has Secure Boot enabled. If this is not the case, or if you do not want to use encryption at all, you can skip this step.

Ensure that either iptables or nftables is installed on the system.

Kubeopsctl-2.0: Prepare Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Before you start with KubeOps Compliance, all cluster nodes must be prepared with a consistent baseline configuration. This Prepare Cluster guide walks you through the required system-level steps on both control-plane and worker nodes, as well as the admin node used to manage the cluster. With these preparations done, you will ensure that all nodes behave predictably, can communicate securely with each other, and meet the prerequisites for the subsequent cluster setup procedure.

⚠ Warning This guide assumes that you are a normal user and are not acting as the root user.

1. Include package repo on all nodes

To simplify the installation of the kosi and kubeopsctl packages, add the kubeops package repository to your operating system’s package manager on every node.

wget https://packagerepo.kubeops.net/pgp-key.public
cat pgp-key.public | sudo gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/kubeops.gpg] https://packagerepo.kubeops.net/deb stable main' | sudo tee /etc/apt/sources.list.d/kubeops.list
sudo apt update

sudo dnf config-manager --add-repo https://packagerepo.kubeops.net/rpm/kubeops.repo

2. Special operating system adaptations on all nodes

Prepare all nodes with the following operating system adaptations.

# remove unattended upgrades on all nodes!
sudo apt remove unattended-upgrades

# no special adaptions required

3. Distribute ssh-keys on all nodes

The hostnames of all nodes must be resolvable via DNS.

If you do not run a DNS server, the simplest solution is to add the IP addresses and hostnames to /etc/hosts. Use the same host list on every node, so that each node can resolve the other nodes by its hostname.

Note Run these commands on all nodes

⚠ Warning Adapt the following command to your environment before using it.

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
<admin ip> <admin hostname>
<master1 ip> <master1 hostname>
<master2 ip> <master2 hostname>
<master3 ip> <master3 hostname>
...
<worker1 ip> <worker1 hostname>
<worker2 ip> <worker2 hostname>
<worker3 ip> <worker3 hostname>
...
EOL_ETC_HOSTS

Full Example

sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
10.2.10.10 admin
10.2.10.110 master1
10.2.10.120 master2
10.2.10.130 master3
10.2.10.210 worker1
10.2.10.220 worker2
10.2.10.230 worker3
EOL_ETC_HOSTS

Create ssh-keys on all nodes

The ssh-keys must belong to the Linux user that will run kubeopsctl later. Run the following ssh commands as this user and without sudo. Only use root if kubeopsctl will also be run as root. This step is required because kubeopsctl needs passwordless SSH from each node to every node in the cluster.

Note Run these commands on all nodes

mkdir -p ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 -N ""

Copy each node’s public SSH key to all nodes

Note Run these commands on all nodes

⚠ Warning Adapt the following commands to your environment before using them. Replace <username> with the Linux user that will run kubeopsctl.

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<admin hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<master1 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<master2 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<master3 hostname>
...
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<worker1 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<worker2 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<worker3 hostname>
...

Full Example

ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@admin
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@master1
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@master2
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@master3
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@worker1
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@worker2
ssh-copy-id -i ~/.ssh/id_ed25519.pub myuser@worker3

Scan host-keys with name and IP-address on all nodes

This test is important, because kubeopsctl may use either form.

Note Run these commands on all nodes

⚠ Warning Adapt the following commands to your environment before using them.

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
ssh-keyscan <admin hostname> >> ~/.ssh/known_hosts
ssh-keyscan <admin ip> >> ~/.ssh/known_hosts
ssh-keyscan <master1 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <master1 ip> >> ~/.ssh/known_hosts
ssh-keyscan <master2 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <master2 ip> >> ~/.ssh/known_hosts
ssh-keyscan <master3 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <master3 ip> >> ~/.ssh/known_hosts
...
ssh-keyscan <worker1 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker1 ip> >> ~/.ssh/known_hosts
ssh-keyscan <worker2 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker2 ip> >> ~/.ssh/known_hosts
ssh-keyscan <worker3 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker3 ip> >> ~/.ssh/known_hosts
...

Full Example

ssh-keyscan admin >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.10 >> ~/.ssh/known_hosts
ssh-keyscan master1 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.110 >> ~/.ssh/known_hosts
ssh-keyscan master2 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.120 >> ~/.ssh/known_hosts
ssh-keyscan master3 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.130 >> ~/.ssh/known_hosts
ssh-keyscan worker1 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.210 >> ~/.ssh/known_hosts
ssh-keyscan worker2 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.220 >> ~/.ssh/known_hosts
ssh-keyscan worker3 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.230 >> ~/.ssh/known_hosts

Test ssh login without password.

Passwordless SSH login must work from each node to all other nodes.

Note Run these commands on all nodes

⚠ Warning Adapt the following commands to your environment before using them. Replace <username> with the Linux user that will run kubeopsctl.

# IMPORTANT: The following command has to be adapted so that every admin, masternode and workernode is included
ssh <username>@<admin hostname> exit
ssh <username>@<admin ip> exit
ssh <username>@<master1 hostname> exit
ssh <username>@<master1 ip> exit
ssh <username>@<master2 hostname> exit
ssh <username>@<master2 ip> exit
ssh <username>@<master3 hostname> exit
ssh <username>@<master3 ip> exit
...
ssh <username>@<worker1 hostname> exit
ssh <username>@<worker1 ip> exit
ssh <username>@<worker2 hostname> exit
ssh <username>@<worker2 ip> exit
ssh <username>@<worker3 hostname> exit
ssh <username>@<worker3 ip> exit
...

Full Example

ssh myuser@admin exit
ssh myuser@10.2.10.10 exit
ssh myuser@master1 exit
ssh myuser@10.2.10.110 exit
ssh myuser@master2 exit
ssh myuser@10.2.10.120 exit
ssh myuser@master3 exit
ssh myuser@10.2.10.130 exit
ssh myuser@worker1 exit
ssh myuser@10.2.10.210 exit
ssh myuser@worker2 exit
ssh myuser@10.2.10.220 exit
ssh myuser@worker3 exit
ssh myuser@10.2.10.230 exit

4. Distribute SUDOERS on all nodes

⚠ Warning Adapt this file to your environment before using it.

Replace the username “myuser” with your username, then copy the sudoers file to /etc/sudoers.d/<username> on all nodes.

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/apt/sources.list.d/kubeops.list
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove unattended-upgrades
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove *
myuser ALL=(root) NOPASSWD: /usr/bin/lsof /var/lib/dpkg/lock*

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kosi*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kubeopsctl*.deb


# kube-vip image import
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# Kubeadm init
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# enable UFW
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reset
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 6443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 2379\:2380/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10250/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10259/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10257/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10256/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 30000\:32767/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 179/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 4789/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5473/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 51820/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 22/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5000/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5001/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 7443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw logging low
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw enable
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reload
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw status
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart systemd-networkd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable --now ufw

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# kubeadm init phase and kubeadm token create commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /usr/bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# upgrade nodes
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin
myuser ALL=(root) NOPASSWD: /bin/rm -f /etc/cni/net.d/87-podman-bridge.conflist

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubeadm*
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubectl*
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubelet*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark hold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark unhold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubelet* kubeadm* kubectl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeadm*

# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install passt_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conmon_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install catatonit_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install netavark_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install aardvark-dns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-image_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-common_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containernetworking-plugins_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libsubid4_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install uidmap_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libslirp0_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install slirp4netns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libyajl2_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install crun_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install fuse-overlayfs_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install buildah_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install podman_*.deb

# Podman Remove of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove podman buildah fuse-overlayfs crun libyajl2 slirp4netns libslirp0 uidmap libsubid4 containernetworking-plugins golang-github-containers-common golang-github-containers-image aardvark-dns netavark catatonit conmon passt

# Podman Installation & Update with apt
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install */pia/kosi_*

# runtime setup
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conntrack_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install runc_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containerd_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y iproute-tc

# Remove packages
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove containerd

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard_*.deb wireguard-tools_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard-tools-*.deb systemd-resolved-*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove wireguard wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y wireguard-tools*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y systemd-resolved*

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/local/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf config-manager --add-repo https\://packagerepo.kubeops.net/rpm/kubeops.repo
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kosi*, !/usr/bin/dnf install -y kosi*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kosi*.rpm, !/usr/bin/rpm --install kosi*[[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kubeopsctl*.rpm, !/usr/bin/rpm --install kubeopsctl*[[\:space\:]]*.rpm

# kube-vip/create-manifest.sh.tmpl
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# delete-node/delete-node.sh.tmpl
# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# kubeadm-init/kubeadm-init.sh.tmpl
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/sysconfig/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm
# if Helm is in repository
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y helm* 

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s/package.kosi
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# ssh remote kubeadm commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# local execution of kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
# Adjustments sed -i * 
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# kubeadm upgrade node
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin, !/bin/tar xzf cni-*[[\:space\:]]* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin, !/bin/tar xzf crictl-*[[\:space\:]]* -C /usr/bin
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubeadm*, !/usr/bin/dnf install -y kubeadm*[[\:space\:]]*, !/usr/bin/dnf install -y kubeadm*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*, !/bin/mv /usr/bin/kubelet /usr/bin/kubelet_*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubectl*, !/usr/bin/dnf install -y kubectl*[[\:space\:]]*, !/usr/bin/dnf install -y kubectl*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubelet*, !/usr/bin/dnf install -y kubelet*[[\:space\:]]*, !/usr/bin/dnf install -y kubelet*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubelet-* kubeadm-* kubectl-*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeadm-*


# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install aardvark-dns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install netavark-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install container-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libnet-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-libs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libslirp-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install slirp4netns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install yajl-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install crun-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containers-common-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install fuse-overlayfs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install shadow-utils-subid-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conmon-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install podman-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-*.rpm iproute-tc-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containerd.io-*.rpm

# Podman Remove of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase podman shadow-utils-subid fuse-overlayfs crun containers-common yajl slirp4netns libslirp criu-libs criu libnet container-selinux netavark aardvark-dns passt passt-selinux

# Podman Installation & Update with dnf
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y podman*, !/usr/bin/dnf install -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf update -y podman*,!/usr/bin/dnf update -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install */pia/kosi-*

# Allow installation of container-runtime
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm, !/usr/bin/rpm --install conntrack-tools-*[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm libnetfilter_cthelper-*.rpm libnetfilter_cttimeout-*.rpm libnetfilter_queue-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-tc-*.rpm, !/usr/bin/rpm --install iproute-tc-*[\:space\:]]*.rpm

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y iproute-tc

# Remove RPM packages
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase containerd.io

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/systemd/system/containerd.service.d/
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/systemd/system/containerd.service.d/override.conf*

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install wireguard-tools-*.rpm systemd-resolved-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y wireguard-tools*, !/usr/bin/dnf install -y wireguard-tools*[\:space\:]]*, !/usr/bin/dnf install -y wireguard-tools*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y systemd-resolved*, !/usr/bin/dnf install -y systemd-resolved*[\:space\:]]*, !/usr/bin/dnf install -y systemd-resolved*.rpm

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

5. Configure time synchronization on all nodes

Note Make sure all nodes are time synchronized with an NTP server.
Install chrony on all nodes.

How to install and configure chrony (recommended)

Install chrony

Run on every cluster node:

sudo apt install -y chrony
sudo systemctl enable --now chrony

sudo dnf install -y chrony
sudo systemctl enable --now chronyd

Configure NTP servers

Edit the chrony configuration file and configure the NTP servers. Replace or remove existing entries if required by your setup:

server pool.ntp.org iburst
# or your internal NTP servers:
# server 10.2.10.10 iburst

sudo vi `/etc/chrony/chrony.conf`

sudo vi `/etc/chrony.conf`

Apply changes:

sudo systemctl restart chrony

sudo systemctl restart chronyd

Verify synchronization

chronyc tracking
chronyc sources -v

Expected:

Stratum ≤ 3
Leap Status = Normal
System time offset < 10 ms
One source marked with *

Example:

Reference ID    : 83BC03DC (ntp0.rrze.uni-erlangen.de)
Stratum         : 2
System time     : 0.000718 seconds slow
Last offset     : -0.000214 seconds
RMS offset      : 0.000093 seconds
Leap status     : Normal

Meaning:

Reference ID → you are synchronizing with ntp0.rrze.uni-erlangen.de → good
Stratum 2 → completely normal for public NTP servers
System time slow: 0.0007s → deviation < 1 ms → excellent
Load/RMS offset → very stable synchronization
Leap status Normal → no time jumps, everything OK

6. Install curl

How to install and configure curl

Install curl

Run on every cluster node:

sudo apt install -y curl

sudo dnf install -y curl

Once all nodes are prepared, you can start setting up the cluster.

Kubeopsctl-2.0: Setup Cluster

Mon, 01 Jan 0001 00:00:00 +0000

After completing the cluster preparation steps, you are ready to configure the kubeops tooling and create the Kubernetes cluster from your cluster admin host. This Setup Cluster document describes how to install and configure KOSI and kubeopsctl, prepare the cluster configuration, and then trigger the automated cluster deployment.

Important: the following commands have to be executed on your cluster admin

⚠ Warning This guide assumes that you are a normal user and are not acting as the root user.

1. Install KOSI

# KOSI-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kosi=<kosi-version>

# KOSI-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kosi-version>


# KOSI-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kosi-version>.deb
sudo dpkg --install <kosi-version>.deb

# KOSI-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kosi
wget https://packagerepo.kubeops.net/rpm/<kosi-version>.rpm
sudo rpm --install <kosi-version>.rpm

2. Set the KUBEOPSROOT env var

Set KUBEOPSROOT and XDG_RUNTIME_DIR in ~/.bashrc

⚠ Warning This file entry has to be adapted to your values before usage.

Note These are not console commands. The following entries must be added to the end of your ~/.bashrc file.

# file ~/.bashrc
# Append these values to the end of your ~/.bashrc file
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

Source .bashrc to apply the values

source ~/.bashrc
echo $KUBEOPSROOT
echo $XDG_RUNTIME_DIR

As a result you should see your KUBEOPSROOT-path two times.

3. Adjust KOSI Configuration

This creates a kubeops directory in your home directory and transfers all necessary files, e.g., the kosi-config and the plugins, to it.

mkdir ~/kubeops
cd ~/kubeops
cp -R /var/kubeops/kosi/ .
cp -R /var/kubeops/plugins/ .

The config.yaml is in your KUBEOPSROOT-path (typically in ~/kubeops/kosi)

Set hub in your kosi config to hub: https://dispatcher.kubeops.net/v4/dispatcher/

⚠ Warning This file entry has to be adapted to your values before usage.

Set the “plugins”-entry in your kosi config to plugins: /home/<yourUser>/kubeops/plugins, where is changed to your username

⚠ Warning This file has to be adapted to your values before usage.

# file $KUBEOPSROOT/kosi/config.yaml
apiversion: kubernative/sina/config/v3

spec:
  hub: https://dispatcher.kubeops.net/v4/dispatcher/ # <-- set hub url
  plugins: <your kubeopsroot>/plugins/ # <-- set the path to your plugin folder (~ for home or $KUBEOPSROOT don't work, it has to be the full path)
  workspace: /tmp/kosi/process/
  logging: info
  housekeeping: false
  proxy: false

4. Install KOSI enterprise plugins

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.2.0_Beta0

⚠ Warning This command has to be adapted to your values before usage.

kosi login -u <yourUser>

At this point it is normal if you get the following error message:
Error: The login to registry is temporary not available. Please try again later.
The reason for this is that podman is not yet installed.

6. Install kubeopsctl

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install -v <kubeopsctl-version>.rpm

7. Create a cluster-values.yaml configuration file

⚠ Warning This file has to be adapted to your values before usage.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local                    # If you have incorporated package repository, replace “local” with "kubeops-repo"
changeCluster: true
zones:
# IMPORTANT: The following part has to be adapted so that every one of your masternodes and workernodes is included
# This file only includes the minimum requirements for the amount of masters and workers and an example usage of zones
# You should adapt this part to your amount of masters and workers and cluster them into as many zones as you like
- name: zone1
  nodes:
  - name: <your master1 hostname>
    iPAddress: <your master1 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker1 hostname>
    iPAddress: <your worker1 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone2
  nodes:
  - name: <your master2 hostname>
    iPAddress: <your master2 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker2 hostname>
    iPAddress: <your worker2 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone3
  nodes:
  - name: <your master3 hostname>
    iPAddress: <your master3 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker3 hostname>
    iPAddress: <your worker3 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>

Full Example

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: myuser
kubernetesVersion: 1.32.2
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local                 # If you have incorporated package repository, replace “local” with "kubeops-repo"
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: dev07-master1-ubuntu2404
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker1-ubuntu2404
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2
- name: zone2
  nodes:
  - name: dev07-master2-ubuntu2404
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker2-ubuntu2404
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2
- name: zone3
  nodes:
  - name: dev07-master3-ubuntu2404
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker3-ubuntu2404
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning value of ‘packageRepository’ in above yaml file should be changed to ‘kubeops-repo’, in case you have incorporated package repository.

7.1 Using KubeVip in your Cluster (optional)

KubeVip provides a highly available virtual IP for the Kubernetes control plane, giving the cluster a stable API endpoint. In multi-control-plane clusters, the virtual IP can move to another control-plane node if the active one fails. For more information, refer to the official KubeVip documentation here.

To enable KubeVip, set kubeVipEnabled to true and configure virtualIP with an unused IP address in the same network as your control-plane nodes.

If KubeVip is disabled, set kubeVipEnabled to false and use the IP address of the first control-plane node as virtualIP in your cluster-values.yaml in the Setup.

Examples:

kubeVipEnabled: true
virtualIP: <IP in your cluster ip range which is not given yet>

kubeVipEnabled: false
virtualIP: <master1 ip>

Note The virtual IP must be in the same network as the machines and must not already be in use.

For highly available clusters, enabling KubeVip is recommended. For test environments or single-control-plane clusters, disabling it is usually sufficient.

8. Pull required KOSI packages

⚠ Warning This command should be adapted to your value before usage.

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion <your wanted Kubernetesversion> you can pull an older Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

kubeopsctl pull --kubernetesVersion <x.xx.x>

9. Install podman

kosi install -p $KUBEOPSROOT/lima/podman_5.2.2.tgz -f cluster-values.yaml

⚠ Warning Updating podman, e.g, by using apt update or dnf update, might break your cluster.
On RHEL9, to update your applications but exclude podman you can run the following command: sudo dnf update --exclude=podman
On Ubuntu, you can hold the package with the command sudo apt-mark hold podman which makes apt skip podman on updates.

10. Install helm

kosi install -p $KUBEOPSROOT/lima/helm_v3.16.4.tgz

11. Install kubernetes tools (kubectl)

⚠ Warning This command has to be adapted to your values before usage.

Make sure the kubernetes version matches the one you pulled before.

kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_<your kubernetes version>.tgz -f cluster-values.yaml

This command also installs kubelet and kubeadm. You can either mask or delete them on your admin as they are not necessary for the cluster creation process. At the first installation there will be an error that shows that the package tries a backup from kublet which does not exists in the moment.

Full Example

kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_1.32.2.tgz -f cluster-values.yaml

12. Cluster Setup

⚠ Warning This command has to be adapted to your values before usage.

Make sure that you are logged in on hub and registry.

kosi login -u <your username>

Now the login for hub and registry should be successful!

Make sure that you changed the kosi config.yaml.

cat $KUBEOPSROOT/kosi/config.yaml

Make sure that you pulled all required packages.

ll -1 $KUBEOPSROOT/lima

Install Kubernetes Cluster with kubeopsctl. Cluster setup takes about 10 to 15 minutes.

kubeopsctl apply -f cluster-values.yaml

Kubeopsctl-2.0: Prepare Cluster (MicroOS)

Mon, 01 Jan 0001 00:00:00 +0000

Before you start with KubeOps Compliance, all cluster nodes must be prepared with a consistent baseline configuration. This Prepare Cluster guide walks you through the required system-level steps on both controlplane and worker nodes, as well as the admin node used to manage the cluster. With these preparations done, you will ensure that all nodes behave predictably, can communicate securely with each other, and meet the prerequisites for the subsequent cluster setup procedure.

⚠ Warning This guide assumes that certain steps are performed as a privileged user. You need sudo privileges for that.

1. Create a user to work with

A user is created on each node, which is then used to perform the installation.
If no SSH keys have been set up for root, you can temporarily allow root login.

⚠ Warning Once the user have been created, this change must be reversed in production!

Note Run these commands on all nodes as root

echo -e "PermitRootLogin yes\nPasswordAuthentication yes" > /etc/ssh/sshd_config.d/permit_root_login.conf
systemctl restart sshd
useradd myuser # You can choose your own username instead of myuser
passwd myuser  # Set password for myuser

2. Hostnames of all nodes must be resolvable via DNS

If you do not run a DNS server, the easiest solution is to enter the IP-addresses and hostnames in etc/hosts

⚠ Warning From this point on, this guide uses the user ‘myuser’

Note Run these commands on all nodes

# IMPORTANT: The following command has to be adapted so that every admin, controlplane and worker node is included
sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
<admin ip> <admin hostname>
<controlplane01 ip> <controlplane01 hostname>
<controlplane02 ip> <controlplane02 hostname>
<controlplane03 ip> <controlplane03 hostname>
...
<worker01 ip> <worker01 hostname>
<worker02 ip> <worker02 hostname>
<worker03 ip> <worker03 hostname>
...
EOL_ETC_HOSTS

Full Example

sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
10.2.10.10 admin
10.2.10.110 node01
10.2.10.120 node02
10.2.10.130 node03
10.2.10.210 node04
10.2.10.220 node05
10.2.10.230 node06
EOL_ETC_HOSTS

3. Distribute ssh-keys on all nodes

Create ssh-keys on all nodes

Note Run these commands on all nodes

ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 -N ""

Copy public ssh-key on all nodes

Note Run these commands on all nodes

⚠ Warning Adapt the following commands to your environment before using them.

# IMPORTANT: The following command has to be adapted so that every admin, controlplane and worker node is included
ssh-copy-id -i ~/.ssh/id_ed25519 <admin hostname>
ssh-copy-id -i ~/.ssh/id_ed25519 <controlplane01 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519 <controlplane02 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519 <controlplane03 hostname>
...
ssh-copy-id -i ~/.ssh/id_ed25519 <worker01 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519 <worker02 hostname>
ssh-copy-id -i ~/.ssh/id_ed25519 <worker03 hostname>
...

Full Example

ssh-copy-id -i ~/.ssh/id_ed25519 admin
ssh-copy-id -i ~/.ssh/id_ed25519 node01
ssh-copy-id -i ~/.ssh/id_ed25519 node02
ssh-copy-id -i ~/.ssh/id_ed25519 node03
ssh-copy-id -i ~/.ssh/id_ed25519 node04
ssh-copy-id -i ~/.ssh/id_ed25519 node05
ssh-copy-id -i ~/.ssh/id_ed25519 node06

Scan host-keys with name and IP-address on all nodes

Note Run these commands on all nodes

⚠ Warning Adapt the following commands to your environment before using them.

# IMPORTANT: The following command has to be adapted so that every admin, controlplane and worker node is included
ssh-keyscan <admin hostname> >> ~/.ssh/known_hosts
ssh-keyscan <admin ip> >> ~/.ssh/known_hosts
ssh-keyscan <controlplane01 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <controlplane01 ip> >> ~/.ssh/known_hosts
ssh-keyscan <controlplane02 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <controlplane02 ip> >> ~/.ssh/known_hosts
ssh-keyscan <controlplane03 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <controlplane03 ip> >> ~/.ssh/known_hosts
...
ssh-keyscan <worker01 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker01 ip> >> ~/.ssh/known_hosts
ssh-keyscan <worker02 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker02 ip> >> ~/.ssh/known_hosts
ssh-keyscan <worker03 hostname> >> ~/.ssh/known_hosts
ssh-keyscan <worker03 ip> >> ~/.ssh/known_hosts
...

Full Example

ssh-keyscan admin >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.10 >> ~/.ssh/known_hosts
ssh-keyscan node01 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.110 >> ~/.ssh/known_hosts
ssh-keyscan node02 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.120 >> ~/.ssh/known_hosts
ssh-keyscan node03 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.130 >> ~/.ssh/known_hosts
ssh-keyscan node04 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.210 >> ~/.ssh/known_hosts
ssh-keyscan node05 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.220 >> ~/.ssh/known_hosts
ssh-keyscan node06 >> ~/.ssh/known_hosts
ssh-keyscan 10.2.10.230 >> ~/.ssh/known_hosts

Test ssh login without password.

Passwordless SSH login must work from each node to all other nodes.

Note Run these commands on all nodes

⚠ Warning Adapt the following commands to your environment before using them.

# IMPORTANT: The following command has to be adapted so that every admin, controlplane and worker node is included
ssh <admin hostname> exit
ssh <admin ip> exit
ssh <controlplane01 hostname> exit
ssh <controlplane01 ip> exit
ssh <controlplane02 hostname> exit
ssh <controlplane02 ip> exit
ssh <controlplane03 hostname> exit
ssh <controlplane03 ip> exit
...
ssh <worker01 hostname> exit
ssh <worker01 ip> exit
ssh <worker02 hostname> exit
ssh <worker02 ip> exit
ssh <worker03 hostname> exit
ssh <worker03 ip> exit
...

Full Example

ssh admin exit
ssh 10.2.10.10 exit
ssh node01 exit
ssh 10.2.10.110 exit
ssh node02 exit
ssh 10.2.10.120 exit
ssh node03 exit
ssh 10.2.10.130 exit
ssh node04 exit
ssh 10.2.10.210 exit
ssh node05 exit
ssh 10.2.10.220 exit
ssh node06 exit
ssh 10.2.10.230 exit

4. Distribute SUDOERS on all nodes

⚠ Warning Adapt this file to your environment before using it.

Replace the username “myuser” with your username and copy the sudoers file to /etc/sudoers.d/<username> on all nodes.

myuser ALL=(ALL) NOPASSWD: ALL

Full Example

sudo tee /etc/sudoers.d/myuser <<EOL_SUDOERS
myuser ALL=(ALL) NOPASSWD: ALL
EOL_SUDOERS

Once all nodes are prepared, you can start setting up the cluster.

Kubeopsctl-2.0: Setup Cluster (MicroOS)

Mon, 01 Jan 0001 00:00:00 +0000

Important The following commands have to be executed on all nodes

⚠ Warning This guide assumes that certain steps are performed as a privileged user. You need sudo privileges for that.

1. Install KOSI

# KOSI-version can be found under : https://packagerepo.kubeops.net/rpm/
curl -O https://packagerepo.kubeops.net/rpm/kosi-2.15.0.4_Beta0-0.x86_64.rpm
sudo transactional-update pkg install --no-confirm --allow-unsigned-rpm --force kosi-2.15.0.4_Beta0-0.x86_64.rpm

# need reboot after transactional update!
sudo reboot now 

# copy plugins and kosi config from last snapshot
sudo mkdir -p /var/kubeops/plugins
sudo cp -r /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/var/kubeops/plugins/ /var/kubeops/
sudo cp -r /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/var/kubeops/kosi/ /var/kubeops/

2. Set the KUBEOPSROOT env var

Set KUBEOPSROOT and XDG_RUNTIME_DIR in ~/.bashrc

⚠ Warning This file entry has to be adapted to your values before usage.

# file ~/.bashrc
# Append these values to the end of your ~/.bashrc file
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

Source .bashrc to apply the values

source ~/.bashrc
echo $KUBEOPSROOT
echo $XDG_RUNTIME_DIR

As a result you should see your KUBEOPSROOT-path two times.

3. Adjust KOSI Configuration

This creates a kubeops directory in your home directory and transfers all necessary files, e.g., the kosi-config and the plugins, to it.

mkdir ~/kubeops
cd ~/kubeops
cp -R /var/kubeops/kosi/ .
cp -R /var/kubeops/plugins/ .

Important The config.yaml is in your KUBEOPSROOT-path (typically in ~/kubeops/kosi)

Set hub in your kosi config to hub: https://dispatcher.kubeops.net/v4/dispatcher/

⚠ Warning This file entry has to be adapted to your values before usage.

Set the “plugins”-entry in your kosi config to plugins: /home/<yourUser>/kubeops/plugins, where is changed to your username

⚠ Warning This file has to be adapted to your values before usage.

# file $KUBEOPSROOT/kosi/config.yaml
apiversion: kubernative/sina/config/v2

spec:
  hub: https://dispatcher.kubeops.net/v4/dispatcher/ # <-- set hub url
  plugins: <your kubeopsroot>/plugins/ # <-- set the path to your plugin folder (~ for home or $KUBEOPSROOT don't work, it has to be the full path)
  workspace: /tmp/kosi/process/
  logging: info
  housekeeping: false
  proxy: false

4. Install KOSI enterprise plugins

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.2.0_Beta0

⚠ Warning This command has to be adapted to your values before usage.

kosi login -u <yourUser>

Important At this point it is normal if you get the following error message:
Error: The login to registry is temporary not available. Please try again later.
The reason for this is that podman is not yet installed.

6. Prepare nodes

Important The following KOSI package must be installed on all nodes (admin, control plane, and worker nodes).

kosi install --hub kubeops kubeops/microos-prepare-node:2.2.0_Beta0

# transactional update need reboot!
sudo reboot now

7. Create a cluster-values.yaml configuration file

⚠ Warning This file has to be adapted to your values before usage.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your controlplane01 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: MicroOS # Currently, only the builtin MicroOS repository is supported. Local support lacks!
changeCluster: true
zones:
# IMPORTANT: The following part has to be adapted so that every one of your controlplane nodes and workernodes is included
# This file only includes the minimum requirements for the amount of controlplanes and workers and an example usage of zones
# You should adapt this part to your amount of controlplanes and workers and cluster them into as many zones as you like
- name: zone1
  nodes:
  - name: <your controlplane01 hostname>
    iPAddress: <your controlplane01 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker01 hostname>
    iPAddress: <your worker01 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone2
  nodes:
  - name: <your controlplane02 hostname>
    iPAddress: <your controlplane02 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker02 hostname>
    iPAddress: <your worker02 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone3
  nodes:
  - name: <your controlplane03 hostname>
    iPAddress: <your controlplane03 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker03 hostname>
    iPAddress: <your worker03 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>

Full Example

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: myuser
kubernetesVersion: 1.34.1
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: MicroOS # Currently, only the builtin MicroOS repository is supported. Local support lacks!
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: node01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.34.1
  - name: node04
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.34.1
- name: zone2
  nodes:
  - name: node02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.34.1
  - name: node05
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.34.1
- name: zone3
  nodes:
  - name: node03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.34.1
  - name: node06
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.34.1

⚠ Warning value of ‘packageRepository’ in above yaml file should be changed to https://packagerepo.kubeops.net/, in case you have incorporated package repository.

7.1 Using KubeVip in your Cluster (optional)

If you want to use KubeVip to setup your Cluster, you need a virtual ip for that. Also you have to set kubeVipEnabled to true and set your virtualIP. If you dont want to use KubeVip you have to set kubeVipEnabled to false and set your first controlplane as your virtualIP in your cluster-values.yaml in the Setup. Refer to the official KubeVip-documentation for details here.

Examples:

kubeVipEnabled: true
virtualIP: <IP in your cluster ip range which is not given yet>

kubeVipEnabled: false
virtualIP: <controlplane01 ip>

Note The virtual IP must be in the same network as the machines and must not already be in use.

8. Install kubernetes tools

Important The following KOSI package must be installed on all nodes (admin, control plane, and worker nodes).

kosi install --hub kubeops kubeops/microos-kubernetes:1.34.1 -f cluster-values.yaml

# transactional update need reboot!
sudo reboot now

Note On your admin node you just need kubectl.
You can specify the following values during installation to install only kubectl.

# file kubectl-only-values.yaml
installKubeadm: false
installKubelet: false
installKubectl: true

kosi install --hub kubeops kubeops/microos-kubernetes:1.35.3 -f cluster-values.yaml -f kubectl-only-values.yaml

# transactional update need reboot!
sudo reboot now

9. Install kubeopsctl on admin node

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
curl -O https://packagerepo.kubeops.net/rpm/kubeopsctl-2.2.0.0_Beta0-0.x86_64.rpm
sudo transactional-update pkg install --no-confirm --allow-unsigned-rpm --force kubeopsctl-2.2.0.0_Beta0-0.x86_64.rpm

# need reboot after transactional update!
sudo reboot now 

sudo cp /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/usr/local/bin/kubeopsctl /usr/local/bin/

10. Pull required KOSI packages

⚠ Warning This command should be adapted to your value before usage.

If you do not specify a parameter, the current Kubernetes version 1.34.1 will be pulled.
With parameter --kubernetesVersion <your wanted Kubernetesversion> you can pull an older Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

kubeopsctl pull --kubernetesVersion <x.xx.x>

11. Cluster Setup

⚠ Warning This command has to be adapted to your values before usage.

Make sure that you are logged in on hub and registry.

kosi login -u <your username>

Important Now the login for hub and registry should be successful!

Make sure that you changed the kosi config.yaml.

cat $KUBEOPSROOT/kosi/config.yaml

Make sure that you pulled all required packages.

ls -l $KUBEOPSROOT/packages

Install Kubernetes Cluster with kubeopsctl. Cluster setup takes about 10 minutes.

kubeopsctl apply -f cluster-values.yaml

KubeOpsDoku – Getting-Started

Kubeopsctl-2.0: About KubeOps Compliance

What is KubeOps Compliance?

Why use KubeOps Compliance?

Highlights

Kubeopsctl-2.0: About KubeOps Compliance Licences

Kubeopsctl-2.0: Prerequisites

Prerequisites for admin

Prerequisites for each node

Kubeopsctl-2.0: Prepare Cluster

1. Include package repo on all nodes

2. Special operating system adaptations on all nodes

3. Distribute ssh-keys on all nodes

4. Distribute SUDOERS on all nodes

5. Configure time synchronization on all nodes

How to install and configure chrony (recommended)

6. Install curl

How to install and configure curl

Kubeopsctl-2.0: Setup Cluster

1. Install KOSI

2. Set the KUBEOPSROOT env var

3. Adjust KOSI Configuration

4. Install KOSI enterprise plugins

5. Login with your user

6. Install kubeopsctl

7. Create a cluster-values.yaml configuration file

7.1 Using KubeVip in your Cluster (optional)

8. Pull required KOSI packages

9. Install podman

10. Install helm

11. Install kubernetes tools (kubectl)

12. Cluster Setup

Kubeopsctl-2.0: Prepare Cluster (MicroOS)

1. Create a user to work with

2. Hostnames of all nodes must be resolvable via DNS

3. Distribute ssh-keys on all nodes

4. Distribute SUDOERS on all nodes

Kubeopsctl-2.0: Setup Cluster (MicroOS)

1. Install KOSI

2. Set the KUBEOPSROOT env var

3. Adjust KOSI Configuration

4. Install KOSI enterprise plugins

5. Login with your user

6. Prepare nodes

7. Create a cluster-values.yaml configuration file

7.1 Using KubeVip in your Cluster (optional)

8. Install kubernetes tools

9. Install kubeopsctl on admin node

10. Pull required KOSI packages

11. Cluster Setup