KubeOpsDoku – Getting-Started

Kubeopsctl: About KubeOps Compliance

Mon, 01 Jan 0001 00:00:00 +0000

What is KubeOps Compliance?

kubeopsctl serves as a versatile utility designed specifically to efficiently manage both the configuration and status of a cluster.

With its capabilities, users can articulate their desired cluster state in detail, outlining configurations and specifications.

Subsequently, kubeopsctl orchestrates the creation of a cluster that precisely matches the specified state, ensuring alignment between intentions and operational reality.

Why use KubeOps Compliance?

In kubeopsctl, configuration management involves defining, maintaining, and updating the desired state of a cluster, including configurations for nodes, pods, services, and other resources in the application environment.

The main goal of kubeopsctl is to match the clusters actual state with the desired state specified in the configuration files. With a declarative model, kubeopsctl enables administrators to express their desired system setup, focusing on „what“ they want rather than „how“ to achieve it.

This approach improves flexibility and automation in managing complex systems, making the management process smoother and allowing easy adjustment to changing needs.

kubeopsctl uses YAML files to store configuration data in a human-readable format. These files document important metadata about the objects managed by kubeopsctl, such as pods, services, and deployments.

Highlights

creating a cluster
adding nodes to your cluster
drain nodes
updating single nodes
label nodes for zones
adding platform software into your cluster

Kubeopsctl: KubeOps Installation and Setup

Mon, 01 Jan 0001 00:00:00 +0000

Welcome to the very first step to getting started with KubeOps Compliance. In this section, you will get to know about

hardware, software and network requirements
steps to install the required software
key configurations for KubeOps

Prerequisites

A total of 7 machines are required:

one admin
three masters
three workers

All of your machines need the Red Hat Enterprise Linux 8 os.
Below you can see the minimal requirements for CPU, memory and disk storage:

OS	Minimum Requirements
Red Hat Enterprise Linux 8	8 CPU cores, 16 GB memory, 50GB disk storage

For each working node, an additional unformatted hard disk with 50 GB each is required. For more information about the hard drives for rook-ceph, visit the rook-ceph prerequisites page

If a node fails, the minimum requirements may not be sufficient to guarantee trouble-free operation.

Requirements on admin

The following requirements must be fulfilled on the admin machine.

All the utilized users require sudo privileges. If you are using Kubeops as a user, you need a user with sudo rights, for RHEL 8 Environments it is the wheel group the user should be added to. Make sure that you change your user with:

su -l <user>

Admin machine must be synchronized with the current time.
You need an internet connection to use the default KubeOps Registry.
Important: Choose the correct KubeOps Registry based on your version:
"registry1.kubernative.net/lima"
"registry.kubeops.net/kubeops"
A local registry can be used in the Airgap environment. KubeOps only supports secure registries. It is important to list your registry as an insecure registry in registry.conf (/etc/containers/registries.conf for podman, /etc/docker/deamon.json for docker), in case of insecure registry usage.

Now you can create your own registry instead of using the default. Checkout how to Guide Create a new Repository. for more info.

Podman must be installed on your machine.

Podman depends on CNI plugins for container networking. If your containers fail to start with errors about missing plugins like bridge or tuning, check if the CNI plugin folder exists:

ls /opt/cni/bin

If the folder is missing or empty, install the CNI plugins with:

sudo dnf install -y containernetworking-plugins

Now, install podman

sudo dnf install -y podman

you need to install jq for rook-ceph:

dnf install -y jq

also you need to login to systemd with your os user:

loginctl enable-linger myuser

Warning

There can be an issue with conflicts with containerd, so it is recommended that containerd.io is removed before installing the podman package.

$KUBEOPSROOT and $LIMAROOT must be set.

echo "export KUBEOPSROOT=\"${HOME}/kubeops\"" >> $HOME/.bashrc
echo "export LIMAROOT=\"${HOME}/kubeops/lima\"" >> $HOME/.bashrc
source $HOME/.bashrc

Install KOSI and Login with Valid Credentials

Install KOSI from an RPM file
```
sudo dnf install -y kosi*.rpm
```
INFO: For additional details on downloading KOSI package found here and installing the KOSI package, refer to this link.

NOTE: Helm package installation is not required, as it will be installed automatically with the kubeopsctl platform.
Login to the kubeops hub using kosi

After entering the password, you will be logged in to the kubeops hub. Use the following command to initiate the login process:
```
  kosi login -u <user>
```

Requirements for each node

The following requirements must be fulfilled on each node.

All the utilized users require sudo privileges. If you are using Kubeops as a user, you need a user with sudo rights, for and RHEL 8 Environments it is the wheel group the user should be added to.
Every machine must be synchronized with the current time.
You have to assign lowercase unique hostnames for every machine you are using.

We recommended using self-explanatory hostnames.

To set the hostname on your machine use the following command:
```
sudo hostnamectl set-hostname <name of node>
```
- Example
  Use the commands below to set the hostnames on each machine as admin, master, node1 node2 (lowercase letters and numbers only).
```
# On admin node and make sure to set hostname for other nodes roo
sudo hostnamectl set-hostname admin
```
Requires sudo privileges
It is recommended to have a DNS service running to resolve node names in your cluster. If you don’t have a DNS service, you can update the /etc/hosts file on each relevant machine.

An example of entries in the /etc/hosts file might look like this:
```
<ip_admin_node> <admin_node_name>
<ip_master_1_node> <master_1_node_name>
...
<------------ place_holder_for_other_nodes-------------->
...
```
Replace the placeholders with your actual IP addresses and hostnames.
To establish an SSH connection between your machines, you either need an SSH key or you need to install sshpass.
1. Generate an SSH key on admin machine using following command
```
ssh-keygen
```
  There will be two keys generated in ~/.ssh directory.
  The first key is the id_rsa(private) and the second key is the id_rsa.pub(public).
2. Copy the ssh key from admin machine to your node machine/s with following command
```
ssh-copy-id <ip address or hostname of your node machine>
```
3. Now try establishing a connection to your node machine/s
```
ssh <ip address or hostname of your node machine>
```
it is recommended that runc is uninstalled
```
sudo dnf remove -y runc
```

tc should be installed.

sudo dnf install -y tc
sudo dnf install -y libnftnl

for opensearch, the /etc/sysctl.conf should be configured, the line
```
  vm.max_map_count=262144
```
should be added. also the command should be executed after that
```
  sudo sysctl -p
```

The user need to install versionlock with this command

sudo dnf install python3-dnf-plugin-versionlock.noarch

Optional: In order to use encrypted traffic inside the cluster, follow these steps:

For RHEL machines, you will need to import the ELRepo Secure Boot key into your system. You can find a detailed explanation and comprehensive instructions in our how-to-guide Importing the Secure-Boot key

This is only necessary if your system has Secure Boot enabled. If this isn´t the case, or you dont want to use any encryption at all, you can skip this step.

Installing KubeOpsCtl

Create a kubeopsctl.yaml file with respective information as shown in kubeopsctl.yaml parameters, in order to use the KubeOps package.
Install the kubeops*.rpm on your admin machine.
```
sudo dnf install -y kubeopsctl*.rpm
```

Working with KubeOpsCtl

Run following command to apply the kubeopsctl configuration:

kubeopsctl apply -f kubeopsctl.yaml

kubeopsctl.yaml parameters

the names of the nodes should be the same as the hostnames of the machines.

Choose the appropriate imagePullRegistry based on your kubeops version.

### General values for registry access ###
imagePullRegistry: "registry1.kubernative.net/lima" # mandatory, the registry from which the images for the cluster are pulled
localRegistry: false # mandatory, set to true if you use a local registry, if you want to set the value to true then you also have to set harbor to true

### General values for registry access ###
imagePullRegistry: "registry.kubeops.net/kubeops" # mandatory, the registry from which the images for the cluster are pulled
localRegistry: false # mandatory, set to true if you use a local registry, if you want to set the value to true then you also have to set harbor to true

The imagePullRegistry parameter is for the registry, from which the images for the platform softeware is pulled. The localRegistry is a parameter for using a insecure, local registry for pulling images.

### Values for setup configuration ###
apiVersion: kubeops/kubeopsctl/alpha/v5 # mandatory
clusterName: "example" # mandatory
clusterUser: "myuser" # mandatory
kubernetesVersion: "1.30.0" # mandatory, check lima documentation
#masterHost: optional if you have an hostname, default value in "masterIP"
masterIP: 10.2.10.11 # mandatory
firewall: "nftables" # mandatory, default "nftables"
pluginNetwork: "calico" # mandatory, default "calico"
containerRuntime: "containerd" # mandatory
clusterOS: "Red Hat Enterprise Linux" # mandatory, must be "Red Hat Enterprise Linux"
### Additional values for cluster configuration
useInsecureRegistry: false # optional, default is false
ignoreFirewallError: false # optional, default is false
serviceSubnet: 192.168.128.0/17 # optional, default "192.168.128.0/17"
podSubnet: 192.168.0.0/17 # optional, default "192.168.0.0/17"
debug: false # optional, default is false
logLevel: vvvvv # optional, default "vvvvv"
systemCpu: "1" # optional, default "1"
systemMemory: "2G" # optional, default "2G"
sudo: true # optional, default is true
tmpCopyDir: "/tmp" # optional, default is /tmp
createCluster: true # optional, default is true
updateRegistry: true # optional, default is true
auditLogPath: "/tmp"

the parameter clusterName is used to interact with and manage the cluster later on, p.e. if you want to change the runtime, you need the clusterName parameter.
the clusteruser is the linux user for using the cluster. the clusterOS is the linux distribution of the cluster.
masterIP is the ip-adress of the clustermaster or the first master, which is later used for interacting with the cluster.
useInsecureRegistry is for using a local and insecure registry for pulling images for the lima software.
ignoreFirewallError is a parameter for ignoring firewall errors while the cluster is created (not while operating on the cluster).
serviceSubnet is the subnet for all kubernetes service IP-adresses.
podSubnet is the subnet for all kubernetes pod IP-adresses.
systemCpu is the maximum of cpu that the kube-apiserver is allowd to use.
sudo is a parameter for using sudo for commands that need sudo rights, if you use a non-root linux-user.
tmpCopyDir is a parameter for templating the folder on the cluster nodes, where images of lima will be copied to.
createCluster is a parameter with which you can specify whether you want to create a cluster or not.
updateRegistry is a parameter for updating the docker registry.

zones:
  - name: zone1 
    nodes:
      master:
        - name: cluster1master1
          ipAdress: 10.2.10.11
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.30.0
        - name: cluster1master2
          ipAdress: 10.2.10.12
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.30.0
      worker:
        - name: cluster1worker1
          ipAdress: 10.2.10.14
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.30.0
        - name: cluster1worker2
          ipAdress: 10.2.10.15
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.30.0
  - name: zone2
    nodes:
      master:
        - name: cluster1master3
          ipAdress: 10.2.10.13
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.30.0  
      worker:
        - name: cluster1worker3
          ipAdress: 10.2.10.16
          user: myuser
          systemCpu: 200m
          systemMemory: 200Mi 
          status: active
          kubeversion: 1.30.0

This YAML content is mandatory and describes a configuration for managing multiple zones in a Kubernetes cluster. Let’s break it down step by step:

zones: This is the top-level key in the YAML file, representing a list of zones within the Kubernetes cluster.
- zone1 and zone2: These are two zones within the cluster, each with its own configuration.
  - nodes: This is a sub-key under each zone, indicating the different types of nodes within that zone.
    - master: This is a sub-key under nodes, representing the master nodes in the zone.
      - cluster1master1, cluster1master2, and cluster1master3: These are individual master nodes in the cluster, each with its own configuration settings. They have attributes like name (node name has to be equal to host name), ipAdress (IP address), user (the user associated with the node), systemCpu (CPU resources allocated to the system), systemMemory (system memory allocated), status (the status of the node, can be either “active” or “drained”), and kubeversion (the Kubernetes version running on the node). Those kubernetes versions in the zone are for the nodes. NOTE: If you drain too many nodes, you may have too few OSDs for Rook.
    - worker: This is another sub-key under nodes, representing the worker nodes in the zone.
      - cluster1worker1, cluster1worker2, and cluster1worker3: Similar to the master nodes, these are individual worker nodes in the cluster, each with its own configuration settings, including name, IP address, user, system resources, status, and Kubernetes version.

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true # if localRegistry is set to true, harbor also needs to be set to true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
kubeops-dashboard: true
certman: true
ingress: true 
keycloak: true
velero: true

this values are booleans for deciding which applications are later installed into the cluster.

# Global values, will be overwritten by the corresponding values of the individual packages
namespace: "kubeops"
storageClass: "rook-cephfs"

These global values will be used for installing the packages, but will be overwritten by the corresponding package-level settings.

namespace defines the kubernetes-namespace in which the applications are deployed
storageClass defines the name of StorageClass-Ressource that will be used by the applications.

rookValues:
  namespace: kubeops
  cluster:
    spec:
      dataDirHostPath: "/var/lib/rook" # optional, default is /var/lib/rook
    storage:
      useAllNodes: true # optional, default value: true
      useAllDevices: true # optional, default value: true
      deviceFilter: "^sd[a-b]" # optional, will only be used if useAllDevices is set to false
      config:
        metadataDevice: "sda" # optional, only set this value, if there is a device available
      nodes: # optional if useAllNodes is set to true, otherwise mandatory
        - name: "<ip-adress of node_1>"
          devices:
            - name: "sdb" 
        - name: "<ip-adress of node_2>"
          deviceFilter: "^sd[a-b]"
          config:
            metadataDevice: "sda" # optional
    resources:
      mgr:
        requests:
          cpu: "500m" # optional, default is 500m, limit: 1000m
          memory: "512Mi" # optional, default is 1Gi, limit: 1Gi
      mon:
        requests:
          cpu: "1" # optional, default is 1, limit: 2000m
          memory: "1Gi" # optional, default is 1Gi, limit: 2Gi
      osd:
        requests:
          cpu: "1" # optional, default is 1, limit: 2
          memory: "1Gi" # optional, default is 4Gi, limit: 4Gi
    dashboard:
      enabled: true
  operator:
    data:
      rookLogLevel: "DEBUG" # optional, default is DEBUG

the namespace parameter is important for the apllications, because this parameter decides, in which namespace the individual applications are deployed.
dataDirHostPath is for setting the path of the configuration fiules of rook.
useAllNodes is parameter of rook-ceph and if it is set to true, all worker nodes will be used for rook-ceph.
useAllDevices is parameter of rook-ceph and if it is set to true, all possible devices will be used for rook-ceph.
deviceFilter is a Global filter to only select certain devicesnames. This example matches names starting with sda or sdb. it will only be used if useAllDevices is set to false and will be ignored if individual devices have been specified on a node.
metadataDevice: Name of a device or lvm to use for the metadata of OSDs(daemons for storing data on the local file system) on each node Performance can be improved by using a low latency device (SSD or NVMe) as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. This global setting will be overwritten by the corresponding node-level setting.
nodes: Names of individual nodes in the cluster that should have their storage included. Will only be used if useAllNodes is set to false. Specific configurations of the individual nodes will overwrite global settings.
resources refers to the cpu and memory that the parts of rook-ceph will be requesting. In this case it is the manager, the monitoring pods and the OSDs (they have the job of managing the local storages of the nodes and together they form the distributed storage) as well as the filesystem and object-store pods (they manage the respecting storage solution).
rookLogLevel: the loglevel of rook-ceph. this provides the most informative logs.

harborValues: 
  namespace: kubeops # optional, default is kubeops
  harborpass: "password" # mandatory: set password for harbor access
  databasePassword: "Postgres_Password" # mandatory: set password for database access
  redisPassword: "Redis_Password" # mandatory: set password for redis access
  externalURL: http://10.2.10.11:30002 # mandatory, the ip address and port, from which harbor is accessable outside of the cluster
  nodePort: 30002 # mandatory
  hostname: harbor.local # mandatory
  harborPersistence:
    persistentVolumeClaim:
      registry:
        size: 40Gi # optional, default is 40Gi
        storageClass: "rook-cephfs" #optional, default is rook-cephfs
      jobservice:
        jobLog:
          size: 1Gi # optional, default is 1Gi
          storageClass: "rook-cephfs" #optional, default is rook-cephfs
      database:
        size: 1Gi # optional, default is 1Gi
        storageClass: "rook-cephfs" #optional, default is rook-cephfs
      redis:
        size: 1Gi # optional, default is 1Gi
        storageClass: "rook-cephfs" #optional, default is rook-cephfs
      trivy: 
        size: 5Gi # optional, default is 5Gi
        storageClass: "rook-cephfs" #optional, default is rook-cephfs

You can set the root password for the postgres-database, redis and harbor. For the persistant volumes of harbor, the sizes and the storageclass are also templatable. So all applications of harbor, i.e. trivy for the image-scanning or the chart museum for helm charts.

###Values for filebeat deployment###
filebeatValues:
  namespace: kubeops #optional, default is kubeops

namespace value specifies the Kubernetes namespace where filebeat will be deployed.

###Values for Logstash deployment###
##For detailed explaination for each key see: https://github.com/elastic/helm-charts/releases/tag/v7.16.3###
logstashValues:
  namespace: kubeops
  volumeClaimTemplate:
    accessModes: 
      - ReadWriteMany #optional, default is [ReadWriteMany]
    resources:
      requests:
        storage: 1Gi # mandatory, depending on storage capacity
    storageClass: "rook-cephfs" #optional, default is rook-cephfs

for logstash the pvc size is also templateable

###Values for OpenSearch-Dashboards deployment###
##For detailed explaination for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch-dashboards###
openSearchDashboardValues:
  namespace: kubeops
  nodePort: 30050
###Values for OpenSearch deployment###
##For detailed explaination for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch###
openSearchValues:
  namespace: kubeops
  opensearchJavaOpts: "-Xmx512M -Xms512M" # optional, default is -Xmx512M -Xms512M
  resources:
    requests:
      cpu: "250m" # optional, default is 250m
      memory: "1024Mi" # optional, default is 1024Mi
    limits:
      cpu: "300m" # optional, default is 300m
      memory: "3072Mi" # optional, default is 3072Mi
  persistence:
    size: 4Gi # mandatory
    enabled: "true" # optional, default is true
    enableInitChown: "false" # optional, default is false
    labels:
      enabled: "false" # optional, default is false
    storageClass: "rook-cephfs" # optional, default is rook-cephfs
    accessModes:
      - "ReadWriteMany" # optional, default is {ReadWriteMany}
  securityConfig:
    enabled: false # optional, default value: false
    ### Additional values can be set, if securityConfig is enabled:
    # path: "/usr/share/opensearch/plugins/opensearch-security/securityconfig"
    # actionGroupsSecret:
    # configSecret:
    # internalUsersSecret: internal-users-config-secret
    # rolesSecret:
    # rolesMappingSecret:
    # tenantsSecret:
    # config:
    #   securityConfigSecret: ""
    #   dataComplete: true
    #   data: {}
  replicas: "3" # optional, default is 3
#--------------------------------------------------------------------------------------------------------------------------------

opensearchJavaOpts is the size of the java heap.
enableInitChown is the procedure of changing the owner of the opensearch configuration files, so non-root users can change the configuration files.
if you want labels for the opensearch pods in the cluster, you can enable the labels with the enabled parameter under the labels subtree.
if you want to use a custom security config, you can enable it and use then paramters like the path to the file. if you want more info, you can find it here
the replicas are 3 by default, but you can template it for better scaling.


###Values for Prometheus deployment###
prometheusValues:
  namespace: kubeops # optional, default is kubeops
  grafanaUsername: "user" # optional, default is user
  grafanaPassword: "password" # optional, default is password
  retentionSize: "24GB" # optional, default is 24GB
  grafanaResources:
    nodePort: 30211 # optional, default is 30211
    storageClass: "rook-cephfs" # optional, default is rook-cephfs
    storage: 25Gi # optional, default is 5Gi
    grafanaUsername: "admin" # optional, default is admin
    grafanaPassword: "admin" # optional, default is admin
    retention: 10d # mandatory
    retentionSize: "24GB" # mandatory

  prometheusResources:
    retention: 10d # optional, default is 10d
    retentionSize: "24GB" # optional, default is "24GB"
    storageClass: "rook-cephfs" # optional, default is rook-cephfs
    storage: 25Gi # optional, default is 25Gi

the nodePort is 30211, so you can visit the grafana apllication on every master with <ip-adress of master>:30211, but you can template and thus change it.

###Values for OPA deployment###
opaValues:
  namespace: kubeops

namespace value specifies the Kubernetes namespace where OPA will be deployed.

###Values for KubeOps-Dashboard (Headlamp) deployment###
kubeOpsDashboardValues:
  namespace: kubeops
  hostname: kubeops-dashboard.local
  service:
    nodePort: 30007

namespace value specifies the Kubernetes namespace where KubeOps-Dashboard will be deployed.
hostname is for accessing the KubeOps-Dashboard service.
the nodePort value specifies the node port for accessing KubeOps-Dashboard.

###Values for cert-manager deployment###
certmanValues:
  namespace: kubeops
  replicaCount: 3
  logLevel: 2

namespace value specifies the Kubernetes namespace where cert-manager will be deployed.
replicaCount specifies the number of replicas for the cert-manager deployment.
logLevel specifies the logging level for cert-manager.

###Values for ingress-nginx deployment###
ingressValues:
  namespace: kubeops
  externalIPs: []

namespace value specifies the Kubernetes namespace where ingress-nginx will be deployed.
externalIPs value specifies a list of external IP addresses that will be used to expose the ingress-nginx service. This allows external traffic to reach the ingress controller. The value for this key is expected to be provided as a list of IP addresses.

###Values for keycloak deployment###
keycloakValues:
  namespace: "kubeops" # Optional, default is "keycloak"
  storageClass: "rook-cephfs" # Optional, default is "rook-cephfs"
  nodePort: "30180" # Optional, default is "30180"
  hostname: keycloak.local
  keycloak:
    auth:
      adminUser: admin # Optional, default is admin
      adminPassword: admin # Optional, default is admin
  postgresql:
    auth:
      postgresUserPassword: "" # Optional, default is ""
      username: bn_keycloak # Optional, default is "bn_keycloak"
      password: "" # Optional, default is ""
      database: bitnami_keycloak # Optional, default is "bitnami_keycloak"
    volumeSize: "8Gi"

namespace value specifies the Kubernetes namespace where keycloak will be deployed.
storageClass value specifies the storage class to be used for persistent storage in Kubernetes. If not provided, it defaults to “rook-cephfs”.
nodePort value specifies the node port for accessing Keycloak. If not provided, it defaults to “30180”.
hostname value specifies the hostname for accessing the Keycloak service.
adminUser value specifies the username for the Keycloak admin user. Defaults to “admin”.
adminPassword value specifies the password for the Keycloak admin user. Defaults to “admin”.
postgresUserPassword value specifie the password for the PostgreSQL database. Defaults to an empty string.
username value specifies the username for the PostgreSQL database. Defaults to “bn_keycloak”.
password value specifies the password for the PostgreSQL database. Defaults to an empty string.
database value specifies the name of the PostgreSQL database. Defaults to “bitnami_keycloak”.

veleroValues:
  namespace: "velero"
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"
  useNodeAgent: false
  defaultVolumesToFsBackup: false
  provider: "aws"
  bucket: "velero"
  useVolumeSnapshots: false
  backupLocationConfig:
    region: "minio"
    s3ForcePathStyle: true
    s3Url: "http://minio.velero.svc:9000"

namespace: Specifies the Kubernetes namespace where Velero will be deployed.
accessKeyId: Your access key ID for accessing the S3 storage service.
secretAccessKey: Your secret access key for accessing the S3 storage service.
useNodeAgent: Indicates whether to use a node agent for backup operations. If set to true, Velero will use a node agent.
defaultVolumesToFsBackup: Specifies whether to default volumes to file system backup. If set to true, Velero will use file system backup by default.
provider: Specifies the cloud provider where the storage service resides.
bucket: The name of the S3 bucket where Velero will store backups.
useVolumeSnapshots: Indicates whether to use volume snapshots for backups. If set to true, Velero will use volume snapshots.
backupLocationConfig: Configuration for the backup location.
region: Specifies the region where the S3 storage service is located.
s3ForcePathStyle: Specifies whether to force the use of path-style URLs for S3 requests.
s3Url: The URL for accessing the S3-compatible storage service.

Kubeopsctl: Set up a Basic Cluster

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

kubeopctl plattform requirements
best practices for machine setup
setup secure headless environment for communication between admin and all masters / workers
how to install required software
how to use the official KubeOps Website to download kubeopsctl
how to create a basic cluster

After the installation, kubeopctl is available as command line interface.

Notice The intent of this guide is to setup an example cluster with 7 machines. We do not recommend this guide for production environments. For more information about kubeopsctl production environments, see LINK.

Prerequisites

To get the most out of this guide, the following requirements should be met:

basic understanding of Linux environments, bash / shell
basic understanding of text editors, vi / nano
administrator privileges (root) are granted

A total of 7 machines (virtual or physical) are required and need to be set up:

one admin - control plane, this machine will manage all tasks on the cluster and integrated machines
three masters
three workers

The final cluster will have the following structure. Masters and workers are added to two clusters zones.

Important The platform setup using this guide can take up to 3 hours. Depending on the resources available.

Step 1 - Minimal Platform Requirements

kubeopsctl is designed to work with the latest versions of the following operating systems.

Notice Make sure all machines use the same operating system.

Supported Operating Systems


Operating system	Red Hat Enterprise Linux (Version 8.2 or newer)

System requirements Admin Nodes


CPU	2x
Memory	2 GB
Diskspace	50 GB
Internet Access	Yes to use default `KubeOps Registry`

Important: Choose the correct KubeOps Registry based on your version:
"registry1.kubernative.net/lima"
"registry.kubeops.net"

System requirements Master Nodes


CPU	4x
Memory	8 GB
Diskspace	50 GB

System requirements Worker Nodes


CPU	8x
Memory	16 GB
Diskspace	50 GB 50 GB unformatted non-partitioned disk storage for `ceph`

Important

By default Red Hat Enterprise Linux restricts package management such as installing or updating. Before installing any software, you may need to register and subscribe with the Red Hat Customer Portal.

To register and subscribe, run the command subscription-manager register.

For more information about rook-ceph, see the prerequisites in its official documentation.

If a node fails, the minimum requirements may not be sufficient to guarantee trouble-free operation.

Step 2 - Set up your Machines

You can setup the admin, master and worker nodes as virtual or as physical machines.

During the setup of the machines, make sure that you meet the following requirements:

heed the platform requirements as mentioned above
all machines need to be synchronized with the current time
all machines need to be within the same network environment

Important

During the setup, note the IP addresses of all machines.

Assign unique hostnames for each machine (lowercase letters and numbers only).

We recommended using self-explanatory hostnames (see example table below).

To get the most out of this guide, use the following hostnames for your basic cluster:

Hostnames

Machine / Purpose	Hostnames
Admin	admin
Masters	master1 master2 master3
Workers	worker1 worker2 worker3

Assigning Hostnames manually

If you need to assign hostnames manually, login to the machine and use the hostnamectl set-hostname command.

hostnamectl set-hostname master1

Repeat this process for all machines where necessary.

Remove firewalld on Red Hat Enterprise Linux 8

Important Repeat this process for all master and worker machines where necessary.

If you are using Red Hat Enterprise Linux 8, you must remove firewalld. Kubeopsctl installs nftables by default.
You can use the following commands to remove firewalld:

systemctl disable --now firewalld
systemctl mask firewalld
dnf remove -y firewalld
reboot

Step 3 - Set up Access for the Admin Machine

The admin machine needs secure and headless access to all other machines.

Set up IP Addresses for DNS

It is recommended, that a DNS service is running. If you do not have a DNS service, you need to edit the /etc/hosts file on the admin machine.

Add the following lines at the end of the /etc/hosts file. Replace the IP addresses with the actual addresses you noted during the setup of all machines. Replace the hostnames with the actual hostnames you assigned during the setup of all machines:

10.2.10.10 admin
10.2.10.11 master1
10.2.10.12 master2
10.2.10.13 master3
10.2.10.14 worker1
10.2.10.15 worker2
10.2.10.16 worker3

Set up Secure Access

To securely access the master and worker machines, you need to create a ssh-key-pair (private and public key) on the admin machine. Afterwards copy the public key onto each machine.

To learn more about ssh and key-pairs see our guide on How to set up SSH keys.

Step 4 - Install Podman on the Admin Machine

To ensure compatibility across different containerized environments, kubeopsctl requires the installation of Podman (latest version).

Notice There can be an issue with conflicts with containerd, so it is recommended that containerd.io is removed before installing the podman package.

Install Podman on the admin machine using the inbuilt package manager.

sudo dnf install -y podman

you need to install jq for rook-ceph:

dnf install -y jq

also you need to login to systemd with your os user:

loginctl enable-linger myuser

Notice

By default, kubeopsctl only supports secure registries.

To setup kubeopsctl with insecure registries, see here.

Step 5 - Install kubeopsctl on the Admin Machine

With everything prepared, the next step is to download and install kubeopsctl on the admin machine.

Downloading KOSI

Download your desired version of the kubeopsctl package file (.rpm) from the official download page onto the admin machine.

Installing KOSI
```
sudo dnf install -y <path>/<kosi_rpm>
```
Login to the kubeops hub using kosi

After you input the password, you will gain access to the kubeops hub. Use the following command to begin the login process:
```
  kosi login -u <user>
```

Installing kubeopsctl

Install kubeopsctl using the inbuilt package manager. Replace <path> and <kubeopsctl_rpm> with the respective path and file name of the kubeopsctl package file.

Notice

On installation, kubeopsctl will perform all necessary steps to setup the environment in dependence of your used operating system. This includes the automatic installation / deinstallation of additional software or packages.

Removes software:

runc

Installs Software:

tc, libnftnl
iproute2, libnftnl

To install kubeopsctl use the following command.

sudo dnf install -y <path>/<kubeopsctl_rpm>

Create Work Folders and Setup Environment Variables

After the setup, you need to create work folders where kubeopsctl can save and manage configurations and other settings.

mkdir -p ~/kubeops
mkdir -p ~/kubeops/lima

To work comfortably, you need to assign these folders to the predefined environment variables KUBEOPSROOT and LIMAROOT.

echo 'export KUBEOPSROOT="${HOME}/kubeops"' >> $HOME/.bashrc
echo 'export LIMAROOT="${HOME}/kubeops/lima"' >> $HOME/.bashrc
source $HOME/.bashrc

Verify your Installation

To verify the installation of kubeopsctl on your system, use the command kubeopsctl version.

kubeopsctl version

Step 6 - Configure the Basic Cluster

With everything ready to start, the next step is to configure the cluster.

For configurations, kubeopsctl uses the YAML format.

Use an editor to create and edit the configuration file:

nano ~/basicCluster.yml

Copy and paste all lines into the file. You need to edit specific parameters according to the assigned IP addresses, hostnames etc.:

master/name - set all master hostnames
master/ipAdress - set all master IP addresses
worker/name - set all worker hostnames
worker/ipAdress - set all worker IP addresses

apiVersion: kubeops/kubeopsctl/alpha/v5 # mandatory
imagePullRegistry: "registry1.kubernative.net/lima"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11
systemCpu: "200m"
systemMemory: "200Mi"

zones:
  - name: zone1
    nodes:
      master:
        - name: master1
          ipAdress: 10.2.10.11
          status: active
          kubeversion: 1.30.0
        - name: master2
          ipAdress: 10.2.10.12
          status: active
          kubeversion: 1.30.0
      worker:
        - name: worker1
          ipAdress: 10.2.10.14
          status: active
          kubeversion: 1.30.0
        - name: worker2
          ipAdress: 10.2.10.15
          status: active
          kubeversion: 1.30.0
  - name: zone2
    nodes:
      master:
        - name: master3
          ipAdress: 10.2.10.13
          status: active
          kubeversion: 1.30.0  
      worker:
        - name: worker3
          ipAdress: 10.2.10.16
          status: active
          kubeversion: 1.30.0


# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
kubeops-dashboard: true
certman: true
ingress: true 
keycloak: true
velero: true

harborValues: 
  harborpass: "password" # change to your desired password
  databasePassword: "Postgres_Password" # change to your desired password
  redisPassword: "Redis_Password" 
  externalURL: http://10.2.10.11:30002 # change to ip adress of master1

prometheusValues:
  grafanaUsername: "user"
  grafanaPassword: "password"

ingressValues:
  externalIPs: []

keycloakValues:
  keycloak:
    auth:
      adminUser: admin
      adminPassword: admin
  postgresql:
    auth:
      postgresPassword: ""
      username: bn_keycloak
      password: ""
      database: bitnami_keycloak
      existingSecret: ""

veleroValues:
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"

apiVersion: kubeops/kubeopsctl/alpha/v5 # mandatory
imagePullRegistry: "registry.kubeops.net"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11
systemCpu: "200m"
systemMemory: "200Mi"

zones:
  - name: zone1
    nodes:
      master:
        - name: master1
          ipAdress: 10.2.10.11
          status: active
          kubeversion: 1.30.0
        - name: master2
          ipAdress: 10.2.10.12
          status: active
          kubeversion: 1.30.0
      worker:
        - name: worker1
          ipAdress: 10.2.10.14
          status: active
          kubeversion: 1.30.0
        - name: worker2
          ipAdress: 10.2.10.15
          status: active
          kubeversion: 1.30.0
  - name: zone2
    nodes:
      master:
        - name: master3
          ipAdress: 10.2.10.13
          status: active
          kubeversion: 1.30.0  
      worker:
        - name: worker3
          ipAdress: 10.2.10.16
          status: active
          kubeversion: 1.30.0


# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
kubeops-dashboard: true
certman: true
ingress: true 
keycloak: true
velero: true

harborValues: 
  harborpass: "password" # change to your desired password
  databasePassword: "Postgres_Password" # change to your desired password
  redisPassword: "Redis_Password" 
  externalURL: http://10.2.10.11:30002 # change to ip adress of master1

prometheusValues:
  grafanaUsername: "user"
  grafanaPassword: "password"

ingressValues:
  externalIPs: []

keycloakValues:
  keycloak:
    auth:
      adminUser: admin
      adminPassword: admin
  postgresql:
    auth:
      postgresUserPassword: ""
      username: bn_keycloak
      password: ""
      database: bitnami_keycloak
    volumeSize: 8Gi

veleroValues:
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"

Step 7 - Start the Basic Cluster

After the configuration is setup correctly, you can start your basic cluster for the first time:

kubeopsctl apply -f ~/basicCluster.yml

Notice This automatic process may take a while. Do not turn off any machine.

Kubeopsctl: Commands, Cluster Health and Modifications

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

basic kubeopsctl command line operations
apply changes to cluster, e.g. add new worker

Prerequisites

To get the most out of this guide, the following requirements should be met:

kubeopsctl is installed, see kubeopsctl Installation Guide
a cluster is installed and running
basic understanding of Linux environments, bash / shell
basic understanding of text editors, vi / nano
administrator privileges (root) are granted

Overview of the kubeopsctl CLI

kubeopsctl provides a set of command line operations. For more information, see here.

The main kubeopsctl commands are:

Command	Description
`--version`	Shows the current version of kubeopsctl.
`--help`	Shows an overview of all available commands:
`apply`	Sets up the kubeops platform with a configuration file.
`change registry`	Changes the currently used registry to a different one with a given configuration file.
`drain`	Drains a cluster, zone or node.
`uncordon`	Uncordons a cluster, zone or node. This ensures that while existing pods will remain running on the node, no new pods can be scheduled.
`upgrade`	Upgrades the Kubernetes version of a cluster, zone or node.
`status`	Prints the state of a cluster.

Print kubeopsctl Version

kubeopsctl --version

Show kubeopsctl Help

kubeopsctl --help

Setup Kubernetes Cluster or Apply Changes

Sets up the kubeops platform with a configuration file. Use the flag -f and pass the configuration file, e.g. kubeopsctl.yaml.

kubeopsctl apply -f kubeopsctl.yaml

You can also set the log level to a specific value. Available log levels are:

kubeopsctl apply -f kubeopsctl.yaml -l Debug3

The default log level is Info.

Error
Warning
Info (default log level)
Debug1
Debug2
Debug3

The command kubeopsctl apply can also be used to modify a cluster. For more information see Apply Changes to Cluster within this document.

Change the Registry

Changes the currently used registry to a different one with a given configuration file. For example:

kubeopsctl change registry -f kubeopsctl.yaml -r 10.2.10.11/library -t localhost/library

The -f parameter is used to use yaml parameter file.
The parameter -r is used to pull the docker images which are included in the package to a given local docker registry.
The -t parameter is used to tag the images with localhost. For the szenario that the registry of the cluster is exposed to the admin via a network internal domain name, but this name cant be resolved by the nodes, the flag -t can be used, to use the cluster internal hostname of the registry.

Draining

For draining clusters, zones or nodes use kubeopsctl drain <type>/<name>.

To drain a cluster use and replace <clustername> with the desired cluster:

kubeopsctl drain cluster/<clustername>

To drain a zone use and replace <zonename> with the desired zone:

kubeopsctl drain zone/<zonename>

To drain a node use and replace <nodename> with the desired node:

kubeopsctl drain node/<nodename>

Uncordoning

Uncordoning is similar to draining but draining means remove all running tasks/pods so the node can be fixed or taken offline safely. Uncordoning is just opening it up again for new tasks/pods.

For uncordoning clusters, zones or nodes use kubeopsctl uncordon TYPE/NAME.

To uncordon a cluster use and replace <clustername> with the desired cluster:

kubeopsctl uncordon cluster/<clustername>

To uncordon a zone use and replace <zonename> with the desired zone:

kubeopsctl uncordon zone/<zonename>

To uncordon a node use and replace <nodename> with the desired node:

kubeopsctl uncordon node/<nodename>

Upgrading

Upgrade clusters, zones or nodes by using kubeopsctl upgrade -v <version>.

To upgrade a cluster use and replace <clustername> with the desired cluster:

kubeopsctl upgrade cluster/<clustername> -v 1.26.6

To upgrade a zone use and replace <zonename> with the desired zone:

kubeopsctl upgrade zone/<zonename> -v 1.26.6

To upgrade a node use and replace <nodename> with the desired node:

kubeopsctl upgrade node/<nodename> -v 1.26.6

Check on Health and State of Clusters

To check on health and state of a cluster use the command kubeopsctl status cluster/<clustername>.

For example:

kubeopsctl status cluster/basiccluster

Apply Changes to Cluster

If the cluster is already running, it may be necessary to make additional settings or carry out updates.

For example:

when IP addresses or host names of nodes change
if you want to add or remove nodes

In any case, you need to modify the base configuration and apply the necessary changes to the configuration file. Changes will be applied state wise - short hand: only the difference will be applied.

For more information, see How to Guides.

Example: Add new Worker

If you want to add a new worker to your cluster, edit the configuration file.

In this example we reuse the basic configuration basiccluster.yml from the previous chapter (see Set up a Basic Cluster). Use an editor to edit the configuration file:

nano ~/basiccluster.yml

Add the lines to the configuration file for an additional worker4 at zone2. Also edit the desired ipAdress for the new worker.

apiVersion: kubeops/kubeopsctl/alpha/v3 # mandatory
imagePullRegistry: "registry1.kubernative.net/lima"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11

zones:
	- name: zone1
		nodes:
			master:
				- name: master1
					ipAdress: 10.2.10.11
					status: active
					kubeversion: 1.30.0
				- name: master2
					ipAdress: 10.2.10.12
					status: active
					kubeversion: 1.30.0
			worker:
				- name: worker1
					ipAdress: 10.2.10.14
					status: active
					kubeversion: 1.30.0
				- name: worker2
					ipAdress: 10.2.10.15
					status: active
					kubeversion: 1.26.2
	- name: zone2
		nodes:
			master:
				- name: master3
					ipAdress: 10.2.10.13
					status: active
					kubeversion: 1.30.0	
			worker:
				- name: worker3
					ipAdress: 10.2.10.16
					status: active
					kubeversion: 1.30.0

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
headlamp: true
certman: true
ingress: true
keycloak: true
velero: true

rookValues:
	cluster:
		resources:
			mgr:
				requests:
					cpu: 500m
					memory: 1Gi
			mon:
				requests:
					cpu: 500m
					memory: 1Gi
			osd:
				requests:
					cpu: 500m
					memory: 1Gi

harborValues:
	harborpass: "password" # change to your desired password
	databasePassword: "Postgres_Password" # change to your desired password
	redisPassword: "Redis_Password"
	externalURL: http://10.2.10.11:30002 # change to ip adress of master1
	nodePort: 30002
	harborPersistence:
		persistentVolumeClaim:
			registry:
				size: 40Gi
			jobservice:
				jobLog:
					size: 1Gi
			database:
				size: 1Gi
			redis:
				size: 1Gi
			trivy:
				size: 5Gi

prometheusValues:
	grafanaUsername: "user"
	grafanaPassword: "password"

logstashValues:
	volumeClaimTemplate:
		resources:
			requests:
				storage: 1Gi

openSearchValues:
	persistence:
		size: 4Gi

keycloakValues:
	keycloak:
		auth:
			adminUser: admin
			adminPassword: admin
	postgresql:
		auth:
			postgresPassword: ""
			username: bn_keycloak
			password: ""
			database: bitnami_keycloak
			existingSecret: ""

veleroValues:
	namespace: "velero"
	accessKeyId: "your_s3_storage_username"
	secretAccessKey: "your_s3_storage_password"
	useNodeAgent: false
	defaultVolumesToFsBackup: false
	provider: "aws"
	bucket: "velero"
	useVolumeSnapshots: false
	backupLocationConfig:
		region: "minio"
		s3ForcePathStyle: true
		s3Url: "http://minio.velero.svc:9000"

apiVersion: kubeops/kubeopsctl/alpha/v3 # mandatory
imagePullRegistry: "registry.kubeops.net"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11

zones:
	- name: zone1
		nodes:
			master:
				- name: master1
					ipAdress: 10.2.10.11
					status: active
					kubeversion: 1.30.0
				- name: master2
					ipAdress: 10.2.10.12
					status: active
					kubeversion: 1.30.0
			worker:
				- name: worker1
					ipAdress: 10.2.10.14
					status: active
					kubeversion: 1.30.0
				- name: worker2
					ipAdress: 10.2.10.15
					status: active
					kubeversion: 1.26.2
	- name: zone2
		nodes:
			master:
				- name: master3
					ipAdress: 10.2.10.13
					status: active
					kubeversion: 1.30.0	
			worker:
				- name: worker3
					ipAdress: 10.2.10.16
					status: active
					kubeversion: 1.30.0

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
headlamp: true
certman: true
ingress: true
keycloak: true
velero: true

rookValues:
	cluster:
		resources:
			mgr:
				requests:
					cpu: 500m
					memory: 1Gi
			mon:
				requests:
					cpu: 500m
					memory: 1Gi
			osd:
				requests:
					cpu: 500m
					memory: 1Gi

harborValues:
	harborpass: "password" # change to your desired password
	databasePassword: "Postgres_Password" # change to your desired password
	redisPassword: "Redis_Password"
	externalURL: http://10.2.10.11:30002 # change to ip adress of master1
	nodePort: 30002
	harborPersistence:
		persistentVolumeClaim:
			registry:
				size: 40Gi
			jobservice:
				jobLog:
					size: 1Gi
			database:
				size: 1Gi
			redis:
				size: 1Gi
			trivy:
				size: 5Gi

prometheusValues:
	grafanaUsername: "user"
	grafanaPassword: "password"

logstashValues:
	volumeClaimTemplate:
		resources:
			requests:
				storage: 1Gi

openSearchValues:
	persistence:
		size: 4Gi

keycloakValues:
	keycloak:
		auth:
			adminUser: admin
			adminPassword: admin
	postgresql:
		auth:
			postgresUserPassword: ""
			username: bn_keycloak
			password: ""
			database: bitnami_keycloak
		volumeSize: "8Gi"

veleroValues:
	namespace: "velero"
	accessKeyId: "your_s3_storage_username"
	secretAccessKey: "your_s3_storage_password"
	useNodeAgent: false
	defaultVolumesToFsBackup: false
	provider: "aws"
	bucket: "velero"
	useVolumeSnapshots: false
	backupLocationConfig:
		region: "minio"
		s3ForcePathStyle: true
		s3Url: "http://minio.velero.svc:9000"

After the modification is set up corretly, you can apply it to your cluster:

kubeopsctl apply -f ~/basiccluster.yml

Best Practices for Changes and Modifications

When configuring the first setup for a cluster, keep your base configuration file, e.g. basiccluster.yml.

Since the modifications are carried out per status / change, we recommend naming the new configuration files and adding a timestamp, if necessary.

For Example:

basiccluster.yml
20240101-add-worker4.yml
20240101-drain-worker2.yml
20240101-update-kubeversion.yml