Setup Cluster (MicroOS)

  6 minute read  

After completing the cluster preparation steps, you are ready to configure the kubeops tooling and create the Kubernetes cluster from your cluster admin host. This Setup Cluster document describes how to install and configure KOSI and kubeopsctl, prepare the cluster configuration, and then trigger the automated cluster deployment.

1. Install KOSI

# KOSI-version can be found under : https://packagerepo.kubeops.net/rpm/
curl -O https://packagerepo.kubeops.net/rpm/kosi-2.15.0.4_Beta0-0.x86_64.rpm
sudo transactional-update pkg install --no-confirm --allow-unsigned-rpm --force kosi-2.15.0.4_Beta0-0.x86_64.rpm

# need reboot after transactional update!
sudo reboot now 

# copy plugins and kosi config from last snapshot
sudo mkdir -p /var/kubeops/plugins
sudo cp -r /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/var/kubeops/plugins/ /var/kubeops/
sudo cp -r /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/var/kubeops/kosi/ /var/kubeops/

2. Set the KUBEOPSROOT env var

Set KUBEOPSROOT and XDG_RUNTIME_DIR in ~/.bashrc

# file ~/.bashrc
# Append these values to the end of your ~/.bashrc file
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

Source .bashrc to apply the values

source ~/.bashrc
echo $KUBEOPSROOT
echo $XDG_RUNTIME_DIR

As a result you should see your KUBEOPSROOT-path two times.

3. Adjust KOSI Configuration

This creates a kubeops directory in your home directory and transfers all necessary files, e.g., the kosi-config and the plugins, to it.

mkdir ~/kubeops
cd ~/kubeops
cp -R /var/kubeops/kosi/ .
cp -R /var/kubeops/plugins/ .
  • Set hub in your kosi config to hub: https://dispatcher.kubeops.net/v4/dispatcher/
  • Set the “plugins”-entry in your kosi config to plugins: /home/<yourUser>/kubeops/plugins, where is changed to your username
# file $KUBEOPSROOT/kosi/config.yaml
apiversion: kubernative/sina/config/v2

spec:
  hub: https://dispatcher.kubeops.net/v4/dispatcher/ # <-- set hub url
  plugins: <your kubeopsroot>/plugins/ # <-- set the path to your plugin folder (~ for home or $KUBEOPSROOT don't work, it has to be the full path)
  workspace: /tmp/kosi/process/
  logging: info
  housekeeping: false
  proxy: false

4. Install KOSI enterprise plugins

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.2.0_Beta0

5. Login with your user

kosi login -u <yourUser>

6. Prepare nodes

kosi install --hub kubeops kubeops/microos-prepare-node:2.2.0_Beta0

# transactional update need reboot!
sudo reboot now

7. Create a cluster-values.yaml configuration file

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your controlplane01 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: MicroOS # Currently, only the builtin MicroOS repository is supported. Local support lacks!
changeCluster: true
zones:
# IMPORTANT: The following part has to be adapted so that every one of your controlplane nodes and workernodes is included
# This file only includes the minimum requirements for the amount of controlplanes and workers and an example usage of zones
# You should adapt this part to your amount of controlplanes and workers and cluster them into as many zones as you like
- name: zone1
  nodes:
  - name: <your controlplane01 hostname>
    iPAddress: <your controlplane01 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker01 hostname>
    iPAddress: <your worker01 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone2
  nodes:
  - name: <your controlplane02 hostname>
    iPAddress: <your controlplane02 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker02 hostname>
    iPAddress: <your worker02 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone3
  nodes:
  - name: <your controlplane03 hostname>
    iPAddress: <your controlplane03 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker03 hostname>
    iPAddress: <your worker03 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
Full Example 
# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: myuser
kubernetesVersion: 1.34.1
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: MicroOS # Currently, only the builtin MicroOS repository is supported. Local support lacks!
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: node01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.34.1
  - name: node04
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.34.1
- name: zone2
  nodes:
  - name: node02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.34.1
  - name: node05
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.34.1
- name: zone3
  nodes:
  - name: node03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.34.1
  - name: node06
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.34.1

7.1 Using KubeVip in your Cluster (optional)

If you want to use KubeVip to setup your Cluster, you need a virtual ip for that. Also you have to set kubeVipEnabled to true and set your virtualIP. If you dont want to use KubeVip you have to set kubeVipEnabled to false and set your first controlplane as your virtualIP in your cluster-values.yaml in the Setup. Refer to the official KubeVip-documentation for details here.

Examples:

kubeVipEnabled: true
virtualIP: <IP in your cluster ip range which is not given yet>

or

kubeVipEnabled: false
virtualIP: <controlplane01 ip>

8. Install kubernetes tools

kosi install --hub kubeops kubeops/microos-kubernetes:1.34.1 -f cluster-values.yaml

# transactional update need reboot!
sudo reboot now 

# file kubectl-only-values.yaml
installKubeadm: false
installKubelet: false
installKubectl: true

kosi install --hub kubeops kubeops/microos-kubernetes:1.35.3 -f cluster-values.yaml -f kubectl-only-values.yaml

# transactional update need reboot!
sudo reboot now

9. Install kubeopsctl on admin node

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
curl -O https://packagerepo.kubeops.net/rpm/kubeopsctl-2.2.0.0_Beta0-0.x86_64.rpm
sudo transactional-update pkg install --no-confirm --allow-unsigned-rpm --force kubeopsctl-2.2.0.0_Beta0-0.x86_64.rpm

# need reboot after transactional update!
sudo reboot now 

sudo cp /.snapshots/$(sudo ls -1 /.snapshots | sed '$!d')/snapshot/usr/local/bin/kubeopsctl /usr/local/bin/

10. Pull required KOSI packages

If you do not specify a parameter, the current Kubernetes version 1.34.1 will be pulled.
With parameter --kubernetesVersion <your wanted Kubernetesversion> you can pull an older Kubernetes version.
Available Kubernetes versions are 

kubeopsctl pull

or

kubeopsctl pull --kubernetesVersion <x.xx.x>

11. Cluster Setup

Make sure that you are logged in on hub and registry.

kosi login -u <your username>

Make sure that you changed the kosi config.yaml.

cat $KUBEOPSROOT/kosi/config.yaml

Make sure that you pulled all required packages.

ls -l $KUBEOPSROOT/packages

Install Kubernetes Cluster with kubeopsctl. Cluster setup takes about 10 minutes.

kubeopsctl apply -f cluster-values.yaml