So in the two earlier installments, I provided an overview of my mind map vis-a-vis Kubernetes and containerization. In this post I will focus on what I did to set up a test rig in my home laboratory environment.
Over the years, my better half and I have acquired a whole bunch of mac minis, which we used to run virtualized Hadoop on. Now one might ask, why mac minis and why not some other form factor for hardware? Here’s why — I had once brought home a bunch of HPE servers to set up a lab in my basement — these were old DL360s and a couple of HP RAID arrays. These were getting discarded from work, and so I requested they be handed over to me, and I set up my little lab. My electric bill skyrocketed since then, and not to mention the noise these server fans made. It became apparent to me that it was untenable. So, subsequently, I found the mac minis to be sizable enough, and not inordinately expensive, that they could serve as desktop computers for all members of the family, and also allow for parasitic deployment of virtual machines of reasonable sizes to run things like Hadoop, or Kubernetes.
So, there’s nothing much to the setup really, only five CentOS 7 VMs built on Virtual Box, each with 2 vCPUs and 4 GB of RAM, spread across the mac minis and my personal MacBook pro. I deployed these on VirtualBox, and each VM was presented with an additional 20GB data disk. The VMs were configured to use the Bridged network mode, and I hard-assigned some IP addresses for them, from outside the DHCP scope of my home network, so their IPs don’t keep changing on me. For hostnames, I just used the hosts files instead of setting up a DNS server for my lab network.
Before delving into the setup itself, let us discuss the planning. A Kubernetes cluster can be set up to be converged or hyper-converged. In the context of virtualization concepts, converged infrastructure is quite well known – there is a shared storage backend (SAN-based arrays), and the hypervisors are connected on the SAN to these arrays. In the case of hyper-convergence, the storage is also resident on the hypervisors, and typically configured in a distributed array configuration (for e.g., Nutanix, VMware vSAN, and so on). In the case of Kubernetes, the hyper-converged model is what is proposed by Cloudera for their private cloud offering for the Cloudera Data Platform (CDP) suite. This seems to be a natural sweet-spot for distributed applications (such as Hadoop-based distros). There are a few solutions available on Kubernetes, such as Ceph (RedHat’s distributed storage framework) or Portworx (which is a very solid storage solution). I decided to try the hyper-converged model, but since I didn’t have enough compute resources or storage to pull off a portworx deployment, I opted to use GlusterFS, which is a “simpler” distributed storage solution that is considered a mature solution on Kubernetes. Portworx has a good article that discusses the hyper-converged model.
Something to consider before setting a cluster up — If you’re simply kicking the proverbial tires you don’t need to set up an HA cluster. But if you plan to build an HA cluster, don’t start with a single master option. I couldn’t find a way to switch a cluster from single-master to multi-master mode. This probably won’t be much of a factor in the ‘real’ world, in data centers, or with cloud providers, unless you are building a sandbox cluster and plan to take it from single master to multi-master.
Have you met “clush” yet?
ClusterShell is a distributed shell framework, wherein it automates running host-level operations that we would run on one node at a time, on a bunch of hosts at the same time. I have used this with Hadoop clusters that I manage(d), OpenStack clusters and pretty much anything that requires more than one host.
Install clustershell (clush) on your admin node of choice.
Set up clustershell and associated configuration file
clustershell
|-- clush.conf
|-- groups
|-- groups.conf
|-- groups.conf.d
| |-- README
| |-- genders.conf.example
| |-- slurm.conf.example
|-- groups.d
| |-- README
| |-- cluster.yaml.example
| |-- local.cfg
|-- topology.conf.example
Set up the /etc/clustershell/groups file as follows –
$ cat clustershell/groups
# ClusterShell groups config local.cfg
#
# Replace /etc/clustershell/groups
#
# Note: file auto-loaded unless /etc/clustershell/groups is present
#
# See also groups.d/cluster.yaml.example for an example of multiple
# sources single flat file setup using YAML syntax.
#
# Feel free to edit to fit your needs.
k8s: 192.168.7.[11-15]
Set up ssh trust to remotely manage all nodes at once
$ ssh-keygen -t ecdsa
$ ssh-add ~/.ssh/id_ecdsa
$ ssh-copy-id -i ~/.ssh/id_ecdsa 196.168.7.11
$ ssh-copy-id -i ~/.ssh/id_ecdsa 196.168.7.12
$ ssh-copy-id -i ~/.ssh/id_ecdsa 196.168.7.13
$ ssh-copy-id -i ~/.ssh/id_ecdsa 196.168.7.14
$ ssh-copy-id -i ~/.ssh/id_ecdsa 196.168.7.15The above sets the trusts up so the user can access the nodes remotely without a password. The ssh key requires a passphrase and should be added to the local ssh-agent for this to work. Each time the admin node is restarted, the ssh key needs to be added to the ssh-agent.
If ssh-agent is not running, run first, before adding the key —
$ eval `ssh-agent`
$ ssh-add ~/.ssh/id_ecdsaOnce the trust has been established, the nodes are ready to be managed in parallel via ssh —
taijitu:.ssh dwai$ cd ~/
taijitu:~ dwai$ clush -g k8s 'uname -a'
196.168.7.11: Linux k8s01 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
196.168.7.12: Linux k8s02 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
196.168.7.13: Linux k8s03 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
and so on...
Now that you’ve been ‘clushed’, update all the nodes to the latest version —
taijitu:etc dwai$ clush -g k8s 'sudo yum -y update'Setting up a single-master cluster
Install the following pre-requisite packages —
taijitu:etc dwai$ clush -g k8s 'sudo yum -y install net-tools wget telnet yum-utils device-mapper-persistent-data lvm2'Add the docker repo —
taijitu:etc dwai$ clush -g k8s 'sudo yum-config-manager add-repo https://download.docker.com/linux/centos/docker-ce.repo'196.168.7.11: Loaded plugins: fastestmirror196.168.7.12: Loaded plugins: fastestmirror196.168.7.13: Loaded plugins: fastestmirrorEnsure the br_netfilter module is loaded in the kernel —
taijitu:etc dwai$ clush -g k8s 'sudo modprobe br_netfilter'
taijitu:etc dwai$ clush -g k8s 'sudo lsmod|grep br_netfilter'
196.168.7.11: br_netfilter 22256 0
196.168.7.11: bridge 151336 2 br_netfilter,ebtable_broute
196.168.7.13: br_netfilter 22256 0
196.168.7.13: bridge 151336 2 br_netfilter,ebtable_broute
196.168.7.12: br_netfilter 22256 0
196.168.7.12: bridge 151336 2 br_netfilter,ebtable_broute
taijitu:etc dwai$
Disable IPv6 and enable bridged traffic (I’ve seen strange issues when ipv6 is in the mix):
- First, create a k8s.conf file
taijitu:~ dwai$ cat k8s.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
- Copy it to all nodes
taijitu:~ dwai$ clush -g k8s -c k8s.conf --dest=~/taijitu:~ dwai$ clush -g k8s 'sudo cp ~/k8s.conf /etc/sysctl.d/k8s.conf'- Enable new sysctl config
taijitu:~ dwai$ clush -g k8s 'sudo sysctl --system'Open Firewall ports
Create a shell script to execute on all nodes —
taijitu:~ dwai$ cat firewall.sh
firewall-cmd --permanent --add-port=6443/tcp # Kubernetes API Server
firewall-cmd --permanent --add-port=2379-2380/tcp # etcd server client API
firewall-cmd --permanent --add-port=10250/tcp # Kubelet API
firewall-cmd --permanent --add-port=10251/tcp # kube-scheduler
firewall-cmd --permanent --add-port=10252/tcp # kube-controller-manager
firewall-cmd --permanent --add-port=10255/tcp
firewall-cmd --permanent --add-port=30000-32767/tcp # NodePort Services
firewall-cmd --permanent --add-port=179/tcp # BGP for Calico
firewall-cmd --permanent --add-port=4789/udp # VXLAN for Calico
firewall-cmd --permanent --add-port=5473/tcp # Typha for Calico
firewall-cmd --permanent --add-port=9153/tcp # DNS
firewall-cmd --permanent --add-port=53/tcp # DNS
firewall-cmd --permanent --add-port=53/udp # DNS
firewall-cmd --permanent --add-port=8472/udp
firewall-cmd --permanent --add-masquerade
firewall-cmd --permanent --add-port=24007/tcp
firewall-cmd --permanent --add-port=24008/tcp
firewall-cmd --permanent --add-port=2222/tcp
firewall-cmd --permanent --add-port=49152-49251/tcp
$ chmod +x firewall.sh
Copy the script over to all nodes of the cluster —
taijitu:~ dwai$ clush -g k8s -c firewall.sh --dest=~/Execute on all nodes of the cluster —
taijitu:~ dwai$ clush -g k8s 'sudo sh -x ~/firewall.sh'Install the container runtime —
taijitu:~ dwai$ clush -g k8s 'sudo yum install -y containerd.io-1.2.13 \
> docker-ce-19.03.11 \
> docker-ce-cli-19.03.11'
Refer to the official k8s docs for correct version, etc.
Set up the Docker daemon config file —
taijitu:~ dwai$ cat daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
Copy and distribute it over to all nodes and start docker —
taijitu:~ dwai$ clush -g k8s -c daemon.json --dest=~/
taijitu:~ dwai$ clush -g k8s 'sudo cp ~/daemon.json /etc/docker/'
taijitu:~ dwai$ clush -g k8s 'sudo mkdir -p /etc/systemd/system/docker.service.d'
taijitu:~ dwai$ clush -g k8s 'sudo systemctl daemon-reload'
taijitu:~ dwai$ clush -g k8s 'sudo systemctl restart docker'
taijitu:~ dwai$ clush -g k8s 'sudo systemctl enable docker'
196.168.7.11: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
196.168.7.12: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
196.168.7.13: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.Install Kubernetes
Create a repo file —
taijitu:~ dwai$ cat kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
Distribute it to all nodes of the cluster —
taijitu:~ dwai$ clush -g k8s -c kubernetes.repo --dest=~/taijitu:~ dwai$ clush -g k8s 'sudo cp ~/kubernetes.repo /etc/yum.repos.d/'Disable selinux
$ clush -g k8s ‘sudo setenforce 0’$ clush -g k8s “sudo sed -i ‘s/^SELINUX=enforcing/SELINUX=permissive/’ etc/selinux/config’Install Kubernetes
taijitu:~ dwai$ clush -g k8s 'sudo yum install -y kubelet kubectl kubeadm --disableexcludes=kubernetes'Start the kubelet
taijitu:~ dwai$ clush -g k8s 'sudo systemctl enable --now kubelet'
192.168.7.11: Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
192.168.7.13: Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
192.168.7.12: Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.Restart all the nodes —
taijitu:~ dwai$ clush -g k8s 'sudo init 6'Single-Master Configuration
Setup the Kubernetes Master
[dwai@k8s01 ~]$ sudo kubeadm init --apiserver-advertise-address=192.168.7.11 --pod-network-cidr=10.244.0.0/16
W1029 16:47:18.787462 20524 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.3
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.7.11]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.7.11 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.7.11 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.002391 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: mi748i.sljdyp3f7t9yvmn7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.7.11:6443 --token mi748i.sljdyp3f7t9yvmn7 \
--discovery-token-ca-cert-hash sha256:1dc7fd39485d17ec585c5b39908e5b6251f3b00667813092facab5fc247a8980
Network Plugin installation
We installed flannel —
https://coreos.com/flannel/docs/latest/kubernetes.html
Important for Flannel to work, the pod-cidr-network needs to be 10.244.0.0/16
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.ymlJoin the Worker nodes to the cluster
On each worker node, run the following as the root user —
kubeadm join 192.168.7.11:6443 --token mi748i.sljdyp3f7t9yvmn7 \
--discovery-token-ca-cert-hash sha256:1dc7fd39485d17ec585c5b39908e5b6251f3b00667813092facab5fc247a8980 Configure authentication/authorization for k8s administration as follows (on the master node) —
$ mkdir -p $HOME/.kube
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ chown $(id -u):$(id -g) $HOME/.kube/config
After running successfully, the kubectl command should show the nodes in ready state —
[dwai@k8s01 ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s01 Ready master 18h v1.19.3
k8s02 Ready <none> 18h v1.19.3
k8s03 Ready <none> 18h v1.19.3
k8s04 Ready <none> 18h v1.19.3
k8s05 Ready <none> 18h v1.19.3Related Posts
- Platform Automation and Infrastructure As Code – 3
- Setting up a log management system on Kubernetes
- Installing metrics-server on a kubeadm managed k8s cluster
