VMware
Creating a Cluster via the govc
CLI
In this guide we will create an HA Kubernetes cluster with 2 worker nodes.
We will use the govc
cli which can be downloaded here.
Prereqs/Assumptions
This guide will use the virtual IP (“VIP”) functionality that is built into Talos in order to provide a stable, known IP for the Kubernetes control plane. This simply means the user should pick an IP on their “VM Network” to designate for this purpose and keep it handy for future steps.
Create the Machine Configuration Files
Generating Base Configurations
Using the VIP chosen in the prereq steps, we will now generate the base configuration files for the Talos machines.
This can be done with the talosctl gen config ...
command.
Take note that we will also use a JSON6902 patch when creating the configs so that the control plane nodes get some special information about the VIP we chose earlier, as well as a daemonset to install vmware tools on talos nodes.
First, download cp.patch.yaml
to your local machine and edit the VIP to match your chosen IP.
You can do this by issuing: curl -fsSLO https://raw.githubusercontent.com/siderolabs/talos/master/website/content/v1.9/talos-guides/install/virtualized-platforms/vmware/cp.patch.yaml
.
It’s contents should look like the following:
- op: add
path: /machine/network
value:
interfaces:
- interface: eth0
dhcp: true
vip:
ip: <VIP>
With the patch in hand, generate machine configs with:
$ talosctl gen config vmware-test https://<VIP>:<port> --config-patch-control-plane @cp.patch.yaml
created controlplane.yaml
created worker.yaml
created talosconfig
At this point, you can modify the generated configs to your liking if needed.
Optionally, you can specify additional patches by adding to the cp.patch.yaml
file downloaded earlier, or create your own patch files.
Validate the Configuration Files
$ talosctl validate --config controlplane.yaml --mode cloud
controlplane.yaml is valid for cloud mode
$ talosctl validate --config worker.yaml --mode cloud
worker.yaml is valid for cloud mode
Set Environment Variables
govc
makes use of the following environment variables
export GOVC_URL=<vCenter url>
export GOVC_USERNAME=<vCenter username>
export GOVC_PASSWORD=<vCenter password>
Note: If your vCenter installation makes use of self signed certificates, you’ll want to export
GOVC_INSECURE=true
.
There are some additional variables that you may need to set:
export GOVC_DATACENTER=<vCenter datacenter>
export GOVC_RESOURCE_POOL=<vCenter resource pool>
export GOVC_DATASTORE=<vCenter datastore>
export GOVC_NETWORK=<vCenter network>
Choose Install Approach
As part of this guide, we have a more automated install script that handles some of the complexity of importing OVAs and creating VMs. If you wish to use this script, we will detail that next. If you wish to carry out the manual approach, simply skip ahead to the “Manual Approach” section.
Scripted Install
Download the vmware.sh
script to your local machine.
You can do this by issuing curl -fsSL "https://raw.githubusercontent.com/siderolabs/talos/master/website/content/v1.9/talos-guides/install/virtualized-platforms/vmware/vmware.sh" | sed s/latest/v1.9.0-alpha.0/ > vmware.sh
.
This script has default variables for things like Talos version and cluster name that may be interesting to tweak before deploying.
The script downloads VMWare OVA with talos-vmtoolsd
from Image Factory extension pre-installed.
Import OVA
To create a content library and import the Talos OVA corresponding to the mentioned Talos version, simply issue:
./vmware.sh upload_ova
Create Cluster
With the OVA uploaded to the content library, you can create a 5 node (by default) cluster with 3 control plane and 2 worker nodes:
./vmware.sh create
This step will create a VM from the OVA, edit the settings based on the env variables used for VM size/specs, then power on the VMs.
You may now skip past the “Manual Approach” section down to “Bootstrap Cluster”.
Manual Approach
Import the OVA into vCenter
A talos.ova
asset is available from Image Factory.
We will refer to the version of the release as $TALOS_VERSION
below.
It can be easily exported with export TALOS_VERSION="v0.3.0-alpha.10"
or similar.
The download link already includes the talos-vmtoolsd
extension.
curl -LO https://factory.talos.dev/image/903b2da78f99adef03cbbd4df6714563823f63218508800751560d3bc3557e40/${TALOS_VERSION}/vmware-amd64.ova
Create a content library (if needed) with:
govc library.create <library name>
Import the OVA to the library with:
govc library.import -n talos-${TALOS_VERSION} <library name> /path/to/downloaded/talos.ova
Create the Bootstrap Node
We’ll clone the OVA to create the bootstrap node (our first control plane node).
govc library.deploy <library name>/talos-${TALOS_VERSION} control-plane-1
Talos makes use of the guestinfo
facility of VMware to provide the machine/cluster configuration.
This can be set using the govc vm.change
command.
To facilitate persistent storage using the vSphere cloud provider integration with Kubernetes, disk.enableUUID=1
is used.
govc vm.change \
-e "guestinfo.talos.config=$(cat controlplane.yaml | base64)" \
-e "disk.enableUUID=1" \
-vm control-plane-1
Update Hardware Resources for the Bootstrap Node
-c
is used to configure the number of cpus-m
is used to configure the amount of memory (in MB)
govc vm.change \
-c 2 \
-m 4096 \
-vm control-plane-1
The following can be used to adjust the EPHEMERAL disk size.
govc vm.disk.change -vm control-plane-1 -disk.name disk-1000-0 -size 10G
govc vm.power -on control-plane-1
Create the Remaining Control Plane Nodes
govc library.deploy <library name>/talos-${TALOS_VERSION} control-plane-2
govc vm.change \
-e "guestinfo.talos.config=$(base64 controlplane.yaml)" \
-e "disk.enableUUID=1" \
-vm control-plane-2
govc library.deploy <library name>/talos-${TALOS_VERSION} control-plane-3
govc vm.change \
-e "guestinfo.talos.config=$(base64 controlplane.yaml)" \
-e "disk.enableUUID=1" \
-vm control-plane-3
govc vm.change \
-c 2 \
-m 4096 \
-vm control-plane-2
govc vm.change \
-c 2 \
-m 4096 \
-vm control-plane-3
govc vm.disk.change -vm control-plane-2 -disk.name disk-1000-0 -size 10G
govc vm.disk.change -vm control-plane-3 -disk.name disk-1000-0 -size 10G
govc vm.power -on control-plane-2
govc vm.power -on control-plane-3
Update Settings for the Worker Nodes
govc library.deploy <library name>/talos-${TALOS_VERSION} worker-1
govc vm.change \
-e "guestinfo.talos.config=$(base64 worker.yaml)" \
-e "disk.enableUUID=1" \
-vm worker-1
govc library.deploy <library name>/talos-${TALOS_VERSION} worker-2
govc vm.change \
-e "guestinfo.talos.config=$(base64 worker.yaml)" \
-e "disk.enableUUID=1" \
-vm worker-2
govc vm.change \
-c 4 \
-m 8192 \
-vm worker-1
govc vm.change \
-c 4 \
-m 8192 \
-vm worker-2
govc vm.disk.change -vm worker-1 -disk.name disk-1000-0 -size 10G
govc vm.disk.change -vm worker-2 -disk.name disk-1000-0 -size 10G
govc vm.power -on worker-1
govc vm.power -on worker-2
Bootstrap Cluster
In the vSphere UI, open a console to one of the control plane nodes. You should see some output stating that etcd should be bootstrapped. This text should look like:
"etcd is waiting to join the cluster, if this node is the first node in the cluster, please run `talosctl bootstrap` against one of the following IPs:
Take note of the IP mentioned here and issue:
talosctl --talosconfig talosconfig bootstrap -e <control plane IP> -n <control plane IP>
Keep this IP handy for the following steps as well.
Retrieve the kubeconfig
At this point we can retrieve the admin kubeconfig
by running:
talosctl --talosconfig talosconfig config endpoint <control plane IP>
talosctl --talosconfig talosconfig config node <control plane IP>
talosctl --talosconfig talosconfig kubeconfig .
Configure talos-vmtoolsd
The talos-vmtoolsd application was deployed as a daemonset as part of the cluster creation; however, we must now provide a talos credentials file for it to use.
Create a new talosconfig with:
talosctl --talosconfig talosconfig -n <control plane IP> config new vmtoolsd-secret.yaml --roles os:admin
Create a secret from the talosconfig:
kubectl -n kube-system create secret generic talos-vmtoolsd-config \
--from-file=talosconfig=./vmtoolsd-secret.yaml
Clean up the generated file from local system:
rm vmtoolsd-secret.yaml
Once configured, you should now see these daemonset pods go into “Running” state and in vCenter, you will now see IPs and info from the Talos nodes present in the UI.