Version v1.6 of the documentation is no longer actively maintained. The site that you are currently viewing is an archived snapshot. For up-to-date documentation, see the latest version.

Replicated Local Storage

Using local storage with OpenEBS Jiva

If you want to use replicated storage leveraging disk space from a local disk with Talos Linux installed, OpenEBS Jiva is a great option. This requires installing the iscsi-tools system extension.

Since OpenEBS Jiva is a replicated storage, it’s recommended to have at least three nodes where sufficient local disk space is available. The documentation will follow installing OpenEBS Jiva via the offical Helm chart. Since Talos is different from standard Operating Systems, the OpenEBS components need a little tweaking after the Helm installation. Refer to the OpenEBS Jiva documentation if you need further customization.

NB: Also note that the Talos nodes need to be upgraded with --preserve set while running OpenEBS Jiva, otherwise you risk losing data. Even though it’s possible to recover data from other replicas if the node is wiped during an upgrade, this can require extra operational knowledge to recover, so it’s highly recommended to use --preserve to avoid data loss.

Preparing the nodes

Create the boot assets which includes the iscsi-tools system extensions (or create a custom installer and perform a machine upgrade if Talos is already installed).

Create a machine config patch with the contents below and save as patch.yaml

      - destination: /var/openebs/local
        type: bind
        source: /var/openebs/local
          - bind
          - rshared
          - rw

Apply the machine config to all the nodes using talosctl:

talosctl -e <endpoint ip/hostname> -n <node ip/hostname> patch mc -p @patch.yaml

The extension status can be verified by running the following command:

talosctl -e <endpoint ip/hostname> -n <node ip/hostname> get extensions

An output similar to below can be observed:

NODE            NAMESPACE   TYPE              ID                                          VERSION   NAME          VERSION   runtime     ExtensionStatus   1         iscsi-tools   v0.1.1

The service status can be checked by running the following command:

talosctl -e <endpoint ip/hostname> -n <node ip/hostname> services

You should see that the ext-tgtd and the ext-iscsid services are running.

NODE            SERVICE      STATE     HEALTH   LAST CHANGE     LAST EVENT   apid         Running   OK       64h57m15s ago   Health check successful   containerd   Running   OK       64h57m23s ago   Health check successful   cri          Running   OK       64h57m20s ago   Health check successful   etcd         Running   OK       64h55m29s ago   Health check successful   ext-iscsid   Running   ?        64h57m19s ago   Started task ext-iscsid (PID 4040) for container ext-iscsid   ext-tgtd     Running   ?        64h57m19s ago   Started task ext-tgtd (PID 3999) for container ext-tgtd   kubelet      Running   OK       38h14m10s ago   Health check successful   machined     Running   ?        64h57m29s ago   Service started as goroutine   trustd       Running   OK       64h57m19s ago   Health check successful   udevd        Running   OK       64h57m21s ago   Health check successful

Install OpenEBS Jiva

helm repo add openebs-jiva
helm repo update
helm upgrade --install --create-namespace --namespace openebs --version 3.2.0 openebs-jiva openebs-jiva/jiva

This will create a storage class named openebs-jiva-csi-default which can be used for workloads. The storage class named openebs-hostpath is used by jiva to create persistent volumes backed by local storage and then used for replicated storage by the jiva controller.

Patching the Namespace

when using the default Pod Security Admissions created by Talos you need the following labels on your namespace: privileged privileged privileged

or via kubectl:

kubectl label ns openebs

Number of Replicas

By Default Jiva uses 3 replicas if your cluster consists of lesser nodes consider setting defaultPolicy.replicas to the number of nodes in your cluster e.g. 2.

Patching the jiva installation

Since Jiva assumes iscisd to be running natively on the host and not as a Talos extension service, we need to modify the CSI node daemonset to enable it to find the PID of the iscsid service. The default config map used by Jiva also needs to be modified so that it can execute iscsiadm commands inside the PID namespace of the iscsid service.

Start by creating a configmap definition named config.yaml as below:

apiVersion: v1
kind: ConfigMap
  labels: pulumi
  name: openebs-jiva-csi-iscsiadm
  namespace: openebs
  iscsiadm: |
    iscsid_pid=$(pgrep iscsid)

    nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsiadm "$@"    

Replace the existing config map with the above config map by running the following command:

kubectl --namespace openebs apply --filename config.yaml

Now we need to update the jiva CSI daemonset to run with hostPID: true so it can find the PID of the iscsid service, by running the following command:

kubectl --namespace openebs patch daemonset openebs-jiva-csi-node --type=json --patch '[{"op": "add", "path": "/spec/template/spec/hostPID", "value": true}]'

Testing a simple workload

In order to test the Jiva installation, let’s first create a PVC referencing the openebs-jiva-csi-default storage class:

kind: PersistentVolumeClaim
apiVersion: v1
  name: example-jiva-csi-pvc
  storageClassName: openebs-jiva-csi-default
    - ReadWriteOnce
      storage: 4Gi

and then create a deployment using the above PVC:

apiVersion: apps/v1
kind: Deployment
  name: fio
      name: fio
  replicas: 1
    type: Recreate
    rollingUpdate: null
        name: fio
      - name: perfrunner
        image: openebs/tests-fio
        command: ["/bin/bash"]
        args: ["-c", "while true ;do sleep 50; done"]
        - mountPath: /datadir
          name: fio-vol
      - name: fio-vol
          claimName: example-jiva-csi-pvc

You can clean up the test resources by running the following command:

kubectl delete deployment fio
kubectl delete pvc example-jiva-csi-pvc