Topics

#claim #cluster #csi #kubernetes #kubernetes-cluster #moosefs #persistent #persistent-storage #persistent-volume #persistent-volume-claim #storage #storage-providers-moosefs #volume

Language breakdown

No language data available.

▸ Files click to expand

README

Container Storage Interface (CSI) driver for MooseFS

Container storage interface is an industry standard that enables storage vendors to develop a plugin once and have it work across a number of container orchestration systems.

MooseFS is a petabyte Open-Source distributed file system. It aims to be fault-tolerant, highly available, highly performing, scalable general-purpose network distributed file system for data centers.

MooseFS source code can be found on GitHub.

Note that a pool of MooseFS Clients that are available for use by containers is created on each node. By default the number of MooseFS Clients in the pool is 1.

Changelog

Driver verson 1.1.0

Update gRPC to version 1.79.3.
Fix RBAC for csi-resizer (nvtkaszpir)
Add volume prefix for multi-storage-class provisioning (chaserhkj)
Update Image to Alpine Linux 3.24.

Driver verson 1.0.0

Update MooseFS client to version 4.59.2.
Update Image to Alpine Linux 3.23.
Update the GO libraries to fix vulnerabilities.
Allow support for fsGroup.
Update csi-node-driver-registrar url.

Driver verson 0.9.8

MooseFS client updated to version 4.58.3.
Image OS changed to Alpine Linux 3.22
The provisioner and registrar images entries have been updated.
Update the Dockerfile to build only mfsmount and mfscli.

Driver verson 0.9.7

Added support for MooseFS 4 client.
Enabled passing additional mfsmount parameters during the mount process (password and more).
Support for cross-platform compilation has been enabled.
Repository images support AMD64, ARM64 and ARMv7 architectures by default.

Installation on Kubernetes

Prerequisites

MooseFS Cluster up and running

--allow-privileged=true flag set for both API server and kubelet (default value for kubelet is true)

Deployment

Edit deploy/kubernetes/csi-moosefs-config.yaml config map file with your settings:

* masterhost – domain name (recommended) or IP address of your MooseFS Master Server(s). It is an equivalent to -H masterhost or -o mfsmaster=master_host passed to MooseFS Client. * masterport – port number of your MooseFS Master Server. It is an equivalent to -P masterport or -o mfsport=master_port passed to MooseFS Client. * k8srootdir – each mount's root directory on MooseFS. Each path is relative to this one. Equivalent to -S k8srootdir or -o mfssubfolder=k8srootdir passed to MooseFS Client. * driverworkingdir – a driver working directory inside MooseFS where persistent volumes, logs and metadata is stored (actual path is: k8srootdir/driverworkingdir) * mount_count – number of pre created MooseFS clients running on each node * mfslogging – driver can create logs from each component in k8srootdir/driverworking_dir/logs directory. Boolean "true"/"false" value.

Apply csi-moosefs-config config map to the cluster:

$ kubectl apply -f deploy/kubernetes/csi-moosefs-config.yaml

Check the config map status:

$ kubectl get configmap -n kube-system
    NAME                                 DATA   AGE
    csi-moosefs-config                   6      42s

Update deploy/csi-moosefs.yaml file with the aproprieate image:

The default image consists of the latest version of the CSI plug-in and the latest version of the MooseFS Community Edition client:

* Locate image definition under the csi-moosefs-plugin plugin name (line 237 and line 337) image: moosefs/moosefs-csi:latest * Update the image version suffix in the plugin's section accordingly: * 1.1.0-4.59.2 – plugin version 1.1.0 and MooseFS CE 4.59.2 * 1.0.0-4.59.2 – plugin version 1.0.0 and MooseFS CE 4.59.2 * 0.9.8-4.58.0 – plugin version 0.9.8 and MooseFS CE 4.58.0 * 0.9.8-4.57.7 – plugin version 0.9.8 and MooseFS CE 4.57.7

You can find a complete list of available images at: \ https://hub.docker.com/r/moosefs/moosefs-csi/tags

Fot driver with MooseFS client PRO version: * moosefs/moosefs-csi:latest-pro – latest plugin version and MooseFS PRO client * moosefs/moosefs-csi:1.1.0-4.59.2-pro – plugin version 1.1.0 and MooseFS PRO 4.59.2 * moosefs/moosefs-csi:1.0.0-4.59.2-pro – plugin version 1.0.0 and MooseFS PRO 4.59.2

Note there are two occurrences of csi-moosefs-plugin in csi-moosefs.yaml file and it is necessary to update the image version in both places of the file.

Deploy CSI MooseFS plugin along with CSI Sidecar Containers:

$ kubectl apply -f deploy/kubernetes/csi-moosefs.yaml

Ensure that all the containers are ready, up and running

kube@k-master:~$ kubectl get pods -n kube-system | grep csi-moosefs
    csi-moosefs-controller-0                   4/4     Running   0          44m
    csi-moosefs-node-7h4pj                     2/2     Running   0          44m
    csi-moosefs-node-8n5hj                     2/2     Running   0          44m
    csi-moosefs-node-n4prg                     2/2     Running   0          44m

You should see a single csi-moosefs-controller-x running and csi-moosefs-node-xxxxx one per each node.

You may also take a look at your MooseFS CGI Monitoring Interface ("Mounts" tab) to check if new Clients are connected – mount points: /mnt/controller and /mnt/${nodeId}[_${mountId}].

Verification

Create a persistent volume claim for 5 GiB:

$ kubectl apply -f examples/kubernetes/dynamic-provisioning/pvc.yaml

Verify if the persistent volume claim exists and wait until it's STATUS is Bound:

$ kubectl get pvc
    NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
    my-moosefs-pvc   Bound    pvc-a62451d4-0d75-4f81-bfb3-8402c59bfc25   5Gi        RWX            moosefs-storage   69m

After its in Bound state, create a sample workload that mounts the volume:

$ kubectl apply -f examples/kubernetes/dynamic-provisioning/pod.yaml

Verify the storage is mounted:

$ kubectl exec my-moosefs-pod -- df -h /data
    Filesystem                Size      Used Available Use% Mounted on
    172.17.2.80:9421          4.2T      1.4T      2.8T  33% /data

You may take a look at MooseFS GUI Monitoring Interface ("Quotas" tab) to check if a quota for 5 GiB on a newly created volume directory has been set. Dynamically provisioned volumes are stored on MooseFS in k8srootdir/driverworkingdir/volumes directory.

Clean up:

$ kubectl delete -f examples/kubernetes/dynamic-provisioning/pod.yaml
    $ kubectl delete -f examples/kubernetes/dynamic-provisioning/pvc.yaml

More examples and capabilities

Multi storage class provisioning

MooseFS CSI Driver starting from version 1.1.0 allows users to provision different Kubernetes storage class volumes to different path prefixes within MooseFS. By configuring distinct volumePrefix values for different StorageClass objects, then users can use the MooseFS tool mfssclass set on subdirectories to map specific k8s storage classes directly to specyfic MooseFS storage classes, enabling fine-grained storage tiering and management.

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: moosefs-3cp
provisioner: csi.moosefs.com
parameters:
  volumePrefix: "3CP/"

Than set correspondingly in moosefs:

$ mfssclass set 3CP /pv_data/volumes/3CP

Volume Expansion

Volume expansion can be done by updating and applying corresponding PVC specification.

Note: the volume size can only be increased. Any attempts to decrease it will result in an error. It is not recommended to resize Persistent Volume MooseFS-allocated quotas via MooseFS native tools, as such changes will not be visible in your Container Orchestrator.

Static provisioning

Volumes can be also provisioned statically by creating or using a existing directory in k8srootdir/driverworkingdir/volumes. Example PersistentVolume examples/kubernetes/static-provisioning/pv.yaml definition, requires existing volume in volumes directory.

Mount MooseFS inside containers

It is possible to mount any MooseFS directory inside containers using static provisioning.

Create a Persistent Volume (examples/kubernetes/mount-volume/pv.yaml):

kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: my-moosefs-pv-mount
    spec:
      storageClassName: ""               # empty Storage Class
      capacity:
        storage: 1Gi                     # required, however does not have any effect
      accessModes:
        - ReadWriteMany
      csi:
        driver: csi.moosefs.com
        volumeHandle: my-mount-volume   # unique volume name
        volumeAttributes:
          mfsSubDir: "/"                 # subdirectory to be mounted as a rootdir (inside k8srootdir)

Create corresponding Persistent Volume Claim (examples/kubernetes/mount-volume/pvc.yaml):

kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: my-moosefs-pvc-mount
    spec:
      storageClassName: ""               # empty Storage Class
      volumeName: my-moosefs-pv-mount
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi                   # at least as much as in PV, does not have any effect

Apply both configurations:

$ kubectl apply -f examples/kubernetes/mount-volume/pv.yaml
    $ kubectl apply -f examples/kubernetes/mount-volume/pvc.yaml

Verify that PVC exists and wait until it is bound to the previously created PV:

$ kubectl get pvc
    NAME                    STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    my-moosefs-pvc-mount    Bound     my-moosefs-pv-mount   1Gi        RWX                           23m

Create a sample workload that mounts the volume:

$ kubectl apply -f examples/kubernetes/mount-volume/pod.yaml

Verify that the storage is mounted:

$ kubectl exec -it my-moosefs-pod-mount -- ls /data

You should see the content of k8srootdir/mfsSubDir.

Clean up:

$ kubectl delete -f examples/kubernetes/mount-volume/pod.yaml
    $ kubectl delete -f examples/kubernetes/mount-volume/pvc.yaml
    $ kubectl delete -f examples/kubernetes/mount-volume/pv.yaml

By using containers[].volumeMounts[].subPath field of PodSpec it is possible to specify a proper MooseFS subdirectory using only one PV/PVC pair, without creating a new one for each subdirectory:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: my-site-app
spec:
  template:
    spec:
      containers:
        - name: my-frontend
          # ...
          volumeMounts:
            - name: my-moosefs-mount
              mountPath: "/var/www/my-site/assets/images"
              subPath: "resources/my-site/images"
            - name: my-moosefs-mount
              mountPath: "/var/www/my-site/assets/css"
              subPath: "resources/my-site/css"
      volumes:
        - name: my-moosefs-mount
          persistentVolumeClaim:
            claimName: my-moosefs-pvc-mount

Version Compatibility

| Kubernetes | MooseFS CSI Driver | |:----------:|:------------------:| | v1.26 | v0.9.7 | | v1.32 | v0.9.8 | | v1.33 | v1.0.0 |

Copyright

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and limitations under the License.