#persistent storage

Persistent Container Storage Solutions | HPE Primera

Persistent Container Storage Solutions | HPE Primera

HPE 3PAR StoreServ 8000 Container Storage

At Red Hat Summit ’19 I cobbled together a deck to glance over some of the best practices for running stateful workloads on Kubernetes that requires persistent storage served by SAN storage. Since these tidbits are difficult to find in one place, I thought I’d summarize my the overview for easy consumption.

Definition and prerequisite reading

What do we mean when we say SAN storage? It’s essentially a block device presented to a compute node which has either an XFS or ext4 formatted filesystem on it. In turn, this filesystem is then attached/detached and mount/unmounted by (in our case) a FlexVolume driver as defined in a PersistentVolume spec. This introduce a number of challenges as any given volume can only be attached to a single Pod at any given time.

This is being discussed at length and depth in the blog I wrote on Highly-Available Stateful Workloads on Kubernetes with SAN storage.

Some of the the key concepts discussed in that blog have been simplified, PowerPoint friendly and outlined below.

Dynamic Provisioning

For context, let’s include the dynamic provisioning diagram. A cluster administrator declares a StorageClass, a user declares a PersistentVolumeClaim (PVC) against a StorageClass API object and the dynamic provisioner will in turn create the PersistentVolume (PV) API object (and create the volume on the Nimble array of course).

StorageClass anatomy

The below example is a pretty typical StorageClass for Nimble.

Annotations

What will drive your users mad is if there’s no default StorageClass declared on the cluster. PVCs will hang indefinitely when a user paste something they found on the Internet and had a PVC nestled in the declaration that assumed a default StorageClass.

Parameters

The provisioner specific parameters should be representable of the underlying storage capabilities. The default StorageClass should in turn reflect what the typical volume should look like for the environment. The .parameters.allowOverrides parameter is Nimble specific and allows PVCs to be annotated with the desired value, such as a Performance Policy.

Reclaim Policy

The default is to delete the PV when a PVC is deleted. Regardless of what behavior is preferred, it should always explicitly be declared to take out the guesswork.

PVC anatomy

The simplest PVC must contain the .spec.accessModes and .spec.resources.requests.storage.

storageClassName

The .spec.storageClassName is optional and will fall back to the default if omitted. I wouldn’t expect a cluster to have more than a handful of storage classes tailored for the different workloads and storage backends. Users can, by default, list storage classes with kubectl get sc.

Selectors

If using static provisioning, like using the built-in iSCSI/FC plugin to manually provision PVs, annotating those PVs to be matched against certain labels is preferable, than hoping for the “right” PV will be mapped to the requesting PVC. This is done with a common concept called selectors.

accessModes

I’ve previously discussed the different accessModes, included here for completeness (right-click and view image for a larger version).

Workload types

The two workload types we care about for running stateful applications is the Deployment for single instance applications and the StatefulSet for distributed applications.

Deployment

A Deployment is a high-level abstraction of a number of API objects as illustrated in the API object stack.

The single most important parameters are the .spec.replicas and .spec.strategy.type. If these are not lined up as shown above, there will be a number of issues arising.

Setting .spec.replicas to anything larger than 1 will spin up another Pod that will try to deploy an identical pod referencing the same PVC. If this Pod comes up on a different host, depending on the plugin implementation, may cause data corruption. Nimble does not allow two hosts access to the same volume but application-level corruption may still occur and you’ll also end up with orphan pods on the cluster.

The cluster default .spec.strategy.type for Deployments is set to RollingUpdate. That means Kubernetes will always try keeping the minimum amount of replicas up at any cost, even if it means scheduling a new replica without fully dismantling the old one, again, causing orphan pods to never terminate due to Kubernetes effectively pulls the rug underneath the pod before it terminates. Not safe!

StatefulSet

For distributed applications that replicate data on the application level, like etcd, MariaDB replication cluster and MongoDB, the StatefulSet workload is the workload type to use.

The fundamental difference between the Deployment and StatefulSet is how persistent volume claims are referenced. The StatefulSet will create a PVC for each replica as illustrated in the API object stack.

Namespace quotas

A well hidden gem is how you would use a ResourceQuota object and apply to a particular namespace. This is primarily used to put caps on how many PVC resources and can be created and how much capacity can be consumed from a particular StorageClass.

This is especially useful to prevent volume sprawl as it can be used as a DoS (Denial Of Service) if the storage backend is unable to service the requests up to a certain limit (count), capacity or rate.

PV and PVC protection

Impossible to miss today is the fact that Kubernetes 1.14 was released. It's immediately available to install via kubeadm.

Many new features and updates in this first 2019 release, most notable is the GA of Windows nodes and in my ballpark, the Local Persistent Volumes and the local provisioner has also been marked GA.

Windows compute node GA

We’ve been kicking the tires on Windows Containers for quite a while. Will Windows Server 2019 and Kubernetes be the big break for containers on Windows? The temperature has been lukewarm at best in what I observe on a daily basis. Sifting through the docs for any persistent storage support I was surprised to see the FlexVolume Plugin being supported and I struggle to find any evidence of Container Storage Interface (CSI) support for Windows compute nodes or is that a given? If it’s not completely obvious, the control plane still requires to be run on Linux. Good news is that networking will simply “work” between Linux and Windows Pods through Flannel and a few others.

Persistent Local Volumes

Anything storage related grep my attention. But why would anyone care about Persistent Local Volumes? I see this as much safer way to provide local storage to end-users than giving access to the hostPath plugin. It allows the cluster-admin to define a directory on the compute nodes that will map a Persistent Volume (PV) to a directory within the defined root directory, dynamically provision through a StorageClass using the accompanying local provisioner. Much safer and more secure. DaemonSets and StatefulSets with distributed databases and filesystems is the most obvious use case. However, external storage vendors without any sophisticated integration with K8s can simply mount their volumes static and still provide storage for the workload types that doesn’t rely on high-availability storage. Added bonus is that it will scale in terms of PV and Persistent Volume Claims (PVC) count far beyond what any external storage, or Software-defined Storage (SDS) solution for that matter, as a PV simply maps to a directory, no more, no less.

kubectl revamped docs and logo

A new mascot also snuck in, we have an octopus named "kubee-cuddle" featured in the revamped kubecuddle documentation: https://kubectl.docs.kubernetes.io/

Read the release notes in its entirety here.

Update!

With the new laptop came reclaimed disk space and with free space come urges to explore Linux distros. I haven't tried Linux Mint for quite a while and the latest 19.1 Cinnamon edition is a gorgeous desktop!