#virtlet

An unsual but persistent part in the KubeVirt development is the following question:

How does KubeVirt compare to virtlet, runv/hyper, oci-cc-runtime/ClearContainers, RancherVM, and alike?

It's a good question. And it actually took a while to get a clear view on this.

To get this clear view, I'll focus on two details:

KubeVirt is not a container runtime, and thus not using the CRI. The VM runtime (libvirt) is provided in a pod instead.

KubeVirt provides a dedicated API for VMs

Especially the first point is raised pretty often. The reason for this is - eventually - that it looks so obvious to just launch VMs instead of containers on the kubelet side - virtlet, runv, and oci-cc-runtime show that it really is an option (without going to much into the details here). Very often this is directly coupled to the question of why we ship our runtime in a pod.

The answer is simply that so far this setup "works":

we can spawn VMs

our runtime is Kubernetes native

because everything lives in pods, we can be delivered to "any" Kubernetes cluster, without messing with the host or kubelet.

We can be a native Kubernetes add-on.

Btw - A nice side effect is that VMs become just a workload - and this is a good separation of concerns and responsibility. Kubernetes owns the platform, and we provide our application to run specialized (virtual machine) processes on this platform.

But even with these benefits: There are some details (i.e. how to get VMs into ressource groups of other pods) which are a little nasty to solve for us, which might be easier to solve in the CRI based approach.

And we see - we are a few lines into the post, and the discussion is going down the winding road of seeing which runtime approach is the best.

That's how it often goes, but that's not where I want to right now.

KubeVirt is not only a runtime. KubeVirt also has an explicit API for virtualization. Initially we used TPRs, then as Kubernetes extension capabilities evolved we went over to CRDs, and are now working on our own User API Server to be used with API Server aggregation. Regardless of this history - The important bit is that we have an explicit API dedicated to virtualization.

And this - in retrospective (and forward facing) - is the significant difference.

The explicit virtualization related API - which consists of new top level types for different things in the virtualization area (like VMS) - gives us the freedom to express all the functionality we need to provide a robust, solid, and stable virtualization platform.

The other approaches (except for RancherVM) have a different view here: They re-use (intentionally and with different motivations) the existing pod specification, to derive VMs from them. But this approach comes to it's limits as soon as virtualization specific functionality (for example live migration) is needed. For those features pod's don't have an API, and annotations or other workarounds need to be used to "logically extend" the pod concept, to enable this missing (virtualization) functionality.

I'm not saying that the one way or the other is technically wrong. It's just that both approaches might just be addressig different goals. I.e. oci-cc-runtime has clearly the goal of using VMs for isolation purpose only - So far it does not look as if they want to expose them as "fully fledged VMs".

And to close the loop: The API is pretty independent of the runtime, and this tells me that in the end it is not that important how the VMs are run - with a runtime in a pod or via a CRI based runtime. The important bit is, that with KubeVirt the user is given the full controll over his VMs.

Side note - Instead of a dedicated API, there is also the option to extend the pod specification using annotations to cover the virtualization specific properties. The issue is, that this does not scale, and will bloat the pod specification.