How to run a virtual machine on Kubernetes using KubeVirt

Contents

Introduction ⚓

I’m a big fan of Kubernetes 🔗, but virtual machines still have their place. Sometimes you just need a full-on virtualized PC and not a container. At the same time, I’ve gotten used to running all of my services on Kubernetes, so I got to thinking: can I run a virtual machine inside Kubernetes? As it turns out (unsurprisingly), I’m not the first person to think this.

Here’s how I learned about KubeVirt, and spun up a virtual machine on my Kubernetes server.

A quick intro to containers and Kubernetes ⚓

For the uninitiated and non-nerdy, a container is a method of running applications as portable, self-contained units. Think of it like sending a letter to someone—instead of just dropping a letter in the postbox, you’d use an envelope as a container to wrap it in before sending it out. But when you have a bunch of letters to send at once, you want some kind of system to handle creating envelopes, wrapping the letters, and tracking them. That’s what Kubernetes is to containers.

Imagine you wanted to host a website, like this blog. You could get a regular computer, run a bunch of commands to install a web server on it, copy the raw files over, then configure the web server to serve those files. That’s the traditional way of doing things, and it’s all well and good for a single website, but what if you wanted to run a dozen websites like this? A hundred? A thousand? Would you want to run through this process each time? Probably not. At any rate, would you want to monitor all of these sites in case one went offline? Sounds like a lot of work.

Instead, you could take your website files and package them into a container. The container includes everything needed to run the site, including the web server, configuration, and files. You can define this container using what’s called a manifest, then use Kubernetes to deploy it. You can reuse this container for different sites by just copying in a different set of files. Kubernetes will take each container, figure out where and how to deploy them, and do all the rest, including monitoring them to see if they go offline.

However, Kubernetes is a complex system. It introduces tons of unique concepts and terminology, has a labyrinthine set of interactions between resources, takes some in-depth system administration knowledge to set up, and has some esoteric error messages (like the dreaded CrashLoopBackOff). The good news is there are projects that make Kubernetes relatively easy to install, including the one I’m using: K3s 🔗.

K3s is a lightweight distribution of Kubernetes that makes installing and managing a Kubernetes cluster relatively painless. All you need to do is run a script on the computers you want to run K3s on. If you want to create a cluster by grouping multiple computers into the same Kubernetes cluster, you just run the script with the IP address of the primary/controlling computer.

A quick intro to virtual machines ⚓

Before containers became vogue, virtual machines (VMs) were the Big Thing™️. Virtual machines are like the big cousins of containers—they also let you run applications and services in a self-contained package, but the package is an entire computer. Imagine taking your laptop and installing a completely separate copy of Windows (or MacOS or Linux or whatever you use) that you could run without having to reboot. You could have a dozen different copies running completely different applications simultaneously, and as long as you had powerful enough hardware, the copies would have no idea they were virtual.

The great thing about VMs are that they’re fully isolated. I don’t have to worry as much about breaking my Arch Linux installation if I do something stupid in a VM, like accidentally downloading a virus or running a dangerous command. The downside is that VMs generally use far more resources than containers to run the same software. This is because they need to emulate an entire computer, including the operating system. It’s like trying to simulate a second, separate mind within your own mind.

Bridging containers and virtual machines with KubeVirt ⚓

This brings us to KubeVirt 🔗. To put it simply, KubeVirt lets you run virtual machines on Kubernetes as if they were containers.

Why would you want to do that? Well, imagine you have applications that require isolation like in a VM, or that you can’t easily package into a container. Instead of having to manage two separate systems (Kubernetes and a VM hypervisor), you can do it all through Kubernetes. KubeVirt also provides tools for connecting to VMs using a GUI or console.

It’s probably better if I give an example. I have a VM that I use to test software for work. This software is a type of QA software that changes things about the system, like consuming excess CPU and RAM, dropping network connections, and adding latency. Running it directly on my server could interfere with the other services running on the server, or just outright crash the server itself. Putting it on a VM gives me a layer of isolation that isn’t as straightforward to get with a container—plus, if something goes wrong, I can always terminate the VM and delete it, or restore from a snapshot.

Deploying KubeVirt ⚓

Right, so, what do you need to run KubeVirt?

1. A Kubernetes (in my case, K3s) cluster.
2. The kubectl command-line tool.

The first step is to deploy the components that let you run KubeVirt. This is pretty easy, just follow the quickstart instructions 🔗. You’ll also want to install virtctl 🔗, which is a wrapper for kubectl that makes managing and interacting with VMs easier. KubeVirt lets you interact with your VMs by connecting directly to serial console, using SSH, or by using VNC 🔗 to access a graphical interface. If you want to use VNC, you’ll also need to install remote-viewer 🔗.

Unlike virtualization tools like Gnome Boxes or VirtualBox, you can’t connect directly to a KubeVirt VM. You have to use VNC. That said, virtctl works everywhere kubectl does, so it’s very easy to set up and use.

Defining a virtual machine ⚓

You can define a virtual machine instance the same way you’d define a Kubernetes manifest for a container.

First, though, if you’re installing a VM from scratch using an ISO image, you’ll need to enable a feature gate to allow KubeVirt to access the ISO image on the host. I created a YAML file called kubevirt-feature-gate.yaml for that:

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  configuration:
    developerConfiguration:
      featureGates:
        - HostDisk

Now, let’s define our new VM. Here, I’m creating a VM called work-lab and giving it 2 CPU cores and 4 GB of RAM. I’m also attaching a Debian ISO image as the boot image, plus a persistent volume named work-lab-hd that will act as the VM’s hard drive. The VM sees this like a regular drive so you can install to it, but on the backend, it uses normal Kubernetes PersistentVolumes, so you can have it store pretty much anywhere 🔗. Isn’t Kubernetes great?

This volume appears like a regular drive to the VM, so you can install to it and save files to it. On the backend, you can store this drive wherever and however you’d like, including remotely using NFS or S3, and the VM won’t know the difference. Here, I’m storing both the ISO and VM hard drive on a folder on my server called /storage/services/kubevirt/:

# Define storage for the VM
apiVersion: v1
kind: PersistentVolume
metadata:
  name: work-lab-hd
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 60Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/storage/services/kubevirt/work-lab"
---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: work-lab-hd
 spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 60Gi
   storageClassName: manual
---
# Define the VM itself
 apiVersion: kubevirt.io/v1alpha3
 kind: VirtualMachine
 metadata:
   name: work-lab
 spec:
   running: false
   template:
     metadata:
       labels:
         kubevirt.io/domain: work-lab
     spec:
       domain:
         cpu:
           cores: 2
         devices:
           disks:
           - cdrom:
               bus: sata
             name: cdromiso
             bootOrder: 1
           - disk:
               bus: virtio
             name: harddrive
             bootOrder: 2
         machine:
           type: q35
         resources:
           requests:
             memory: 4G
       volumes:
       - name: cdromiso
         hostDisk:
           path: "/storage/Software/ISOs/debian-12.1.0-amd64-netinst.iso"
           type: Disk
       - name: harddrive
         persistentVolumeClaim:
           claimName: work-lab-hd
---

To run this, just save it as a YAML file and apply it using kubectl:

kubectl apply -f work-vm.yaml

Now, we just wait for the instance to come up:

kubectl wait --for=condition=Ready vmis/work-vm

Once it’s up, you can open a graphical interface to the instance using virtctl 🔗:

virtctl vnc work-vm

And there you go! You can now install and use the VM just like if you were using VirtualBox, KVM, or any other VM management tool!

One thing though: after you install the OS and shut down the system, either change the boot order for the ISO image or delete that part of the manifest. Otherwise, the VM will boot into the ISO each time you start it up.

Networking, additional features, etc. ⚓

This blog is already getting super long, so I’ll make this section quick. Your VM instance now behaves like any other Kubernetes resource. You can use Services and Ingresses to manage connectivity, you can scale it up by creating replicas, you can set resource limits, and much more. These are all way beyond the scope of this blog, so check out the KubeVirt docs 🔗 to see what features it supports and how to enable and configure them.

Conclusion ⚓

I’m all for reducing the number of different tools I need to use to manage my systems, and KubeVirt finally let me consolidate my VMs into Kubernetes. It only took me a few hours to learn KubeVirt and recreate my work VM in it, so it’s very accessible if you’re already a Kubernetes user. Check it out at kubevirt.io 🔗!