#imgbased

oVirt Node, the slim OS acting as a hypervisor for oVirt deployments showed it's age already a few years ago. It was time to take some action, to not drown in endless bugs which were contradicting how operating systems are put together today.

We started researching around the end of 2014. Main goals were to leverage more open source projects - basically anti-NIH. The benefits are obvious: Why write our own installer if there is one? Why create an admin interface if there are a few?

We looked at deploying oVirt in containers back then, and also at delivering it as an ostree - both had drawbacks which we could not work around at that time.

Anyhow, after a long decision period, a couple of POCs, more bug fixes on vintage Node we are finally here: oVirt Node 4.0.

Features, in chronological order

Ready to use with oVirt Engine

Self Hosted Engine support

CentOS 7 as a base

Anaconda for installation

squashfs and rpm as a delivery format

Cockpit for administration

imgbased for LVM management

rpm for update delivery

Writable root file-system

Continuous integration and testing is a hot topic. It is so important, it is so good, because it saves us from doing lame stuff ourselfs.

Lago is oVirt's answer to allow fully automated testing with multiple hosts. And it is good at this. Node support has landed a while ago, we just need to get it into production. But for basic Node centric sanity this full blown setup is not needed.

With a simple setup, we are now running sanity tests on a running VM in jenkins.

Our sanity tests now cover the basic building blocks of Node:

Correct bootloader configuration

Correct package set

Correct LVM layout

No SELinux denials

Correct enabled services

Correct mount options

Working imgbased

Thesse items are crucial for the core functionality of node (working upgrades, reliable rollback). And up to now there is nothing oVirt specific, and this is fine, because integration tests in Lago will ensure that the integration is working.

And these tests are isolated (snapshotted), and fast. The complete testing is done in under 2 minutes. The value is high because it will catch at least:

Any build related error

Any partitioning related error

Any severe bug in imgbased

The biggest gap is to ensure that updates are working. More tests can be added, all of this can be driven from python unit-tests.

Let me note that the testing process is robust. The tests are run in real VMs, and are thus not impacted by the host configuration.

The testing is triggered quite easily, in the ovirt-node-ng repository, run:

# Is implicitly called, but this does build the image # 85% of the time make squashfs # This will install the image into a qcow2 # 10% of the time make installed-squashfs # This will check the qcow2 # 5% of the time make check

And this is finally how such a test run looks:

00:23:51.931 The squashfs 'ovirt-node-ng-image.squashfs.img' got installed into the file 'ovirt-node-ng-image.installed.qcow2' 00:23:51.986 ++ sudo -E make check 00:23:51.998 make -C tests check 00:23:52.000 make[1]: Entering directory '/home/jenkins/workspace/ovirt-node-ng_master_build-artifacts-fc22-x86_64/ovirt-node-ng/tests' 00:23:52.004 pyflakes testIntegration.py virt.py agent.py testSanity.py testVirt.py 00:23:52.190 pep8 testIntegration.py virt.py agent.py testSanity.py testVirt.py 00:23:52.481 nosetests --with-xunit -v --no-byte-compile --stop testVirt.py testSanity.py 00:24:43.359 test_agent_works (testVirt.TestNodeTestcase) ... ok 00:24:55.888 test_reboot_works (testVirt.TestNodeTestcase) ... ok 00:25:46.463 test_bootloader (testSanity.TestNode) ... ok 00:25:51.445 test_imgbase (testSanity.TestNode) ... ok 00:25:54.761 test_lvm (testSanity.TestNode) ... ok 00:25:57.731 test_mounts (testSanity.TestNode) ... ok 00:26:00.835 test_packages (testSanity.TestNode) ... ok 00:26:03.741 test_selinux (testSanity.TestNode) ... ok 00:26:06.666 test_services (testSanity.TestNode) ... ok 00:26:06.969 00:26:06.969 ---------------------------------------------------------------------- 00:26:06.970 XML: /home/jenkins/workspace/ovirt-node-ng_master_build-artifacts-fc22-x86_64/ovirt-node-ng/tests/nosetests.xml 00:26:06.970 ---------------------------------------------------------------------- 00:26:06.970 Ran 9 tests in 134.311s 00:26:06.970 00:26:06.970 OK 00:26:06.983 xsltproc nosetests.xslt nosetests.xml 00:26:06.987 00:26:06.987 Xunit Test Results - For: 00:26:06.987 Results for nosetests 00:26:06.987 00:26:06.987 * Tests run: 9 00:26:06.987 * Failures: 0 00:26:06.987 * Skipped: 0 00:26:06.987 * Errors: 0 00:26:06.990 PASSED

imgbase update now basically works:

[root@localhost ~]# imgbase update --format liveimg ovirt-node-ng-4.0.0-0.0 /usr/share/ovirt-node-ng/image/ovirt-node-ng-4.0.0-0.0.master.20160204.0el7.squashfs.img [INFO] Extracting image '/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.0.0-0.0.master.20160204.0el7.squashfs.img' … [INFO] Update was pulled successfully [root@localhost ~]# imgbase layout ovirt-node-ng-1.0-0.0 └╼ ovirt-node-ng-1.0-0.0+1 ovirt-node-ng-4.0.0-0.0 └╼ ovirt-node-ng-4.0.0-0.0+1

I actually had this post in my drafts for some while. Now it's more of a reminder than a full post.

We do it, Atomic does it, the systemd-cabal does it, OpenWRT also does it - Delivering the OS or parts of it's - in bigger chunks. We and OpenWrt ship images, atomic trees, and the systemd cabal is also thinking about images.

The projects got in common that due to the similar concepts of delivery, we also share a common set of problems. One problem we all share is, how the state of a system instance - how the system specific configuration - is persisted/kept, between the (image/tree) based updates.

This can seen pragramatically, like it's the defautl way with rpm: The old configuration is not overwritten. But: rpm also has postlets (%post, and others) to apply some logic to files when an rpm is updated. This mechanism is used to i.e. convert files if they need some special handling between version upgrades (db schema migration, binary file conversion, ...). So in the package based scenario, necessary changes to configuration files are done during the package update. The same thing needs to happen with images.

And so all of us, who do a "monolithic os delivery" are facing the same issue: How do we "migrate" the configuration files between image updates - or: How do we persist the state of the system between upgrades.

Besides the technical differences in the solutions to this problem, there is one aspect that is unique1 to systemd: They try to solve conceptual problems.

OpenWRT's luci might be an interestnig approach to persistence; maybe it's more about re-configuration than persistence.

The diagram illustrates how we plan to bring a writable root filesystem to Node, without any limitations. No limitations, because the fs is justa regular filesystem, nothing special about it, the special handling is below, on the block layer, provided by LVM thin volumes (thanks to dm-thin):

. . . . . . . . . . . . . . . . : : : : +--------------+--------------+ : | layer-1 (rw) | layer-2 (rw) | : +--------------+--------------+ : | base-1 (ro) | base-2 (ro) | +------------+ +--------------+--------------+-----------+ | bootloader | | lvm vg | +------------+--+-----------------------------------------+ | disk | +---------------------------------------------------------+

Each image (called base-N) we publish will be stored as a read-only (thin) logical volume in the volume group (lvm vg). For each base base-N a writable layer named layer-N will be created atop of the base. To boot into the writable layers, boot entries will be added to the bootloader, pointing to all available (writable) layers. There is no possibility to boot into the read-only bases.

State If modifying/customizing of a layer is interpreted as the state of a base, then migrating the custom configuration between layers (i.e. migrating the changes to from an old to a new layer), can be called persisting the state of a base.

Apparently there are many ways how the state of a base can be persisted. It has been discussed and will be discussed, yet it is not clear what the Königsweg is.

And it's obvious that base-N is intended to be an ancestor of base-(N+1).

Block layer Hiding the sparseness in the block layer has the big advantage that it is completely transparent to file based stuff, like permissions and (especially) SELinux.

Igor does his job, testing Node images on virtual machines and real hardware. But one drawback is that it's "big". It is not huge, but it is not directly usable out of the box.

Out of the blue - while playing with some ideas - I coupled pexpect (now updated to 3.1 in Fedora) with qemu (and a stdio serial).

What I got is a simple mechanism to control a VM. With much less assumptions then Igor has.

The drawback is obviously that only VMs can be tested, but the benefit is that it is very simple, and there no high requirements. And this is also limited to the console - which is optimal for Node.

Long story short, the snippet below takes an image, boot's it and tries to login via the serial console.

This can further be extended by using 9pfs to exchange data between the host and guest.

A minimal functional test is now also in imgbased to test some basic functionality at runtime.