#immutable os

As we’ve mentioned on multiple occasions, our Jidoteki virtual appliances are immutable and run entirely in memory. Customer data is stored on a secondary disk (disk2), while the OS and application is stored on the primary disk (disk1), and loaded in memory on boot.

System settings

The system settings are typically independent of the customer data. They define basic things such as network settings, admin passwords, SSH keys, and secondary storage type (local disk, NFS, etc). For that reason, those settings can’t be stored on disk2. This leaves us with no choice but to save them on the disk1.

Two existing approaches

There are two main approaches to this, as seen in various operating systems:

TinyCore Linux: The list of settings (files) are backed up to a .tgz with a specific command, and then automatically restored on boot.

Alpine Linux: All modified files are kept in memory and are committed to disk with a specific command, and then automatically restored on boot.

An even better approach

You probably saw this coming, but of course, we’ve improved on both those processes and released it as an open source tool: symlinktool.sh

I’ll begin by explaining why, and then how.

Why a new tool?

The main problem with the two existing approaches from TinyCore and Alpine are clear: a specific manual user action is required to persist the changes. If the end-user forgets to manually type the command, then the changes are lost on reboot.

The other problem with the TinyCore approach is the .tgz provides no insight into the history of changes. The file constantly gets overwritten and you have no way of rolling back to a previously good/known configuration. To their credit, the command-line tool does have a “-s” flag which essentially allows you to keep ONE (1) extra copy of the settings. You can’t roll-back to 2, 3, or even 10 configuration ago.

The other problem with the Alpine approach is any file can be modified and then overwritten on boot. In an immutable OS provided to customers, you definitely want them to edit some files, not all. The difference is subtle, but it’s essentially the difference between guaranteeing a functioning system, and not guaranteeing that they won’t accidentally overwrite /etc/fstab and brick the appliance.

How does symlinktool.sh work?

Now that we’ve cleared up the why, I’ll explain what our tool does differently.

For starters, the symlinktool.sh uses the exact same /opt/.filetool.lst as TinyCore Linux - which means it’s a drop-in replacement for the TinyCore filetool.sh. That .filetool.lst specifies the list of files and directories which should be persisted to disk. There’s also an /opt/.xfiletool.lst file which lists files to be excluded. This allows us to control exactly what settings files can/should be persisted, and those .filetool.lst/.xfiletool.lst files can’t be modified by the end user.

Secondly, the settings files are stored and versioned in a Git repository on disk1. Every time one of the system tools, or the Jidoteki Admin API makes changes to a settings file, it will run the symlinktool.sh which will commit the changes to the Git repo. Of course, if a file is edited manually, the changes will not be commited, but you can always enter the directory and type git diff to see the changes since the last commit. The end user can also manually commit changes to the Git repo.

Finally, and here’s the most magical property, once symlinktool.sh is used, as its name suggests, it generates symlinks in the original location of the file, and points them to their persistent location on disk1.

Example: /etc/shadow -> /mnt/sda1/ova-backups/etc/shadow

This means that ANY change to /etc/shadow will be automatically reflected on the persistent disk. Performing a Git commit is not necessary, as the changes will remain even after reboot.

We slightly modified the TinyCore Linux tc-restore.sh script to ensure the symlinks get restored on boot. Another benefit of our approach is it’s much faster than extracting a .tgz to memory.

What about a non-immutable OS?

A non-immutable OS, for example, Debian installed on disk, is quite different. On that system, any/every file can be modified and will automatically be persisted to disk. We consider this bad as it renders it almost impossible to guarantee working and atomic system updates the way Jidoteki does.

Our new tool, which we hope to deploy in our appliances very soon, is a perfect hybrid solution between immutable and non-immutable OS’s, and provides some nice features such as being able to see the exact change history of every “settings file”, and even manually restore old settings with just one command.

Feel free to check out the source on GitHub. It’s MIT licensed, so everyone is free to use and modify as they wish. Please submit a pull request if you can suggest some improvements.

If you’re currently deploying a SaaS or Installer-based software (ex: Java app) and want to provide an absolutely rock solid Enterprise virtual appliance to your customers. Please contact us so we can discuss your requirements.