#jidoteki

At the end of November 2019, we quietly released v1.13 of our On-Prem Meta appliance.

This release contains a few security fixes, minor bug fixes, and a major OS upgrade from TinyCore 7.x to 10.x.

Some notable changes below:

Improved token setup

One feature which we quickly implemented in the Jidoteki Admin API was the ability to configure the initial API token on first-use. Unfortunately this has some important security implications for those deploying an OVA in a hostile network (or on a public network).

To fix this, we've enabled the ability to optionally enable our "first-run" feature (disabled by default). This generates a random passphrase as an API token when the OVA is first launched. The random passphrase can only be seen from the console (ex: by an infrastructure admin), or programmatically by an app running on the OVA appliance.

This initial passphrase must then be used to reset the API token and complete the "first-run" process.

Broken GitHub integration

Our GitHub integration was broken since the day we launched it. Although it worked on our test appliance because we weren't testing it correctly. Oops. This is fixed, and the Meta OVA can now receive GitHub push webhook events and automatically pull the latest code changes.

New options in certain API calls

We're often adding new input/output parameters to API calls to provide more functionality for our customers without breaking existing functionality. A favourite of mine is the builddate, which is central to pretty much every API call. Certain API endpoints required you to parse the reply to determine the builddate, so for those endpoints we now return that value directly at the top level of the JSON response, to facilitate chaining API calls which depend on the builddate.

We've also added the ability to suppress the log output when polling for a build's status. The log output could easily reach a few hundred KB, and polling that much data every 5 seconds was not efficient. Now it can be disabled and the status output reduced to just a few KB instead.

Finally, for our customers building multiple flavours of the same appliance (ex: small, medium, large), it is now possible to only build the small OVA, or any combination of flavours through the ova_files parameter on the POST /builds endpoint. This is good for testing when you only need to test one OVA (since technically, the only difference between various flavours are the disk sizes). For production builds, the ova_files parameter can simply be omitted.

The future

Ths year we've begun working on Meta OVA v20 (v2.x) which is so far quite a significant improvement.

Our focus has always been to help our customers deploy Enterprise on-prem virtual appliances, so with that in mind, our next OVA will contain many more enterprise features which can also be integrated into customer appliances.

We've also recently launched a new set of Open Source tools for working with the Meta OVA, so make sure to read our next blog post for details on that.

Today we're releasing our On-Prem Meta OVA release v12.

This release contains a few bug fixes, and a ton of awesome new features.

Update Oct. 1, 2018: We shipped a new minor update with a ton of performance optimizations.

Extensions management

A feature added to our TODO list back in 2016, it is now possible to manage the TinyCore .tcz extensions directly through the Meta UI or API. Not only can you view all the existing extensions on disk, but it's also possible to upload new .tcz extensions and start building with them. It is not yet possible to "automatically" download extensions from the official TinyCore repos, but that will be added eventually.

OVA signing

VirtualBox and VMware provide the ability to validate an imported OVA's "signature". This feature is not well known, but it's similar to signed Android or iOS apps. Essentially, it allows the person importing the OVA to confirm who built the OVA, and to ensure it wasn't tampered with in transit. When uploading RSA certificates to the Meta OVA, they will automatically be used to sign all future OVAs.

While working on this feature, we discovered an issue with the way our .vmdk disks were created, so we fixed that as well, which led to the following two additional features:

Auto-generating VMDK disks

With our new approach to creating .vmdk disks, it is now possible to create them on-the-fly for every build. This provides our customers with more flexibility for their OVA configurations, since the disks don't need to be created in advance, by us, anymore. We allow arbitrary sizes in 1GiB increments and the value is specified in the Ansible builder config file.

Exporting to QCOW2, RAW, VHD

Since we were knee-deep in the OVA disk world, we decided to also provide the ability to export the OVA's OS disk to .qcow2, .raw, and .vhd formats.

This is a build-time option and results in a disk1..tar.gz which can be downloaded and easily deployed to various hypervisors and/or cloud providers (such as Xen, AWS, GCP, Azure).. and that leads to the last new feature of this Meta release:

Deploying an AMI on Amazon EC2

We opted not to implement an automated AWS -> AMI export feature for various reasons. Instead, "Cloud documentation" is accessible through from support section, with instructions on how to deploy the RAW OVA disk image to Amazon's EC2 service.

We plan on adding instructions for other cloud providers in our next release.

Of course, this means our On-Prem Meta appliance is also officially available on Amazon Web Services.

Performance optimizations

We weren't satisfied with the v1.12.0 release, so we worked hard to ship v1.12.1 as quickly as possible, with a ton of performance improvements.

To list some of the changes we've made:

trimmed hundreds of DB queries down to one

replaced gzip with pigz for multi-core parallel compression

disabled compression for update packages, at the expense of slightly larger files

lazy-loading images only when entering a specific help docs section

changed the way we create disk1 (NBD vs Loopback)

move certain files from memory to disk to reduce memory usage during a build

With these changes, our own Meta appliance builds are now 2x faster (down from 4 to 2 minutes).

One more thing

We've moved away from updating a wiki with our release information, and built dedicated Release pages and an RSS feed to centralize everything. We'll be rolling that out to all our customers shortly.

We recently released v11 of our On-Prem Meta OVA. This release contains a handful of security updates, some great new features, and a huge memory optimization.

Memory

Previously, one build would require nearly 3x the amount of RAM for the size of the build (i.e: a 350MB build would require 1GB). We've made some changes which allow us to lower that requirement between 33% and 50% less RAM for each new build. This means it's now possible to do 2x more concurrent builds, or provision a 2x smaller OVA.

Features

The biggest new feature is the official integration of GitHub webhooks. The Meta OVA can now directly accept a GitHub push event, and can automatically clone/pull the repository changes prior to making a build. This removes the need to manually push Git changes to the appliance, and opens the door for automatically launching builds when a webhook is received.

We've also added a few new API endpoints and parameters, with no breaking changes.

Security

The usual suspects, such as Nginx. OpenSSL, and Qemu, have received security updates.

Of course, we've incorporated many bug fixes implemented over the last few months, particularly in regards to race conditions with certain asynchronous tasks.

Five years ago, we created the name "Jidoteki" (a Japanese word) to refer to the process of automatically creating virtual appliances. During that time, our business and approach has evolved, but the premise remains the same.

We decided to rename our product and service to On-Prem, to reflect what we truly care about.

To some, it may appear that On-Prem is just a generic word with the addition of being conversationally awkward. While we agree to some extent, our intention is to cement the fact that we strive to provide the absolute best on-premises experience with the creation of our enterprise appliances.

In the years since we started, we've helped our customers build and ship thousands of appliances, which are running stable in offline, secure environments of many Fortune 500 companies. We want to thank them for working with us all this time, and we plan to continue moving forward in this direction.

We've updated a few links, so please update your bookmarks:

Website: https://on-premises.com

GitHub: https://github.com/on-prem

Twitter: https://twitter.com/on_premises

IRC #On-Prem: ircs://chat.freenode.net:6697/on-prem

Note: We're not on Facebook, LinkedIn, Instagram, AngelList, or ProductHunt.

New Trial OVA

Today, we've also launched a new trial OVA for you to evaluate:

https://try.on-premises.com

It's a modified version of a typical enterprise virtual appliance created for our customers. It contains a sample stack: MySQL, Nginx, and NodeJS, as well as our open source REST API and Admin Dashboard.

The trial OVA is designed to help you see exactly what we mean when we say it's tiny, bullet-proof, and rock solid.

We're looking forward to continue helping businesses ship enterprise virtual appliances, and we'll be happy to help and answer any questions.

Today, we finally released an update to Jidoteki Meta - v1.9.0, which contains a handful of security updates, and only one new major feature: backups.

The security updates are the typical ones you would find in any Linux system: OpenSSL, Nginx, Kernel, etc.

The backups feature makes it possible to quickly retrieve the most important Jidoteki Meta data (logs and database), and restore them on another appliance to get up and running quickly.

Of course, this release does contain a multitude of useful bug fixes and minor improvements, but nothing major like our previous releases.

During the last six months, we had our heads down pumping out feature after feature, adding new whiz-bang functionality, only to drop them a couple weeks upon completion. Why? Because focus.

One of our biggest concerns is adding features nobody will use. Since we're also daily users of Jidometa, we quickly realized that our "new features" were not actually very useful. They definitely had a cool factor, but if we aren't using those features daily, then they're not very important.

Who got the boot?

Some of the features we built include: lsyncd+rsync+ssh replication, auto-update from a remote server, user and group management, S3 object data storage, kexec fast reboots, and server-side HTML page rendering.

Yes, we've essentially been busy spinning our wheels, but the outcome is that our Jidometa OVA remains a 60Mb download, and does its job extremely well, with no overhead or confusion.

What's next?

We do plan to rehash some of those unreleased features into better and more lean implementations, but we'll wait for the right time for that (i.e: when we see a real need for it).

At the moment, we want to focus on a Jidometa appliance which makes our customers (and our own) lives simpler, while helping us work, develop, and build faster. We've listened to feedback and already have a list of important features to implement, but I promise we won't waste our time, this time around.

One of the main things to look forward to, is the ability to manage a cluster of Jidometa appliances from one central location. This would allow builds to be distributed across developer workstations, while allowing everyone to share their builds with the rest of the team. We want to provide the ability to manage the .tcz extensions uploaded to the appliance, so updating something like nodejs can be done without our intervention. Finally, we're looking at integrating SyncThing as part of every OVA (dev option enabled), to allow developers to quickly test and iterate on their app directly in an existing appliance, without going through the full build/update/reboot/test loop (note: we've been doing this for over a year inside our dev Jidometa appliance, and it's been a huge time saver).

In [part 2 of Inside Jidometa], we discussed our implementation of caching for JSON responses.

The initial goal was to solve the “slow page load” issues, but after a few months of running this in production, we eventually found ourselves battling more and more odd race conditions.

In retrospect, caching was a bad idea

I knew this from the start, but we were quite confident in our ability to implement a proper cache invalidation scheme to remove any possible race conditions.

The reality is that as our app grew, new features were added, and those features introduced more places where race conditions would surface.

Finding the root cause

I tasked myself with the goal of finding the exact root cause of our slow page loads. This required the disabling of caching, and then optimizing of SQL queries.

Our SQL backend was performing simple and multiple SQL queries per page load (~30 queries per load). This was extremely inefficient and un-necessary.

I identified every page which had such issues, and managed to reduce and combine queries down to 2 or 3 queries per page, thus significantly decreasing page load time.

Some numbers

Without caching, loading a page (or API call) with ~700 builds would previously take nearly 30 seconds.

After my optimizations, this was reduced to 300 milliseconds, or 0.3 seconds. Those changes were sufficient for us to completely remove our JSON caching implementation, and also fixed other random issues/race conditions as a side-effect.

The lesson here is: caching sucks. Don’t do it unless.. just don’t do it. It’s pretty much always a hack, and will eventually somehow someway end up biting you in the rear. Even if you’re a seasoned expert in race-condition handling and cache invalidation, other things which you didn’t even know exist will surface and cause headaches and random failures -- because of caching. The typical approach to solving caching issues is to add more caching (see Intel CPUs), or to perform more ugly hacks to work around the caching issues.

I’m a fan of using no caching, and then moving on with my happy life ;)

Moving forward

We released Jidoteki Meta v1.8 at the beginning of June, which included these changes and a few other extremely useful features.

We’re continuously working to improve Jidometa in order to provide the best, fastest, and most reliable way of consistently building virtual appliances designed to run on-premises.

We recently released v1.7.0 of Jidoteki Meta, and with it came a little unexpected surprise: it bricked a few customer appliances.

In this post, I’ll explain how not to brick a customer appliance, as well as a few techniques to recover when it happens.

Oops!

Our build process is entirely automated - duh, that’s why it’s called Jidoteki (automatically, in Japanese) - but our release process isn’t. There’s a series of roughly 40 steps on a long checklist prior to making a Jidometa release. Most tasks are automated and only take a few seconds to complete/validate, but others are manual and require a bit more time.

In our most recent release, we changed something which shouldn’t have been changed (the default /etc/fstab file), which prevented the /data disk from being mounted - but only on older deployments.

Problems compounded

To make matters worse, our fstab blunder had the cascading effect of not starting Openssh, Nginx, or the Jidoteki API services, which made remote administration impossible (thus impossible to update the appliance).

A missing validation

What we didn’t realize was that our release process didn’t include a step to validate our builds/updates against, you guessed it, older deployments! We did test the updates against slightly older appliances, but not against the oldest ones - the ones which some of our great customers were still running.

No worries though, we’ve added that to our process/checklist and can guarantee it won’t happen again.

Recovery

Luckily (and stupidly), our initial appliances shipped with a default root password (known only to us). We were able to provide instructions to login via the Linux console, start the necessary services, and then access the UI to upload an update package containing the fstab fix.

In our latest appliances, which are slightly more secure, there is no default root password anymore. In fact, logins are completely disabled, even by SSH. We provide the ability to change the admin password via the console GUI, but that only provides access to the files in /data (customer files), not root.

Recovery without password

We’ve customized the boot menu to provide just enough time to modify the boot command. Simply removing the ,/boot/rootfs entry will load the default TinyCore installation, which includes a root user with no password! omg! That’s a good thing. It means it’s still possible to fix a bricked appliance.

I know, it seems quite insecure at first glance, but the reality about a Linux virtual appliance is that, anyone with access to the host machine can get root. There’s no way to prevent that.

Other techniques include mounting the disk(s) in another appliance, or booting from a recovery ISO/CD.

In the end, there’s so many ways to obtain root access and to recover from such issues, that there’s no real point in preventing your customers from having it. Security through obscurity is a wasted effort.

Moving forward

To avoid bricking customer appliances, we’ve decoupled essential services from the boot process. They will start no matter what, and always provide remote administration capabilities.

Secondly, the design of Jidoteki appliances makes it very easy to either obtain root access (from the console, obviously not over the network), that we don’t need to worry about the consequences of a “bricked” appliance. The customer data is never touched by the updates, and they’re free to obtain access and perform a manual recovery procedure.

We’re working on an automated process which actually validates (integration test?) an appliance once it’s built, against a set of criteria (ex: does X service start, are disks mounted correctly, etc). We’ve already written the test suite and have been using it for a while on newly built appliances (not updated ones). Our last step is to integrate it to Jidometa and automatically run it against the builds/updates when they complete.

Finally, wouldn’t it be nice for an appliance to self-heal? Yes, it would, and we’re working on just that. The idea isn’t a new one (I implemented something similar in 2009 while working on a custom Linux OS), but essentially rather than overwrite the existing OS during an update, we could rename the file and have a second “recovery” boot option which boots from a working version when the primary one fails.

Contact us

As we’ve mentioned on multiple occasions, our Jidoteki virtual appliances are immutable and run entirely in memory. Customer data is stored on a secondary disk (disk2), while the OS and application is stored on the primary disk (disk1), and loaded in memory on boot.

System settings

The system settings are typically independent of the customer data. They define basic things such as network settings, admin passwords, SSH keys, and secondary storage type (local disk, NFS, etc). For that reason, those settings can’t be stored on disk2. This leaves us with no choice but to save them on the disk1.

Two existing approaches

There are two main approaches to this, as seen in various operating systems:

TinyCore Linux: The list of settings (files) are backed up to a .tgz with a specific command, and then automatically restored on boot.

Alpine Linux: All modified files are kept in memory and are committed to disk with a specific command, and then automatically restored on boot.

An even better approach

You probably saw this coming, but of course, we’ve improved on both those processes and released it as an open source tool: symlinktool.sh

I’ll begin by explaining why, and then how.

Why a new tool?

The main problem with the two existing approaches from TinyCore and Alpine are clear: a specific manual user action is required to persist the changes. If the end-user forgets to manually type the command, then the changes are lost on reboot.

The other problem with the TinyCore approach is the .tgz provides no insight into the history of changes. The file constantly gets overwritten and you have no way of rolling back to a previously good/known configuration. To their credit, the command-line tool does have a “-s” flag which essentially allows you to keep ONE (1) extra copy of the settings. You can’t roll-back to 2, 3, or even 10 configuration ago.

The other problem with the Alpine approach is any file can be modified and then overwritten on boot. In an immutable OS provided to customers, you definitely want them to edit some files, not all. The difference is subtle, but it’s essentially the difference between guaranteeing a functioning system, and not guaranteeing that they won’t accidentally overwrite /etc/fstab and brick the appliance.

How does symlinktool.sh work?

Now that we’ve cleared up the why, I’ll explain what our tool does differently.

For starters, the symlinktool.sh uses the exact same /opt/.filetool.lst as TinyCore Linux - which means it’s a drop-in replacement for the TinyCore filetool.sh. That .filetool.lst specifies the list of files and directories which should be persisted to disk. There’s also an /opt/.xfiletool.lst file which lists files to be excluded. This allows us to control exactly what settings files can/should be persisted, and those .filetool.lst/.xfiletool.lst files can’t be modified by the end user.

Secondly, the settings files are stored and versioned in a Git repository on disk1. Every time one of the system tools, or the Jidoteki Admin API makes changes to a settings file, it will run the symlinktool.sh which will commit the changes to the Git repo. Of course, if a file is edited manually, the changes will not be commited, but you can always enter the directory and type git diff to see the changes since the last commit. The end user can also manually commit changes to the Git repo.

Finally, and here’s the most magical property, once symlinktool.sh is used, as its name suggests, it generates symlinks in the original location of the file, and points them to their persistent location on disk1.

Example: /etc/shadow -> /mnt/sda1/ova-backups/etc/shadow

This means that ANY change to /etc/shadow will be automatically reflected on the persistent disk. Performing a Git commit is not necessary, as the changes will remain even after reboot.

We slightly modified the TinyCore Linux tc-restore.sh script to ensure the symlinks get restored on boot. Another benefit of our approach is it’s much faster than extracting a .tgz to memory.

What about a non-immutable OS?

A non-immutable OS, for example, Debian installed on disk, is quite different. On that system, any/every file can be modified and will automatically be persisted to disk. We consider this bad as it renders it almost impossible to guarantee working and atomic system updates the way Jidoteki does.

Our new tool, which we hope to deploy in our appliances very soon, is a perfect hybrid solution between immutable and non-immutable OS’s, and provides some nice features such as being able to see the exact change history of every “settings file”, and even manually restore old settings with just one command.

Feel free to check out the source on GitHub. It’s MIT licensed, so everyone is free to use and modify as they wish. Please submit a pull request if you can suggest some improvements.

If you’re currently deploying a SaaS or Installer-based software (ex: Java app) and want to provide an absolutely rock solid Enterprise virtual appliance to your customers. Please contact us so we can discuss your requirements.