Today marks the company’s 10 year anniversary.

It’s amazing that we’ve managed to stay in business this long, and I'm thankful to all our customers, suppliers, contractors, and interns for making it possible.

This post will also serve as the last one for Unscramble. The company name has changed to On-Prem, and an EOL has been set for the On-Prem Meta appliance and service.

The company will remain in business, but we’re shifting our focus to other things.

We’re excited for the future, and look forward to another decade of awesomeness.

Today we're announcing the official release of cloud-init deployed on our On-Prem virtual appliances.

The entire setup is fully Open Source under the permissive MIT license, and can be used/tweaked by anyone as they see fit.

Cloud-init on-premises

For those unfamiliar with cloud-init, it's a tool installed on the appliance which allows it to be configured with certain settings on boot. The settings are not pre-baked into the VM, but rather fetched from the VM's host (ex: EC2, VMware, Proxmox, MaaS, etc). They can be obtained in the form of a removable drive (ex: /dev/sr0), or from a "Metadata URL" such as http://169.254.169.254/.

Pre-configured settings are fine for defaults, but there has always been a need to provide additional settings such as NTP, network, DNS, storage, apitoken, before the VM is first booted. That's where cloud-init comes in. It allows a true zero-touch deployment of On-Prem appliances on various on-prem AND cloud providers.

Unfortunately with TinyCore Linux and an immutable OS, that "feature" has always been impossible to implement, until now.

The changes we've implemented for cloud-init to work on our appliances only add a few MB (~3.8MiB) to the size of the appliance, and require an ugrade from Python v2.7 to Python v3.

How it works

We add a system overlay into the TinyCore Linux rootfs, along with 3 new extensions (and their dependencies) which are loaded on boot:

ifupdown.tcz

cloud-init.tcz

cloud-init-deps.tcz

The /opt/bootsync.sh will run cloud-init and then try to setup networking through /opt/network.sh, which is simply calling cloud-init once more with different arguments. The reason for this is to provide backward compatibility for existing On-Prem TinyCore deployments.

Note: If there is no usage of the network.tcz, jidoteki-admin.tcz, or jidoteki-admin-api.tcz extensions, /opt/network.sh can be removed from /opt/bootsync.sh and replaced with:

/usr/local/bin/cloud-init modules --mode config

Each step in the cloud-init runs can be found in /etc/cloud/cloud.cfg.

The config modules are bootcmd which are the actual Shell scripts found in /etc/cloud/scripts.

Customizations can and should be added to custom.sh and custom-once.sh. If it's not immediately obvious, the once.sh and custom-once.sh scripts will only run once per boot. Unlike typical cloud-init installations, the once scripts will run again on reboot (because the OS is immutable).

Userdata, Metadata, Vendordata

The Shell scripts provided for this cloud-init deployment will read settings from various files on the system, including any userdata or vendordata provided to the system.

The order of precendence for reading configuration values are:

backed up files from /opt/.filetool.lst

userdata from /var/lib/cloud/instance/user-data.txt

vendordata from /var/lib/cloud/seed/nocloud-net/vendor-data

Note: Metadata is only currently used by cloud-init to configure SSH public-keys. All other metadata is ignored, but still accessible to applications running on the TinyCore Linux appliance.

Userdata

Tested with EC2, Proxmox, and NoCloud, the userdata will be discovered by cloud-init and read directly from /var/lib/cloud/instance/user-data.txt. From testing, it appears cloud-init handles Base64 decoding of userdata automatically provided by EC2.

Note: Userdata is not parsed by cloud-init, but rather read manually by various Shell scripts. This means it is not possible for userdata to overwrite values from /etc/cloud/cloud.cfg, and it is not possible to provide userdata which contains a shell script or other malicious features. Userdata should only contain key: value pairs.

Vendordata

The vendor-data is added to the /var/lib/cloud/seed/nocloud-net/ directory on the rootfs. It is also possible to add a network-config (v1 config) to that directory for pre-configuring the network.

Note: Similar to userdata, the vendor-data is not parsed by cloud-init and should only contain key: value pairs (except for network-config, which is parsed by cloud-init's net-convert tool).

More info

For more information, please see the README.md from our Open Soure repository.

Moving forward

We're hoping to add cloud-init to all our customer appliances in the coming month, so all their customers can benefit from the advantages cloud-init provides.

A few weeks ago we released an open source version of our first integration with GitHub Actions: build-ova.

This Action integrates into a GitHub Workflow in order launch an OVA Build on your On-Prem Meta virtual appliance.

Here's some screenshots of the Action in action:

Since our customer appliance settings are typically stored on GitHub, this Action makes it extremely easy to launch a new OVA build automatically on various events, such as push and pull request. It accepts the exact same input parameters as the REST API's POST /builds endpoint, so it should be fairly easy to migrate existing REST API integrations to a GitHub workflow.

At the same time, we also released our official API client which can talk directly to the Meta or Admin API and easily make API requests to those REST APIs.

The client is written in CoffeeScript (for NodeJS) and has a handful of utility functions to facilitate certain common API calls, such as buildOVA(), pollStatus(), and getDownloads(). This API client is also fully open source and can be used freely for working with the APIs listed above.

At the end of November 2019, we quietly released v1.13 of our On-Prem Meta appliance.

This release contains a few security fixes, minor bug fixes, and a major OS upgrade from TinyCore 7.x to 10.x.

Some notable changes below:

Improved token setup

One feature which we quickly implemented in the Jidoteki Admin API was the ability to configure the initial API token on first-use. Unfortunately this has some important security implications for those deploying an OVA in a hostile network (or on a public network).

To fix this, we've enabled the ability to optionally enable our "first-run" feature (disabled by default). This generates a random passphrase as an API token when the OVA is first launched. The random passphrase can only be seen from the console (ex: by an infrastructure admin), or programmatically by an app running on the OVA appliance.

This initial passphrase must then be used to reset the API token and complete the "first-run" process.

Broken GitHub integration

Our GitHub integration was broken since the day we launched it. Although it worked on our test appliance because we weren't testing it correctly. Oops. This is fixed, and the Meta OVA can now receive GitHub push webhook events and automatically pull the latest code changes.

New options in certain API calls

We're often adding new input/output parameters to API calls to provide more functionality for our customers without breaking existing functionality. A favourite of mine is the builddate, which is central to pretty much every API call. Certain API endpoints required you to parse the reply to determine the builddate, so for those endpoints we now return that value directly at the top level of the JSON response, to facilitate chaining API calls which depend on the builddate.

We've also added the ability to suppress the log output when polling for a build's status. The log output could easily reach a few hundred KB, and polling that much data every 5 seconds was not efficient. Now it can be disabled and the status output reduced to just a few KB instead.

Finally, for our customers building multiple flavours of the same appliance (ex: small, medium, large), it is now possible to only build the small OVA, or any combination of flavours through the ova_files parameter on the POST /builds endpoint. This is good for testing when you only need to test one OVA (since technically, the only difference between various flavours are the disk sizes). For production builds, the ova_files parameter can simply be omitted.

The future

Ths year we've begun working on Meta OVA v20 (v2.x) which is so far quite a significant improvement.

Our focus has always been to help our customers deploy Enterprise on-prem virtual appliances, so with that in mind, our next OVA will contain many more enterprise features which can also be integrated into customer appliances.

We've also recently launched a new set of Open Source tools for working with the Meta OVA, so make sure to read our next blog post for details on that.

Since our last pricing update 1.5 years ago, we've automated most of the manual tasks in our "Setup plan". Thanks to our awesome customers, we've managed to sand down any rough edges, and build a reliable appliance for their daily appliance builds.

Our Setup process still requires some manual work, but much less than when we first started six years ago, so most of the value we provide exists in our On-Prem Meta Appliance (Meta OVA) instead.

For these reasons, we are waiving the Setup fee on the condition that new customers purchase a minimum 1 year license for the Meta OVA.

You can find more info on our pricing here.

The Meta OVA has gone a long way in just two years. It now has the ability to run on AWS and build AWS/GCP/Azure appliances in addition to the other on-prem platforms (KVM/Xen/VMware/Hyper-V), it integrates directly with GitHub, provides powerful build management tools, and contains a ton of performance optimizations to reduce memory and CPU usage - all while remaining under 60MB.

Today we're releasing our On-Prem Meta OVA release v12.

This release contains a few bug fixes, and a ton of awesome new features.

Update Oct. 1, 2018: We shipped a new minor update with a ton of performance optimizations.

Extensions management

A feature added to our TODO list back in 2016, it is now possible to manage the TinyCore .tcz extensions directly through the Meta UI or API. Not only can you view all the existing extensions on disk, but it's also possible to upload new .tcz extensions and start building with them. It is not yet possible to "automatically" download extensions from the official TinyCore repos, but that will be added eventually.

OVA signing

VirtualBox and VMware provide the ability to validate an imported OVA's "signature". This feature is not well known, but it's similar to signed Android or iOS apps. Essentially, it allows the person importing the OVA to confirm who built the OVA, and to ensure it wasn't tampered with in transit. When uploading RSA certificates to the Meta OVA, they will automatically be used to sign all future OVAs.

While working on this feature, we discovered an issue with the way our .vmdk disks were created, so we fixed that as well, which led to the following two additional features:

Auto-generating VMDK disks

With our new approach to creating .vmdk disks, it is now possible to create them on-the-fly for every build. This provides our customers with more flexibility for their OVA configurations, since the disks don't need to be created in advance, by us, anymore. We allow arbitrary sizes in 1GiB increments and the value is specified in the Ansible builder config file.

Exporting to QCOW2, RAW, VHD

Since we were knee-deep in the OVA disk world, we decided to also provide the ability to export the OVA's OS disk to .qcow2, .raw, and .vhd formats.

This is a build-time option and results in a disk1..tar.gz which can be downloaded and easily deployed to various hypervisors and/or cloud providers (such as Xen, AWS, GCP, Azure).. and that leads to the last new feature of this Meta release:

Deploying an AMI on Amazon EC2

We opted not to implement an automated AWS -> AMI export feature for various reasons. Instead, "Cloud documentation" is accessible through from support section, with instructions on how to deploy the RAW OVA disk image to Amazon's EC2 service.

We plan on adding instructions for other cloud providers in our next release.

Of course, this means our On-Prem Meta appliance is also officially available on Amazon Web Services.

Performance optimizations

We weren't satisfied with the v1.12.0 release, so we worked hard to ship v1.12.1 as quickly as possible, with a ton of performance improvements.

To list some of the changes we've made:

trimmed hundreds of DB queries down to one

replaced gzip with pigz for multi-core parallel compression

disabled compression for update packages, at the expense of slightly larger files

lazy-loading images only when entering a specific help docs section

changed the way we create disk1 (NBD vs Loopback)

move certain files from memory to disk to reduce memory usage during a build

With these changes, our own Meta appliance builds are now 2x faster (down from 4 to 2 minutes).

One more thing

We've moved away from updating a wiki with our release information, and built dedicated Release pages and an RSS feed to centralize everything. We'll be rolling that out to all our customers shortly.

We recently released v11 of our On-Prem Meta OVA. This release contains a handful of security updates, some great new features, and a huge memory optimization.

Memory

Previously, one build would require nearly 3x the amount of RAM for the size of the build (i.e: a 350MB build would require 1GB). We've made some changes which allow us to lower that requirement between 33% and 50% less RAM for each new build. This means it's now possible to do 2x more concurrent builds, or provision a 2x smaller OVA.

Features

The biggest new feature is the official integration of GitHub webhooks. The Meta OVA can now directly accept a GitHub push event, and can automatically clone/pull the repository changes prior to making a build. This removes the need to manually push Git changes to the appliance, and opens the door for automatically launching builds when a webhook is received.

We've also added a few new API endpoints and parameters, with no breaking changes.

Security

The usual suspects, such as Nginx. OpenSSL, and Qemu, have received security updates.

Of course, we've incorporated many bug fixes implemented over the last few months, particularly in regards to race conditions with certain asynchronous tasks.

Five years ago, we created the name "Jidoteki" (a Japanese word) to refer to the process of automatically creating virtual appliances. During that time, our business and approach has evolved, but the premise remains the same.

We decided to rename our product and service to On-Prem, to reflect what we truly care about.

To some, it may appear that On-Prem is just a generic word with the addition of being conversationally awkward. While we agree to some extent, our intention is to cement the fact that we strive to provide the absolute best on-premises experience with the creation of our enterprise appliances.

In the years since we started, we've helped our customers build and ship thousands of appliances, which are running stable in offline, secure environments of many Fortune 500 companies. We want to thank them for working with us all this time, and we plan to continue moving forward in this direction.

We've updated a few links, so please update your bookmarks:

Website: https://on-premises.com

GitHub: https://github.com/on-prem

Twitter: https://twitter.com/on_premises

IRC #On-Prem: ircs://chat.freenode.net:6697/on-prem

Note: We're not on Facebook, LinkedIn, Instagram, AngelList, or ProductHunt.

New Trial OVA

Today, we've also launched a new trial OVA for you to evaluate:

https://try.on-premises.com

It's a modified version of a typical enterprise virtual appliance created for our customers. It contains a sample stack: MySQL, Nginx, and NodeJS, as well as our open source REST API and Admin Dashboard.

The trial OVA is designed to help you see exactly what we mean when we say it's tiny, bullet-proof, and rock solid.

We're looking forward to continue helping businesses ship enterprise virtual appliances, and we'll be happy to help and answer any questions.

Today, we finally released an update to Jidoteki Meta - v1.9.0, which contains a handful of security updates, and only one new major feature: backups.

The security updates are the typical ones you would find in any Linux system: OpenSSL, Nginx, Kernel, etc.

The backups feature makes it possible to quickly retrieve the most important Jidoteki Meta data (logs and database), and restore them on another appliance to get up and running quickly.

Of course, this release does contain a multitude of useful bug fixes and minor improvements, but nothing major like our previous releases.

During the last six months, we had our heads down pumping out feature after feature, adding new whiz-bang functionality, only to drop them a couple weeks upon completion. Why? Because focus.

One of our biggest concerns is adding features nobody will use. Since we're also daily users of Jidometa, we quickly realized that our "new features" were not actually very useful. They definitely had a cool factor, but if we aren't using those features daily, then they're not very important.

Who got the boot?

Some of the features we built include: lsyncd+rsync+ssh replication, auto-update from a remote server, user and group management, S3 object data storage, kexec fast reboots, and server-side HTML page rendering.

Yes, we've essentially been busy spinning our wheels, but the outcome is that our Jidometa OVA remains a 60Mb download, and does its job extremely well, with no overhead or confusion.

What's next?

We do plan to rehash some of those unreleased features into better and more lean implementations, but we'll wait for the right time for that (i.e: when we see a real need for it).

At the moment, we want to focus on a Jidometa appliance which makes our customers (and our own) lives simpler, while helping us work, develop, and build faster. We've listened to feedback and already have a list of important features to implement, but I promise we won't waste our time, this time around.

One of the main things to look forward to, is the ability to manage a cluster of Jidometa appliances from one central location. This would allow builds to be distributed across developer workstations, while allowing everyone to share their builds with the rest of the team. We want to provide the ability to manage the .tcz extensions uploaded to the appliance, so updating something like nodejs can be done without our intervention. Finally, we're looking at integrating SyncThing as part of every OVA (dev option enabled), to allow developers to quickly test and iterate on their app directly in an existing appliance, without going through the full build/update/reboot/test loop (note: we've been doing this for over a year inside our dev Jidometa appliance, and it's been a huge time saver).

In [part 2 of Inside Jidometa], we discussed our implementation of caching for JSON responses.

The initial goal was to solve the “slow page load” issues, but after a few months of running this in production, we eventually found ourselves battling more and more odd race conditions.

In retrospect, caching was a bad idea

I knew this from the start, but we were quite confident in our ability to implement a proper cache invalidation scheme to remove any possible race conditions.

The reality is that as our app grew, new features were added, and those features introduced more places where race conditions would surface.

Finding the root cause

I tasked myself with the goal of finding the exact root cause of our slow page loads. This required the disabling of caching, and then optimizing of SQL queries.

Our SQL backend was performing simple and multiple SQL queries per page load (~30 queries per load). This was extremely inefficient and un-necessary.

I identified every page which had such issues, and managed to reduce and combine queries down to 2 or 3 queries per page, thus significantly decreasing page load time.

Some numbers

Without caching, loading a page (or API call) with ~700 builds would previously take nearly 30 seconds.

After my optimizations, this was reduced to 300 milliseconds, or 0.3 seconds. Those changes were sufficient for us to completely remove our JSON caching implementation, and also fixed other random issues/race conditions as a side-effect.

The lesson here is: caching sucks. Don’t do it unless.. just don’t do it. It’s pretty much always a hack, and will eventually somehow someway end up biting you in the rear. Even if you’re a seasoned expert in race-condition handling and cache invalidation, other things which you didn’t even know exist will surface and cause headaches and random failures -- because of caching. The typical approach to solving caching issues is to add more caching (see Intel CPUs), or to perform more ugly hacks to work around the caching issues.

I’m a fan of using no caching, and then moving on with my happy life ;)

Moving forward

We released Jidoteki Meta v1.8 at the beginning of June, which included these changes and a few other extremely useful features.

We’re continuously working to improve Jidometa in order to provide the best, fastest, and most reliable way of consistently building virtual appliances designed to run on-premises.

I guess we kind of went full-circle on this one.

As you’ve guessed from the title, we’re now able to deploy Jidoteki virtual appliances into Amazon’s datacenters.

What this means is if you’re currently deploying Jidoteki appliances on-premises, you can now deploy them on Amazon’s premises (on the cloud). Of course, Jidometa also runs on AWS.

The future of Jidoteki

We’re still focusing all our energy on true rock-solid enterprise on-prem virtual appliances. Jidoteki’s goal remains the same, and we have no intention of “switching” to cloud. We still strongly believe SaaS providers should make their apps available on-premises, and we plan to continue pushing in that direction.

No vendor lock-in

The great advantage of using Jidoteki is that it’s the exact same solution whether it’s deployed on VMware, Hyper-V, VirtualBox, XenServer, or AWS. We don’t lock you or your customers into one platform or unstable technology. On the contrary, customers can now migrate from their on-prem datacenter and Amazon, or vice-versa.

We recently released v1.7.0 of Jidoteki Meta, and with it came a little unexpected surprise: it bricked a few customer appliances.

In this post, I’ll explain how not to brick a customer appliance, as well as a few techniques to recover when it happens.

Oops!

Our build process is entirely automated - duh, that’s why it’s called Jidoteki (automatically, in Japanese) - but our release process isn’t. There’s a series of roughly 40 steps on a long checklist prior to making a Jidometa release. Most tasks are automated and only take a few seconds to complete/validate, but others are manual and require a bit more time.

In our most recent release, we changed something which shouldn’t have been changed (the default /etc/fstab file), which prevented the /data disk from being mounted - but only on older deployments.

Problems compounded

To make matters worse, our fstab blunder had the cascading effect of not starting Openssh, Nginx, or the Jidoteki API services, which made remote administration impossible (thus impossible to update the appliance).

A missing validation

What we didn’t realize was that our release process didn’t include a step to validate our builds/updates against, you guessed it, older deployments! We did test the updates against slightly older appliances, but not against the oldest ones - the ones which some of our great customers were still running.

No worries though, we’ve added that to our process/checklist and can guarantee it won’t happen again.

Recovery

Luckily (and stupidly), our initial appliances shipped with a default root password (known only to us). We were able to provide instructions to login via the Linux console, start the necessary services, and then access the UI to upload an update package containing the fstab fix.

In our latest appliances, which are slightly more secure, there is no default root password anymore. In fact, logins are completely disabled, even by SSH. We provide the ability to change the admin password via the console GUI, but that only provides access to the files in /data (customer files), not root.

Recovery without password

We’ve customized the boot menu to provide just enough time to modify the boot command. Simply removing the ,/boot/rootfs entry will load the default TinyCore installation, which includes a root user with no password! omg! That’s a good thing. It means it’s still possible to fix a bricked appliance.

I know, it seems quite insecure at first glance, but the reality about a Linux virtual appliance is that, anyone with access to the host machine can get root. There’s no way to prevent that.

Other techniques include mounting the disk(s) in another appliance, or booting from a recovery ISO/CD.

In the end, there’s so many ways to obtain root access and to recover from such issues, that there’s no real point in preventing your customers from having it. Security through obscurity is a wasted effort.

Moving forward

To avoid bricking customer appliances, we’ve decoupled essential services from the boot process. They will start no matter what, and always provide remote administration capabilities.

Secondly, the design of Jidoteki appliances makes it very easy to either obtain root access (from the console, obviously not over the network), that we don’t need to worry about the consequences of a “bricked” appliance. The customer data is never touched by the updates, and they’re free to obtain access and perform a manual recovery procedure.

We’re working on an automated process which actually validates (integration test?) an appliance once it’s built, against a set of criteria (ex: does X service start, are disks mounted correctly, etc). We’ve already written the test suite and have been using it for a while on newly built appliances (not updated ones). Our last step is to integrate it to Jidometa and automatically run it against the builds/updates when they complete.

Finally, wouldn’t it be nice for an appliance to self-heal? Yes, it would, and we’re working on just that. The idea isn’t a new one (I implemented something similar in 2009 while working on a custom Linux OS), but essentially rather than overwrite the existing OS during an update, we could rename the file and have a second “recovery” boot option which boots from a working version when the primary one fails.

Contact us

As we’ve mentioned on multiple occasions, our Jidoteki virtual appliances are immutable and run entirely in memory. Customer data is stored on a secondary disk (disk2), while the OS and application is stored on the primary disk (disk1), and loaded in memory on boot.

System settings

The system settings are typically independent of the customer data. They define basic things such as network settings, admin passwords, SSH keys, and secondary storage type (local disk, NFS, etc). For that reason, those settings can’t be stored on disk2. This leaves us with no choice but to save them on the disk1.

Two existing approaches

There are two main approaches to this, as seen in various operating systems:

TinyCore Linux: The list of settings (files) are backed up to a .tgz with a specific command, and then automatically restored on boot.

Alpine Linux: All modified files are kept in memory and are committed to disk with a specific command, and then automatically restored on boot.

An even better approach

You probably saw this coming, but of course, we’ve improved on both those processes and released it as an open source tool: symlinktool.sh

I’ll begin by explaining why, and then how.

Why a new tool?

The main problem with the two existing approaches from TinyCore and Alpine are clear: a specific manual user action is required to persist the changes. If the end-user forgets to manually type the command, then the changes are lost on reboot.

The other problem with the TinyCore approach is the .tgz provides no insight into the history of changes. The file constantly gets overwritten and you have no way of rolling back to a previously good/known configuration. To their credit, the command-line tool does have a “-s” flag which essentially allows you to keep ONE (1) extra copy of the settings. You can’t roll-back to 2, 3, or even 10 configuration ago.

The other problem with the Alpine approach is any file can be modified and then overwritten on boot. In an immutable OS provided to customers, you definitely want them to edit some files, not all. The difference is subtle, but it’s essentially the difference between guaranteeing a functioning system, and not guaranteeing that they won’t accidentally overwrite /etc/fstab and brick the appliance.

How does symlinktool.sh work?

Now that we’ve cleared up the why, I’ll explain what our tool does differently.

For starters, the symlinktool.sh uses the exact same /opt/.filetool.lst as TinyCore Linux - which means it’s a drop-in replacement for the TinyCore filetool.sh. That .filetool.lst specifies the list of files and directories which should be persisted to disk. There’s also an /opt/.xfiletool.lst file which lists files to be excluded. This allows us to control exactly what settings files can/should be persisted, and those .filetool.lst/.xfiletool.lst files can’t be modified by the end user.

Secondly, the settings files are stored and versioned in a Git repository on disk1. Every time one of the system tools, or the Jidoteki Admin API makes changes to a settings file, it will run the symlinktool.sh which will commit the changes to the Git repo. Of course, if a file is edited manually, the changes will not be commited, but you can always enter the directory and type git diff to see the changes since the last commit. The end user can also manually commit changes to the Git repo.

Finally, and here’s the most magical property, once symlinktool.sh is used, as its name suggests, it generates symlinks in the original location of the file, and points them to their persistent location on disk1.

Example: /etc/shadow -> /mnt/sda1/ova-backups/etc/shadow

This means that ANY change to /etc/shadow will be automatically reflected on the persistent disk. Performing a Git commit is not necessary, as the changes will remain even after reboot.

We slightly modified the TinyCore Linux tc-restore.sh script to ensure the symlinks get restored on boot. Another benefit of our approach is it’s much faster than extracting a .tgz to memory.

What about a non-immutable OS?

A non-immutable OS, for example, Debian installed on disk, is quite different. On that system, any/every file can be modified and will automatically be persisted to disk. We consider this bad as it renders it almost impossible to guarantee working and atomic system updates the way Jidoteki does.

Our new tool, which we hope to deploy in our appliances very soon, is a perfect hybrid solution between immutable and non-immutable OS’s, and provides some nice features such as being able to see the exact change history of every “settings file”, and even manually restore old settings with just one command.

Feel free to check out the source on GitHub. It’s MIT licensed, so everyone is free to use and modify as they wish. Please submit a pull request if you can suggest some improvements.

If you’re currently deploying a SaaS or Installer-based software (ex: Java app) and want to provide an absolutely rock solid Enterprise virtual appliance to your customers. Please contact us so we can discuss your requirements.

We’ve recently updated our pricing to make it even easier to get started with Jidoteki.

Having celebrated 5 years of being in business a few days ago, we decided to make our service more accessible to companies, by providing a new pricing structure.

Previous Setup plan

Previously, we offered only one “Setup” plan to get started with Jidoteki. It included lots of custom work, consulting, and onboarding in order to get a custom appliance up and running. It’s great for companies with lots of custom requirements for their software to work on-premises.

One recent example involved us converting a Java-based installer application to an on-prem virtual appliance with custom tools and scripts.

Splitting the Setup plans

We’ve renamed the original Setup plan to “Pro Setup” plan, which includes a fully customized virtual appliance covering all your needs for the perfect on-prem appliance. It includes customizations at every level to ensure you can easily transition from Installer/SaaS -> to On-Premises.

Today, we’re also announcing the “Business Setup” plan, which includes a slightly more generic virtual appliance. We still provide the same open source tools, scripts, and features of our rock-solid enterprise virtual appliance, along with its great management features. The only difference is we decreased the amount of custom work to finalize the OVA. Of course, we’ll still customize a few things to ensure your app actually works as intended.

Benefits of the Business Setup

The main benefit of the new setup plan is that we can have the appliance ready in just a few hours. Typically with lots of customization, the time to completion can take between one week and up to one month, or more.

... ready in just a few hours

To be eligible for our Business Setup plan, we require your stack to conform to the typical software stacks such as LAMP, LEMP, MEAN, etc. With those types of stacks, we can get you up and running real quickly, and you can start providing your on-prem OVA to enterprise customers with very little delay.

Another great advantage is that it’s always possible to upgrade to the “Pro Setup” in the future, if the need arises.

Moreover, Jidoteki Meta will work just as well with either Setup plan, so you’re still free to maintain full control of your builds, on your own premises.

Get your feet wet

I want to extend an invitation to companies to try our service. We can pretty much guarantee you’ll love the end product and will want to continue working with us in the long-term. It’s a win-win situation, as you’ll also gain the ability to serve enterprise customers (which translates to $$$).

[Read part 3 of Inside Jidometa]

One of the major differences between Jidoteki appliances, and most other setups, is that we isolate customer data and ensure it’s not tightly coupled to the system.

Looking at the graph above, we can see the customer data is stored on disk 2, while the OS/apps/kernel are stored on disk 1.

Issues with tight coupling

Adding disks to a virtual appliance is quite trivial. Moving them to another appliance is just as simple. We know this, so we made a plan to support it.

In a typical non-Jidoteki appliance, databases, log files, application settings and data are stored on the same disk. This tight coupling introduces problems which are almost impossible to workaround once deployed:

Failed OS updates could accidentally wipe all customer data

It’s impossible to expand the disk once it’s full

It’s difficult to move customer data to a newly deployed appliance

Customers may not have full control of their data

The disk must be sufficiently large from the start

Our disk setup

Jidoteki appliances use a minimum of two disks. The first disk is quite small (8GB sparse disk) and used exclusively for storing the system and custom settings (ex: network settings). The small OS disk means importing an OVA for the first time is very quick. The second disk is sized based on our customers’ requirements, but typically range between 10GB and 2TB (sparse disk), depending on the application. In any case, the customer can opt to not import the second disk, or simply delete it and re-create it with whatever size they want.

The best part about disk 2 is we deploy it using LVM, which allows anyone to attach a third, fourth.. eighth disk (max 8 disks) of virtually any size. If a deployed appliance uses 2TB and needs to increase to 42TB, an additional disk can be added and the appliance will automatically grow the LVM disk.

Other benefits to loose coupling

By dedicating the entire disk to the customer data, we can provide full permissions to the appliance admin (the customer) to login via SSH and modify any file(s) on that disk. The customer controls their data and remains free to alter it how they want (at their own risk, of course). We do this in our own on-premises virtual appliance, Jidoteki Meta (Jidometa), and guide our customers in doing it this way as well.

Another benefit is having the ability to detach the disk from the appliance, deploy a new appliance on a different (physical?) server, and then re-attach the disk. It’s not as fancy as live migration, but moving disks is effortless compared to transferring data with rsync or some other wild migration scheme.

Of course, if it’s that easy to move data, that means it’s just as easy to backup the data, or create a fault-tolerant setup for disaster recovery scenarios.

Finally, as mentioned in a previous blog post, dedicating the second disk to customer data means it can also be replaced by network storage, such as NFS, AoE, iSCSI, or NBD. I can’t imagine the logistics (and coding nightmare) of having to support network storage when the application was designed for storing on the same disk. With Jidoteki, we simply mount the network storage to /data, instead of mounting the local disk 2 to /data. No software changes required. Easy.

Moving forward

As always, we’re happy to discuss your requirements if you plan on providing an on-prem appliance to enterprise customers. We’re familiar with enterprise requirements, and try our best to provide the most rock-solid on-premises virtual appliances out there. In a few days we’ll be celebrating 5 years of doing this stuff. It really is our specialty, and we’d love to help more companies go on-prem, the right way.

[Read part 2 of Inside Jidometa]

In this post, we’ll highlight some of the open source tools we’ve created and deploy in every Jidoteki appliance, including Jidometa itself.

The OS

We'll start from the buttom up. Jidometa is built on top of TinyCore Linux - a small footprint in-memory operating system built on GNU/Linux. We use the unmodified OS with a few minor changes.

The sources for the OS toolchain

The sources for the OS busybox

The TinyCore OS initramfs and kernel

The TinyCore OS scripts modifications

The kernel

TinyCore Linux ships with a slightly modified Linux kernel. In our tests, we were able to deploy our appliances with a completely unmodified (vanilla) Linux kernel, so we provide the full unmodified kernel sources.

The kernel build scripts

The sources for the Linux kernel

The extensions

Extensions are similar to Debian .deb packages and RedHat .rpm packages. In Jidometa, they are squashfs .tcz files which contain pre-compiled and stripped binaries. They're typically much smaller because we don't include man pages, headers, and other files not needed in an immutable OS.

The sources for the extensions

We have yet to publish all the build scripts for our extensions, but they all use basic commands:

./configure;make;make install

Anyone can easily rebuild them with the original sources.

The admin scripts

We always include our own Open Source scripts to help manage the appliance. The scripts vary in importance and are either written in POSIX shell, or PicoLisp. We're slowly working on replacing all our POSIX shell scripts with PicoLisp shell scripts.

The Jidoteki Admin scripts, which manage the appliance from the console

The Jidoteki Admin API scripts, which manage the appliance from a REST API

The helper scripts, which provide additional functionality to the appliance scripts

We have quite a few more administration scripts, but we haven't open sourced them yet.

The libraries

The Admin scripts are built on top of a set of open source PicoLisp libraries which provide the foundation for stable, tested, and functional tools. The libraries include:

A unit testing library, to help write simple unit tests for every PicoLisp script and library, and ensure correct functionality (as well as reduce bugs/regressions)

A JSON library, to natively parse and generate JSON documents directly in PicoLisp

A SemVer library, to help manage, compare, and validate appliance and update package versions

An API library, to help build simple REST APIs as quickly and easily as possible

The licenses

It's almost impossible to gather every single license for every single software used in the appliance. We provided the software source packages intact, which include all the unmodified license files as well.

Additional licenses can be found directly in the appliance in:

/usr/share/doc/License/

The ISOs

Since downloading all individual source packages can be quite troublesome at times, we also provide direct links to download ISOs which bundle all the sources together.

Fauxpen source

We're not running a fauxpen source operation here. Admittedly not all our code is Open Source, but we do our best to comply with the GNU and OSD licenses by making it easy for everyone to access the sources files for the appliances and binaries we distribute.

We're constantly releasing new open source tools and libraries, so make sure you sign-up for our mailing list to stay up-to-date on our work, tools, and solutions.

Even big companies like GitHub have difficulty implementing a correct “update” feature (2 years work in progress).

I believe I have never seen a proper implementation of a software’s online and auto-update / upgrade feature. They all have one or more of the following problems:

It’s enabled by default (ex: SyncThing), so an unsuspecting user can suddenly have their software changed right under their nose, without their knowledge.

The update settings and “update”, “check now”, or “check for updates” buttons are deeply hidden in the software. Sometimes they’re in the “About” menu, other times in “Settings”, or more frustratingly in the “Preferences->Advanced->Update” menu (ex: Firefox).

Automatic updating/checking can’t always be disabled, or if it can, it’s not immediately obvious how to do it (ex: Chrome).

The update process secretly sends telemetry, analytics, and who knows what other data to the update server (ex: all of the above).

The update process is extremely slow and often must not be interrupted.

The end-user has no idea what’s happening behind the scenes during the update.

The update can’t be done manually, offline.

Automatic updates occur while the end-user is using the application. It interrupts their workflow and is poorly timed.

The reason for this

I’ve come to believe the primary reason for this is updating the software is an afterthought. Naturally, it would contain one or more of those flaws.

A first class citizen

What if, for a moment, we all decided to identify our software update process as a first class citizen - an integral part of our application: to be clear, well documented, simple, private, possibly offline, interruptible, and fail-proof?

With Jidoteki, that has been part of our plan from the start. We’ve successfully done that with our offline update process, and today we’re presenting our approach for online updates as well.

Our approach

Looking at the screenshot above, we see three things:

The “Update” button is not hidden, but is part of the main menu, and leads to a dedicated Update section.

Update checks are only performed when the “Check now” button is clicked. Checks are not enabled by default and don’t occur automatically when entering the Update section.

The Update section displays the current software version, and a summarized log of update activity.

This allows the end-user to control if and when they want to check for updates. There’s no bouncing number or nags which notify them of pending updates.

In the above screenshot, we note the following:

An “Update now” button appears, which is different from “Check now”. Clicking “Check now” should not perform the update. That should happen in another step once the end-user confirmed they want to apply the available updates.

The list of available updates is displayed, with downloadable links to the individual updates, the file size, the hash of the file, version, and release notes / changelog for the end-user to read/download. We use signed, encrypted, binary deltas for updates, to save on network bandwidth and storage, and to ensure the security and integrity of each file.

The end-user can decide on their own if they want to use the online update process, or if they prefer to manually download the updates and apply them at their own convenience. It also allows the end-user to review the changes before they are applied. 

Clicking the “Update now” button does not send any private data to the update server. It should be a simple HTTPS GET request to fetch a list of available updates. It is acceptable to modify the URL based on the version/platform/architecture of the software (ex: /linux/x86_64/6.0/).

In the screenshot above, we highlight:

The update status is displayed with a real percentage of the current and pending work.

The “Stop now” button provides the ability to interrupt the process at any time, without leaving the system or software in an unknown state.

Once again, the log is updated with details regarding the update process.

Before and after

This software update process is sane, and represents a real implementation which gives privacy and control back to the end-user. Compared with existing software’s update process, this requires merely a few changes to the UI and functionality, and could greatly improve the experience of keeping software up-to-date.

We strongly encourage people to turn off auto-updating by default. I understand the rationale of thinking “if it’s off by default, nobody will ever update”. Well, if your update process can correctly handle outdated versions, then that’s not actually a problem. It also becomes much easier for end-users to perform updates when the “Update” button is prominently displayed in the UI.

Soon to be open sourced

We couldn’t tell you all of this without making the source code available as well. For the moment we’ve only implemented this update feature in Jidometa, but we’ll soon be adding it to the open source Jidoteki Admin API & Dashboard. Stay tuned for that.