#jetpants

Tumblr is a big user of MySQL, and MySQL automation at Tumblr is centered around a tool we built called Jetpants. Jetpants does an incredible job making risky operations safe and reliable, even fairly complex tasks like replacing failed master servers, or splitting a shard.

While Jetpants is an incredibly effective and valuable tool for Tumblr's day-to-day operation, it has remained very difficult to implement a meaningful testing framework. Integration testing at this level is very challenging. In this article I'll go through these challenges and how we've tackled them at Tumblr.

Requirements

Jetpants operates under the assumption you're managing MySQL daemons on a fully functional host, and that it can:

ssh to the target system

manage processes via service or systemctl commands

copy data around between systems

allocate spare servers from the asset management system, Collins

Right away this means we have some challenges with respect to infrastructure testing:

We need a Collins deployment

We need an environment with spare servers running MySQL

We need these spare servers to actually be servers, not light-weight Docker containers

Problems

For most of the life of Jetpants, these requirements were fulfilled using actual hardware in a testing pool in our datacenter. This wasn't ideal, however. Running a test which allocated more replicas, or tested shard splitting means using an extensive amount of real hardware that takes hours to reprovision. Testing changes to the Collins code meant talking to a real Collins deployment. What if we messed up?

This test strategy has all the hallmarks of manual testing. It doesn't prevent regressions. Test coverage of our featureset is spotty based on what was interesting at the time. Public contributors can't run the tests.

For a new user to pick up Jetpants and Collins, it can be very difficult to get started. Jetpants requires Collins to be configured it certain ways that aren't publicly documented. When I first built the testing environment, I had to regularly compare what I had to our actual deployment to figure out why Jetpants wasn't working correctly.

Solution

During a Tumblr hackathon earlier this year, I devoted my time to developing an isolated, automatic testing system. We have since integrated this system directly into Jetpants and are using it in our day-to-day development and testing.

Our test framework is based on the NixOS test framework, the same framework NixOS uses to verify it is safe to release a new version. These tests use QEMU to start an isolated environment of at least one VM, and NixOS configuration to build the VMs.

Our testing framework adds lots of tooling on top to let us create robust tests. By default, a test has a running Collins instance, a master database server, and one replica. Simple options allow provisioning additional spares or additional replicas on that initial master.

Below is an example test we've written for performing a dead master promotion. This is where the current master database is dead, and we replace it with one of the existing replicas.

Here you can see what a test looks like, and how easily we can express the components and phases of our tests:

import ../make-test.nix ({ helpers, ... }: { name = "shard-dead-master-promotion"; starting-slave-dbs = 2; test-phases = with helpers; [ (jetpants-phase "shutdown-master" '' Jetpants.pool("POSTS-1-INFINITY").master.stop_mysql '') (phase "jetpants-promotion" '' ( echo "YES" # Approve for promotion echo "YES" # Approve after summary output. Confirmation. ) | jetpants promotion --demote=10.50.2.10 --promote=10.50.2.11 '') (assert-shard-master "POSTS-1-INFINITY" "10.50.2.11") (assert-shard-slave "POSTS-1-INFINITY" "10.50.2.12") ]; })

Running this test first provisions the base environment, by

starting Collins

starting 3 Linux systems running MySQL

creating a master-replica relationship between one MySQL server as a master, and two MySQL servers as replicas, then loading in a default schema, and naming it the POSTS-1-INFINITY shard

Once all this preparation is done, our test phases begin.

First we shut down the current master, to simulate a dead master situation. We then run the jetpants promotion command which will replace the old master (10.50.2.10) with a new master we have selected, 10.50.2.11. jetpants promotion will prompt for confirmations, so we echo approvals to its stdin.

We continue by validating that the jetpants command did what we expected, and verifying the master and slaves.

Initial Results

Through this testing, we have already identified and fixed several race conditions and very old interface bugs. Nix's functional nature allows us to create and tear down test VMs in minutes, as it isn't a convergence-based configuration management tool. The stability of the test framework, and consistency of its results have allowed us to more aggressively change the underlying code in Jetpants while remaining confident our tools will work correctly during our day-to-day production maintenance.

Jetpants has been under continuous and vigorous development at Tumblr for many years now, and I'm excited about where the future will be taking MySQL automation at Tumblr.

me we came up with a patch for jetpants, by using the following, sneaky test case:

db = pools.first.slaves.first loop do db.collins_secondary_role = "foo" db.collins_secondary_role = "bar" end

me I’m sure the Jepsen guy would be impressed by our efforts to defeat the guarantees of our DB

    Recently we went through an exercise where we moved all of our database masters between data centers. We planned on doing this online with minimal user impact. Obviously when performing this sort of action there are a variety of considerations such as cache consistency and other pieces of shared state in stores like HBase, but the focus of this post will be primarily on MySQL.

    During this move we had a number of constraints. As mentioned above this was to be online when serving production traffic with minimal user impact. In aggregate we service hundreds of thousands of database queries per second. Additionally we needed to encrypt all data transferring between data centers. MySQL replication supports encryption, but connections to the servers themselves present several challenges. Specifically, from a performance standpoint the handshake to establish a connection across a WAN can impact latency if there is significant connection churn. Additionally, servicing read queries across a backhaul link adds latency, which is never desirable.

    We decided to tackle these issues in several ways. We were able to leverage a number of existing features of our applications and infrastructure, as well as developing new automation to fill gaps in functionality. Our configuration and applications in various runtimes, were able to support a read/write split (which may seem obvious to some, but isn’t always easy to accomplish in every scenario). We used the read/write split, along with encrypted replication, to provide a local read replica. Some runtimes can set up a persistent encrypted connection to a remote master, which serviced read requests in those cases, as the per-connection latency was amortized over a large number of queries. For runtimes which have a high churn rate, such as PHP, we used a MySQL proxy, ProxySQL, which provided persistent, encrypted connections, as well as meeting our performance requirements. We built automation to deploy proxies for numerous database pools, servicing thousands of requests per second, per pool.

    When performing the cutover, our workflow was as follows. In each data center, there was a config which pointed to a local read slave, a remote master, and a local proxy with the master (remote or local) as a backend. When moving masters between datacenters, our database automation, Jetpants (new release coming soon!), reparented all replicas, and our automation updated the proxy backend to point to the new master. This resulted in seconds of read-only state per database pool and minimal user impact.

Slides from our talk at Percona Live NYC are now available. The presentation covers the design and implementation of Jetpants, Tumblr’s open source toolkit for interacting with hundreds of MySQL database servers.

Tim Ellis of PalominoDB wrote up a great walk-through for installing and using Jetpants on Ubuntu as well as Linux Mint.

Jetpants[https://github.com/tumblr/jetpants]

Tumblr is one of the largest users of MySQL on the web. At present, our data set consists of over 60 billion relational rows, adding up to 21 terabytes of unique relational data. Managing over 200 dedicated database servers can be a bit of a handful, so naturally we engineered some creative solutions to help automate our common processes.

Today, we’re happy to announce the open source release of Jetpants, Tumblr’s in-house toolchain for managing huge MySQL database topologies. Jetpants offers a command suite for easily cloning replicas, rebalancing shards, and performing master promotions. It’s also a full Ruby library for use in developing custom billion-row migration scripts, automating database manipulations, and copying huge files quickly to multiple remote destinations.