MLTSHP BL(O)G!

CW: nerdy, technical details.

Originally, MLTSHP (well, MLKSHK back then) was developed for Python 2. That was fine for 2010, but 15 years later, and Python 2 is now pretty ancient and unsupported. January 1st, 2020 was the official sunset for Python 2, and 5 years later, we’re still running things with it. It’s served us well, but we have to transition to Python 3.

Well, I bit the bullet and started working on that in earnest in 2023. The end of that work resulted in a working version of MLTSHP on Python 3. So, just ship it, right? Well, the upgrade process basically required upgrading all Python dependencies as well. And some (flyingcow, torndb, in particular) were never really official, public packages, so those had to be adopted into MLTSHP and upgraded as well. With all those changes, it required some special handling. Namely, setting up an additional web server that could be tested against the production database (unit tests can only go so far).

Here’s what that change comprised: 148 files changed, 1923 insertions, 1725 deletions. Most of those changes were part of the first commit for this branch, made on July 9, 2023 (118 files changed).

But by the end of that July, I took a break from this task - I could tell it wasn’t something I could tackle in my spare time at that time.

Time passes…

Fast forward to late 2024, and I take some time to revisit the Python 3 release work. Making a production web server for the new Python 3 instance was another big update, since I wanted the Docker container OS to be on the latest LTS edition of Ubuntu. For 2023, that was 20.04, but in 2025, it’s 24.04. I also wanted others to be able to test the server, which means the CDN layer would have to be updated to direct traffic to the test server (without affecting general traffic); I went with a client-side cookie that could target the Python 3 canary instance.

In addition to these upgrades, there were others to consider — MySQL, for one. We’ve been running MySQL 5, but version 9 is out. We settled on version 8 for now, but could also upgrade to 8.4… 8.0 is just the version you get for Ubuntu 24.04. RabbitMQ was another server component that was getting behind (3.5.7), so upgrading it to 3.12.1 (latest version for Ubuntu 24.04) seemed proper.

One more thing - our datacenter. We’ve been using Linode’s Fremont region since 2017. It’s been fine, but there are some emerging Linode features that I’ve been wanting. VPC support, for one. And object storage (basically the same as Amazon’s S3, but local, so no egress cost to-from Linode servers). Both were unavailable to Fremont, so I decided to go with their Chicago region for the upgrade.

Now we’re talking… this is now not just a “push a button” release, but a full-fleged, build everything up and tear everything down kind of release that might actually have some downtime (while trying to keep it short)!

I built a release plan document and worked through it. The key to the smooth upgrade I want was to make the cutover as seamless as possible. Picture it: once everything is set up for the new service in Chicago - new database host, new web servers and all, what do we need to do to make the switch almost instant? It’s Fastly, our CDN service.

All traffic to our service runs through Fastly. A request to the site comes in, Fastly routes it to the appropriate host, which in turns speaks to the appropriate database. So, to transition from one datacenter to the other, we need to basically change the hosts Fastly speaks to. Those hosts will already be set to talk to the new database. But that’s a key wrinkle - the new database…

The new database needs the data from the old database. And to make for a seamless transition, it needs to be up to the second in step with the old database. To do that, we have take a copy of the production data and get it up and running on the new database. Then, we need to have some process that will copy any new data to it since the last sync. This sounded a lot like replication to me, but the more I looked at doing it that way, I wasn’t confident I could set that up without bringing the production server down. That’s because any replica needs to start in a synchronized state. You can’t really achieve that with a live database. So, instead, I created my own sync process that would copy new data on a periodic basis as it came in.

Beyond this, we need a proper replication going in the new datacenter. In case the database server goes away unexpectedly, a replica of it allows for faster recovery and some peace of mind. Logical backups can be made from the replica and stored in Linode’s object storage if something really disastrous happens (like tables getting deleted by some intruder or a bad data migration).

I wanted better monitoring, too. We’ve been using Linode’s Longview service and that’s okay and free, but it doesn’t act on anything that might be going wrong. I decided to license M/Monit for this. M/Monit is so lightweight and nice, along with Monit running on each server to keep track of each service needed to operate stuff. Monit can be given instructions on how to self-heal certain things, but also provides alerts if something needs manual attention.

And finally, Linode’s Chicago region supports a proper VPC setup, which allows for all the connectivity between our servers to be totally private to their own subnet. It also means that I was able to set up an additional small Linode instance to serve as a bastion host - a server that can be used for a secure connection to reach the other servers on the private subnet. This is a lot more secure than before… we’ve never had a breach (at least, not to my knowledge), and this makes that even less likely going forward. Remote access via SSH is now unavailable without using the bastion server, so we don’t have to expose our servers to potential future ssh vulnerabilities.

So, to summarize: the MLTSHP Python 3 upgrade grew from a code release to a full stack upgrade, involving touching just about every layer of the backend of MLTSHP.

Here’s a before / after picture of some of the bigger software updates applied (apologies for using images for these tables, but Tumblr doesn’t do tables):

And a summary of infrastructure updates:

I’m pretty happy with how this has turned out. And I learned a lot. I’m a full-stack developer, so I’m familiar with a lot of devops concepts, but actually doing that role is newish to me. I got to learn how to set up a proper secure subnet for our set of hosts, making them more secure than before. I learned more about Fastly configuration, about WireGuard, about MySQL replication, and about deploying a large update to a live site with little to no downtime. A lot of that is due to meticulous release planning and careful execution. The secret for that is to think through each and every step - no matter how small. Document it, and consider the side effects of each. And with each step that could affect the public service, consider the rollback process, just in case it’s needed.

👋 Hi, I’m Brad Choate. Here’s a recollection of my involvement with MLKSHK and MLTSHP over the years. The following is pieced together from bits of email, Twitter, TypePad, Tumblr, database records, and aging neurons.

A timeline of mlkshk.com

Domain registration - 4/13/2008

Initial commit from Andre - 3/13/2010

First commit from Ivan - 10/1/2010

AWS database created - 10/20/2010

First posted file (“farrrrrt bird”) - 10/27/2010

First users created (82) - 10/28/2010

Opens to limited audience - 10/28/2010

First TypePad blog post - 12/29/2010

Opens to wider audience via invites, waitlist - ~1/4/2011

First comment - 1/11/2011

Opens to general public - ~4/26/2011

First commit from Mark - 5/19/2011

Shutdown announcement - 5/1/2014

Andre is hired at Slack - ?

Shutdown averted? - 6/18/2014

Shutdown averted announcement - 6/30/2014

First commit from Brad - 8/3/2014

Good Web Bundle promotion - 11/19/2014

Subscription woes - 5/20/2015

More subscription woes - 7/9/2015

Subscriptions restored (delaying another imminent shutdown) - 3/24/2016

Final shutdown announcement - 2/22/2017

Enters read-only mode - 3/31/2017

Waxy.org post about MLKSHK’s closure - 4/13/2017

Closes - 5/1/2017

Lifespan from 10/27/2010 - 2,379 days, or 6 years, 6 months, 5 days.

Lifespan extended 3 years from 5/1/2014 shutdown announcement.

A timeline of mltshp.com

Continuation community assembles itself… on Facebook?! - 11/3/2016

mltshp.com domain registration date - 3/9/2017

Initial commit to Github by Brad - 3/17/2017

Fundraising - 3/24/2017

Gettin things done - 3/31/2017

MLKSHK user login support - 4/30/2017

GIF to Video feature - 5/9/2017

First Tumblr blog post and launch - 5/16/2017

Dropped Google Analytics - 5/10/2019

2019 State of the SHP - 8/29/2019

Site search introduced - 9/3/2019

4th birthday - 5/16/2021

Fastly CDN switch(back) - 12/22/2022

6th birthday - 5/16/2023

Happy 7th Birthday - 5/16/2024

Lifespan to date (as of this writing on 12/24/2024): 2,780 days, or 7 years, 7 months, 9 days.

Some personal history

I joined MLKSHK by invitation on January 6, 2011 as a regular user. I found MLKSHK to be a breath of fresh air compared to your typical social media site (which still holds true today). After the shutdown announcement in 2014, I tried to offer help and reached out again around July 2014 after the announcement that the site would not be closing. I had met Andre once before, but mostly knew of him through colleagues at Six Apart. Andre took me up on the volunteer help… remember, by this time he wasn’t running MLKSHK as a business, so the ~3 years that follow are maintaining the site as a hobby, particularly since it was still losing money.

Initially, I helped with site performance issues, including some query optimizations. As time passed, it became clear that the site was still in danger of closing due to the cost of services it was incurring, which was not offset enough by subscriptions. So Andre and I went through a cost assessment process and we identified a number of things that could be done. Switching away from Fastly to a cheaper CDN (KeyCDN at the time) shaved CDN operational costs by 66% (Fastly charges 12 cents per gigabyte of transfer and KeyCDN charges 4 cents) so we started there, since it was the easiest change to make and had the most impact.

I also helped with the site’s participation in the “Good Web Bundle” promotion which brought in a few subscribers and returning members. The following year (2015), we had to rebuild subscriptions from scratch since both Tugboat and Amazon payments had to be retired, and we replaced those with Stripe. Alas, the site was still just not able to sustain itself as it was, and a decision was made to close for good in 2017.

Before that happened, the MLKSHK community began a conversation with Andre to transition the site to a community-run operation. I wasn’t involved in discussions, so don’t have insight into the choice behind creating a new domain and brand instead of adopting MLKSHK outright (I suspect Andre just preferred a clean break). Andre agreed to share all data and source code to the transition team. I helped with transitioning the source code for MLKSHK; cleaning it up for the basis of the MLTSHP repository to become an open source project.

It was decided that it would be best to make MLTSHP more closed in nature… so much of MLKSHK was open to free users, even some allowance for post creation. The plan was to relaunch with a membership requirement, but also with two price plans - the regular $24/year membership (plus an option to specify any amount over $24/year if the member wants to), and a new $3/year membership which effectively was the same level of service that was provided for free registered users before. While posting was being restricted to members, we wanted to keep certain things open for all: our “Popular” page, and individual post links which can be shared anywhere.

There would be some downtime between MLKSHK’s closure and the launch of MLTSHP. We aimed to keep that downtime low, but it gave us some time to make some larger changes that would make for a more sustainable and cost-efficient service.

Switching off of Amazon AWS for web server and database service was a big one. I had wanted to do this for MLKSHK as early as 2014, but we never did. I took a snapshot of AWS expenses from January 2017 and costs for that month were $400 (just RDS + S3 + EC2 and outbound data transfer to our CDN). We switched these to use Linode. For CDN service, KeyCDN was more cost effective than Fastly was, but we decided to switch to using Cloudflare for CDN since it wouldn’t cost anything (ironically, MLTSHP has since switched back to Fastly after receiving an offer to operate at no cost). The only remaining AWS feature MLTSHP is using is S3 for block storage. There are some cheaper S3-compatible services available, but the potential savings have never justified the task of a migration. Operational costs today are less than half of what we were spending for AWS alone, and we pay nothing for CDN service.

We also set up a real deployment process, utilizing Buildkite (they offer a free account for us to use), making it easier to test and deploy community-led changes. This also included utilizing Docker which also makes it easier for contributors to run their own copy of the site locally for development.

Ahead of MLTSHP’s launch, a service was built that would transcode GIF images (a popular, but heavy file format) to video formats that those would serve instead of the original GIF file. This was a cost-saving measure, since popular GIFs lead to a lot of traffic, so the smaller the better.

Another must-have for relaunch was a process that would migrate a user’s MLKSHK data when they signed into MLTSHP. This required us to load all the MLKSHK data into the MLTSHP database (in a separate set of tables) and we also had permission from Andre to make a wholesale copy of the MLKSHK S3 bucket to MLTSHP’s S3 bucket so all images were preserved.

And we also had a full rebranding to do. Replacing all site design and assets so that it was different, but retaining the spirit of the original site.

There have been a lot of changes to the site since that time. We’ve added site-wide search, responsive design so the site works well for mobile devices, support for light and dark mode, an alt text field so images can be described, and many other things. Big feature releases are few and far between, but the site is stable which is a great feature.

It's been quite some time since our last update, but back in the SHP, we haven't gone anywhere. :)

Here's a short list of everything that happened over the past year:

MLTSHP turned 7 this year!

We're now using Fastly.

We now support alt text.

Our unique pink color now has a name: mltpnk.

We're now blocking AI scrapers. (as best as we can)

You can now add your MLTSHP profile in MLTSHP Slack.

Our "li'l dudes" now have canonical names.

Thanks to our members for making MLTSHP one of the best places on the Internet! Please follow SHKN BUT NOT BROKEN for more frequent updates. :)

Just updated a few little things today:

We're caching some of our public pages a little better (makes the site faster for members) #310

We're now supporting higher resolution userpic and shake images. You'll need to upload new ones if you want to take advantage (200x200 resolution is supported now) #308

We improved the "Save This" icon. #307

We fixed the comment field for Firefox users. #304

Release Notes for MLTSHP 1.0 - May 1516, 2017

We've been working hard to make a stronger, faster, better MLKSHK and we're calling it MLTSHP. We owe a great debt to Andre and Amber for donating the source code and data from MLKSHK so that the community they built could live on in this new incarnation.

With the help of our fantastic volunteers, we've taken some steps to make MLKSHK better than ever. More importantly, MLTSHP will be 100% self-funded from day one, thanks to many generous community patrons who helped with our re-launch fundraiser, and our subscribing members. Here's a rundown of what has changed since you were last using MLKSHK (you can see the ugly details at Github, if you really want to. We closed out 69 issues for this release!):

We have release notes! You're reading them.

We have a Slack team and you can join it.

We have a new Twitter account for you to follow.

Our source code is public.

We have updated our terms of service. #

We wrote a code of conduct. #174

We require membership for using the site. This means that a login is necessary as well as a subscription. A basic subscription is just $3/year. "Pro" memberships are the same price: $24/year. #7

We made a new first-time login that will recognize your MLKSHK credentials and give you the option to restore your MLKSHK data automagically. #11

We have a new site logo and brand. #2

We've polished the design of MLTSHP to a fine sheen. It's high resolution and looks amazing on mobile devices. #36

User and post "view" statistics have been fixed. #51

We have new browser plugins. Be sure to install them since the MLKSHK ones will not work with the new site. #6

We now transcode GIF files into video formats which are smaller in size and less taxing on your devices to play. #34

Post descriptions and comments support Emoji characters! 😍️

The site now prefers and enforces SSL for your security/privacy. #69

With the recent closure of MLKSHK, we’ve had a lot of requests for status updates on the MLTSHP project.

The technology team continues to meet weekly to review progress, and things are moving steadily towards launch. We identified several features that must be completed before launch:

1. Incorporating: We needed to create a new company: MLTSHP, Inc, to handle all business things like billing and legal matters. This is done!

2. Hotlinking Prevention: While reviewing MLKSHK’s bandwidth costs, we flagged hotlinking as a way to reduce costs. However, some clever changes in our caching infrastructure mean that we’ve removed this as a launch feature. We may still look into it in the future, but for now it’s moved to the back burner.

3. Ability to import MLKSHK content: This work has been completed and is being tested in our closed beta.

4. New Subscription Workflow: This work is nearly complete and is being tested in our closed beta.

We’ve also already completed many other tasks such as a new brand & logo, and reworking the site to be fully responsive for mobile devices

We don’t have a firm timeline for launching the new site, but as you can see we’re nearly done with all the launch blockers, and have entered the final stage of our beta test.