Tumblr Security

Self-Hosted Bug Bounty

When we first launched, the submission form wasn’t the prettiest. It looked like something you’d expect a backend-engineer writing HTML and CSS for the first time to generate, but it got the job done!

Submissions would come through an internal ticketing system that would send us an email. We’d reply, triage if valid, but generally have a pretty stressful time internally, because we considered even the smallest report to be a critical vulnerability.

Our definition of what was in scope was quite narrow and rigid, and our payout scale was very low in comparison to some of the other public programs that were coming out at the time ($1000 for critical!). But, our lower payout was a conscious choice: Our program explicitly stated that we wanted people to participate if they wanted to help improve security, not just because they wanted to make a living. 

We caught on, possibly years too late, that higher payouts were better. When a researcher has spent their time looking into the security of Tumblr, helped discover a vulnerability, and then responsibly disclosed it to us, we should be paying them fairly for helping to protect Tumblr! It was laughable to reward $100 and a "Thank You"—for protecting all of our users against a potentially wormable XSS vulnerability.

We shifted our payouts up a bit; they still weren’t very high for the industry, but we were happier knowing that we were doing what we could to help support the community of researchers. We also realized that our self-hosted approach wasn’t going to scale well. If we wanted to keep improving, we would need to change things up a bit.

Private Bug Bounty

And so, in early 2018, we launched a private program on the HackerOne platform. It was initially limited to 100 invited hackers while we got our bearings within the system. We learned pretty quickly that this shiny new tool was amazingly more advanced than what we were used to—it alone had us responding faster.

We grew to just over 1,000 invited hackers within a few months. We were receiving at least double the number of reports we had been getting from our self-hosted program, which was also still active. In short, things were looking very promising for our public launch in September 2018.

We also bumped up our payout scale for our private program. It was like money was raining from the sky for anyone who wanted to explore the depths of Tumblr! On our end, the learning continued, as we found that canned, email-style responses weren’t the norm. Some researchers are extremely knowledgeable, while others blatantly ignore program policies, and no matter how much we expanded the scope of our program, we were told it was still too limited!

Then, in September 2018, we held a Live Hacking Event that was hugely successful (A++, would do again). The goal was to launch our program to the public immediately after.

There and back again

Tumblr’s bug bounty program did go public shortly after our Live Hacking Event, but in a different way than anticipated: We merged our HackerOne program with that of our parent company at the time, Verizon Media (VZM). While the program itself was a success, and we still received reports, we ended up losing some of the progress and growth we had achieved in our own program. 

For one, we lost the ability to manage our own tickets at Tumblr and had to rely on the triage teams at HackerOne and VZM. These interactions were often great, but our efforts to shorten response times were being throttled by third parties. Additionally, the huge scope that was available under the VZM program meant that researchers' visibility into Tumblr’s assets was restricted, which in turn reduced our ticket volume substantially. 

That said, the perks of having a merged program were pretty great. Communication was consistent between tickets because there was a defined runbook for everything. The payout scale, yet again, was increased from what we previously offered. We had visibility into the inner workings of a much larger bug bounty program. We were invited to other Live Hacking Events and talked to a lot of awesome folks who dealt with bug bounties at a much, much larger scale than what we were used to.

When Automattic bought Tumblr, our bug bounty program was destined to separate from VZM’s. Even without access to managing our own tickets, the number of processes and capabilities we learned from the VZM team was enormous, and we’ll always be thankful.

Public Launch!

In October 2019, we finally launched our public program on HackerOne! Our ticket volume suddenly quadrupled from what it had been when we were private, and we've already received hundreds of reports in such a short time.

Using what we learned under VZM’s program, we’ve expanded our program policy. Now, everything that's public at Tumblr is in scope. We’ve set up a custom SSRF server for researchers to target during testing, and we’re going to expand tools just like that in the future, too. We also try to be as open and communicative to reporters as possible, as hurting Reputation or Signal is one of the last things we want to do.

Since the first few months have passed, the initial waves of reports have slowed down a bit, but we’re still very enthusiastic at being autonomous and interacting with the community in an efficient and successful program.

Operational efficiency

Our response statistics have continued to improve over the years. It used to take us days to respond to researchers, whereas our average response time is now within an hour of receiving a report. Remediation has always been a priority for us. Even as response times decrease, the time-to-remediation has remained low, with incidents usually being completed on the day we receive and triage the report.

It has taken many iterations of our program—both internally and externally—to achieve our current efficiency stats. We used to copy and paste common responses from a central repository, but are now able to rely on HackerOne’s inline tooling. And don’t get us started on how helpful triggers are! Also, by leveraging their Slack integration, we can receive report notifications in real-time, as opposed to relying on email and hoping no one else is already replying to the same report. 

We still use the HackerOne platform directly instead of their API, but even so, our process is very efficient. At the end of the day, it’s the community that’s helping Tumblr, so we want to make sure it’s a great experience for researchers, too. We may not be able to reduce our response times further, but we’re certainly going to try!

What’s next?

Going into 2020 with a new parent company, we’re planning to merge our program into Automattic’s. Both programs have proven successful, and we’re looking forward to learning from one another and making both as great as possible!

We’ll be launching more helpful tools and information for reporters to use, which will help make successes more noticeable. We’ll also continue to highlight new Tumblr features and services as they become available throughout the year, so that researchers have an idea of good places to target.

We’re excited for what's to come and look forward to working with everyone in the future!

Solving challenges & scoring points

To complete a challenge, the player had to achieve the goal within one of the listed categories.

In XSS challenges, the player would need to cause the browser to create an alert dialog (e.g. alert()).

Conversely, in SQL Injection challenges the player would need to read the flag column from the flags table in that challenge's database.

When the player successfully solved the challenge they were awarded with a flag, each in the format ofTumblr{s0mE_cHalL3nGe_j0kE-abcdef012}. That last piece is a unique hash for the user, per challenge, so that they couldn't directly share their flag. They can help others — even provide the solution — but they can't simply give away their flag.

Each challenge, when solved, is worth a certain number of points based on the challenge's difficulty and whether or not the player used the challenge's hints.

There were 3800 points available, though no player was able to break 1000!

At the end, we locked the leaderboard and announced the winners.

Prizes

We awarded the top four players based on their ranking on the leaderboard. First place got first dibs from the list. Second place gets to select theirs from the remaining lot, and so on.

Up for grabs this year:

Hak5 Elite Field Kit

Proxmark3 RDV2 Kit

Samsung Chromebook Plus

Lockpick set and a "how to" manual

Challenge snapshot

Throughout the eight categories we had a total of 46 challenges. We wanted to have a wide range of challenges that welcomed players of all backgrounds and experience levels.

The goal for XSS challenges was to get an alert dialog to appear. The player is presented with a vulnerable web page and they needed to determine where the vulnerability is and how to exploit it. Example:

These challenge levels ranged everywhere between simple non-sanitized output to DOM reflection to CSP bypasses.

One fo the more unique challenges to develop was SQL Injection. These offered players the ability to put their SQL skills to the test with a variety of basic input injection, blind injection, and filter bypassing challenges.

In at least one of the SQLi challenges, players had to inject into an INSERT statement. When creating challenges like this, special care had to be taken to give players the full capabilities of MySQL but also prevent them from revealing the flag to other players — it's a tricky thing making vulnerabilities secure!

The infrastructure

A frequent question I receive when I talk about deving on the CTF is "are you using CTFd?" Short answer? Nope! A slightly less short answer is that CTFd wasn't out when we started this =P.

The framework we're using is called "Security Training Grounds" and it's a custom-written project using PHP, PhantomJS, and MySQL (with HTML + JavaScript too, of course), running in Amazon Web Services.

An advantage of writing this in-house was that it gave us the ability to create a dynamic and robust system that has endless capabilities.

PHP + MySQL

The website was created from scratch, written in PHP with a little bit of jQuery + Bootstrap on the frontend and MySQL as the database.

The big thing here are the challenges themselves. Each challenge is hosted on its own subdomain. This enables us to provide live and interactive challenges like XSS or SQLi while still providing support static challenge types like Crypto or Reverse Engineering.

We accomplished this by allowing dynamic hostnames on the webserver and defining a subdomain hostname for each challenge that's stored in MySQL. When a web request comes in, the app checks whether it's a subdomain or not. If so, it hits the database to determine what to display.

For most challenges, we were able to handle all of the dynamic pieces directly in PHP. For some, such as the C or Java reverse engineering challenges, we did need to shell out to gcc or javac to build the custom binaries for each user.

PhantomJS

A difficulty for XSS and CSRF challenges is determining whether or not the participant successfully exploited the system. Surely we don't want to manually confirm for each flag, and attempting to pattern-match on their input would be crazy.

Our solution: execute what the player submits!

This is my own little baby, a piece of the system I'm so excited by. See, what better way to test XSS than to actually test XSS. As mentioned in the "Challenge snapshot" section above, when a player is working on a XSS challenge, they are given a website that has a XSS vulnerability. Their goal is to make an alert dialog appear. This is key and the requirement of the XSS framework itself.

On the client, we use this fancy little snippet:

var ctf_alert = alert; alert = function(msg) {    ctf_phantomjs_alert_test(document.location, msg); };

This overrides the window's actual alert() function and lets us put some processing logic in the middle. The logic is to take a snapshot of the current page - the URL, query string, POST parameters, the cookies and then pass the full snapshot to a backend PhantomJS service (via a PHP proxy, to help prevent tampering).

The PhantomJS service replicates that entire request and loads the target web page. If the page invokes an alert() call, which we catch via PhantomJS's onAlert, then we return with a "success" and the PHP proxy will return the user's flag. Our alert() overriding logic will then replace whatever message the user attempted to display and display their flag instead. Fancy af.

CSRF has a similar setup, except the player needs to submit their full CSRF payload:

After submitting the payload to the PHP proxy, we pass the payload to PhantomJS. This executes the payload in the context of an empty web page. If the PhantomJS worker successfully falls victim to the targeted action, the PHP proxy will return a flag to the user!

Open source

The framework code, as-is, is still relatively hacky and written with internal dependencies. We do believe in OSS though! We expect a near-future initiative to rewrite portions of it so we can release it for others to use for their CTF events, too.

Wanna Play?