Making Pinterest

This week we announced the open-sourcing of PINRemoteImage, a library we use to download more than three billion images to the Pinterest iOS app every day and quickly render them back to tens of millions of Pinners. So if you have an image-heavy app, instead of writing new code, this technology allows you to download, cache (on disk and memory) and prefetch images to reliably display them to your users. Unlike other similar technologies, PINRemoteImage maintains the following features while also avoiding deadlocks, and can even improve the experience on slower connections.

PINRemoteImage features include:

Download and processing priority

Image decompression off the main thread

Attractive blurred progressive JPEGs

Image post-processing

Automatically choose and download different images depending on recent network conditions

Optional support of FLAnimatedImage (Flipboard’s GIF library)

Optional support for WebP

Never blocks on the main thread

Motivation

Displaying remote images is a core part of Pinterest’s functionality, so the download and display of images need to be easy and efficient. There are many existing libraries that offer this type of functionality, but we found they often had limitations such as deadlocking when large numbers of images were requested at once or efficiencies that can only be gained if all images downloaded are the same size. PINRemoteImage was born out of the need for an efficient, thread-safe image downloading library.

How it works

To download an image and set it on a UIImageView, the easiest thing to do is use the built-in category method setImageFromURL:. Calling this takes care of everything from downloading the image to caching it and setting it on the UIImageView. There are several categories already included with PINRemoteImage, but writing a new category is made easy by implementing a defined protocol, PINRemoteImageCategory. Implementing this protocol is further simplified: almost all instance methods call matching class methods on PINRemoteImageCategory.

If you’d rather use the image directly you can call methods on a shared instance of PINRemoteImageManager, which is the workhorse class of PINRemoteImage. It will coalesce downloads, cache fetches and image processing operations so no matter how many times you request an image to be downloaded, the download and disk cache fetches will only happen once. It can also be used if you need to modify options which may not be exposed on the UIView categories.

All caching to disk and memory is handled by PINCache, our open-source object caching library.

Processing images

PINRemoteImage also supports processing images and storing the resulting image in the image cache. For example, the code below would download an image, resize it and apply a corner radius:

[imageView setImageFromURL:heroURL processorKey:@"rounded-hero" processor:^UIImage *(PIRemoteImageManagerResult *result, NSUInteger *cost) { UIImage *image = result.image; CGFloat radius = 7.0f; CGSize targetSize = CGSizeMake(100, 100); CGRect imageRect = CGRectMake(0, 0, targetSize.width, targetSize.height); UIGraphicsBeginImageContext(imageRect.size); [[UIBezierPath bezierPathWithRoundedRect:imageRect cornerRadius:radius] addClip]; CGFloat widthMultiplier = targetSize.width / image.size.width; CGFloat heightMultiplier = targetSize.height / image.size.height; CGFloat sizeMultiplier = MAX(widthMultiplier, heightMultiplier); CGRect drawRect = CGRectMake(0, 0, image.size.width * sizeMultiplier, image.size.height * sizeMultiplier); if (CGRectGetMaxX(drawRect) > CGRectGetMaxX(imageRect)) { drawRect.origin.x -= (CGRectGetMaxX(drawRect) - CGRectGetMaxX(imageRect)) / 2.0; } if (CGRectGetMaxY(drawRect) > CGRectGetMaxY(imageRect)) { drawRect.origin.y -= (CGRectGetMaxY(drawRect) - CGRectGetMaxY(imageRect)) / 2.0; } [image drawInRect:drawRect]; UIImage *processedImage = UIGraphicsGetImageFromCurrentImageContext(); UIGraphicsEndImageContext(); return processedImage; }];

This method could be called multiple times, and the image would only be processed once.

Combatting slow connections

PINRemoteImage has several methods for improving the experience of downloading images on slow connections, including support for Google’s WebP format. If WebP is available (by including the webp CocoaPod, for example), PINRemoteImage will detect WebP downloads and decode them appropriately with the WebP library:

[imageView setImageFromURL:[NSURL URLWithString:@”webPImage.webp”]];

Alternatively, if you’re downloading progressive JPEGs, PINRemoteImage supports displaying progressive scans of the image with an attractive blur applied (more attractive when not filtered through a GIF as below, as this feature is only available on iOS 8 and above). To support progressive scans, enable updates on a UIImageView by calling setUpdateWithProgress:YES, or use one of the progressive methods on PINRemoteImageManager:

[imageView setUpdateWithProgress:YES]; [imageView setImageFromURL[NSURLURLWithString:@”progressiveJPEG.jpg”]];

A final method for gracefully supporting slow connections is to provide the library with several URLs of varying quality.

[imageView setImageFromURLs:@[[NSURL URLWithString:@"lowQualityURL"], [NSURL URLWithString:@"mediumQualityURL"], [NSURL URLWithString:@"highQualityURL"]]];

PINRemoteImage will download one of the images based on most recently observed network  conditions. So if the last few images download slowly, it’ll choose a lower quality image to download. If you’d like to upgrade these images to a higher quality version, you can tell the image manager it should upgrade low quality images and set the image URLs on the image view again:

[[PINRemoteImageManager sharedManager] setShouldUpgradeLowQualityImages:YES completion:nil]; [imageView setImageFromURLs:@[[NSURL URLWithString:@"lowQualityURL"], [NSURL URLWithString:@"mediumQualityURL"], [NSURL URLWithString:@"highQualityURL"]]];

Support for FLAnimatedImage

PINRemoteImage also has native support for FLAnimatedImage, a memory efficient GIF decoder and player. If you have an instance of an FLAnimatedImageView, downloading and setting the GIF is as easy as on a UIImageView:

[flaAnimatedImageView setImageFromURL:[NSURLURLWithString:@”gifURL.gif”];

Contributing to PINRemoteImage

As with our other open-source contributions, we believe that the community can make PINRemoteImage even better. Whether it be through pull requests or bug reports, we’re excited to make PINRemoteImage better. To get started, check out our GitHub repository where you can see how to install and use PINRemoteImage, as well as play with a sample app (it’s full of kittens).

Garrett Moon is a software engineer on the iOS Team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

As a mobile-first company, a big focus is designing and engineering for the 80 percent of Pinners who access Pinterest on a mobile device. The small team of cross-platform mobile engineers recently spoke at a Discover Pinterest event. Wendy Lu and Carl Rice presented their work on improving a Pinner’s initial experience on iOS and the overall experience on Android. We also invited Google Product Director Luke Wroblewski to talk about working in a mobile-forward world with a focus on designing apps for various screen sizes. Here’s a recap of the presentations below, and we’ve posted a video of the event on our YouTube channel.

Screen time (Luke Wroblewski)

Currently a product director at Google, Luke is a serial entrepreneur who led his two startups, Polar and Bagcheck, to acquisitions by Google and Twitter. In his talk, he discusses screen time, a concept that's come a long way from its initial definition of how long we could sit in front of a TV. Today, people move between and simultaneously use screens of various sizes, proportions and quality throughout the day. Luke hones in on ways engineers and designers compensate for the intricacies of each device, modifying UI designs using RESS techniques, vertical media queries and more.

Rebuilding new user education on mobile (Wendy Lu)

Wendy joined Pinterest in early 2013 as a part-time engineer while she finished her undergraduate degree in computer science at Stanford University. Since joining full-time after graduation, she’s focused on iOS engineering, and has worked on the iOS7 redesign, our launch in the Japan, the 'News' feature, buyable Pins and more. In her presentation, she discusses building a dynamic new user experience that performs significantly better than previous experiences across all engagement metrics,  and the cross-platform system that supports it.

Optimizing the Pinner’s Android experience (Carl Rice)

Carl's experience in mobile engineering spans Android and iOS on both mobile apps and tablets. At Pinterest, he's focused on Android development, and played a big part in the initial app release in 2012, and throughout all app updates. For this event, Carl talked about the development and redesign of our Android app to create a better experience for Pinners, including optimizing content layout, streamlining common flows and catering to different devices.

Evrhet Milam is an engineering manager at Pinterest.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

We recently announced the Pinterest Developers Platform, a suite of APIs for building integrations and apps that help people do more with their Pins online and off. Now, to help developers create the best experiences  possible, we’ve given our developers site a facelift.

On our new site you’ll find docs, downloads and tools—you can even use the site to submit your  ideas. (Before submitting, just make sure you’ve read all the guidelines and policies we’ve added to the site.)

Need inspiration? Check out the first round of Pinterest integrations:

IFTTT: If you use IFTTT, starting today you can connect your Pinterest account to hundreds of products and devices through the Pinterest Channel. Logging in with Pinterest lets you automatically save any photo you like on Instagram as a Pin. It also works with actions in other apps, like Reddit upvotes, Etsy favorites or any other Pinterest "Recipe" that's created on IFTTT.

Polyvore: You can also seamlessly save your Polyvore sets to Pinterest. Any fashion, beauty and home decor collections and sets you make can be saved as Pins, right from your iPhone, iPad or desktop.

Pinners can remove access at any time by going to the Apps section of their Pinterest settings.

We’re just getting started—if you’ve already sent us an idea and you haven’t heard from us yet, you’ll be hearing from us soon. We’ve gotten tons of great submissions so far, and we’re looking forward to working with the developer community in the coming months!

Josh Inkenbrandt is a product manager on the Developer Platform team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

When I joined Pinterest, my first three weeks were spent in Base Camp, where the newest engineering hires work on real production issues across the entire software stack. In Base Camp, we learn how Pinterest is built by building it, and it's not uncommon to be pushing code and making meaningful contributions within just a few days. At Pinterest, newly hired engineers have the flexibility to choose which team they’ll join, and working on different parts of the code as part of the Base Camp experience can help with this decision. Base Campers typically work on a variety of tasks, but my project was a deep dive into a MySQL performance optimization project.

Pinterest, MySQL and AWS, oh my!

We work with MySQL running entirely inside Amazon Web Services (AWS). Despite using fairly high-powered instance types with RAID-0 SSDs and a fairly simple workload (many point selects by PK or simple ranges) that peaks around 2,000 QPS, we had been unable to realize anywhere near the expected IO performance levels. Exceeding roughly 800 write IOPS would lead to unacceptable increases in latency and replication lag, and this replication lag or insufficient read performance on the slaves would slow down ETL and batch jobs, which would then have downstream impact on any other team relying on those jobs. The only options available were either to go to an even larger instance size, thus doubling our cost and obliterating our efficiency, or find ways to make the existing systems perform better.

I took over the project from my colleague, Rob Wultsch, who had already made the significant discovery that Linux kernel version appears to be quite important when running on SSD inside AWS. The default 3.2 kernel that ships with Ubuntu 12.04 just doesn't cut it, nor does the 3.8 kernel that AWS recommends as a minimum version (although it's still more than twice as fast as 3.2). Running sysbench on an i2.2xlarge (2 disk RAID-0 of SSDs) instance with kernel 3.2 could barely hit 100MB/sec of 16K random writes. Upgrading the kernel to 3.8 got us to 350MB/sec with the same test, but this was still much lower than expected. Seeing this kind of improvement from such a simple change opened up many new questions and hypotheses about other inefficiencies and poor configuration options: Could we get better performance from an even newer kernel? Should we change other settings at the OS level? Are there optimizations to be found in my.cnf? How can we make MySQL go faster?

In pursuit of answers, I set up almost 60 different sysbench fileIO test configurations with different kernels, filesystems, mount options and RAID block sizes. Once the best fit configuration was chosen from these experiments, I ran another 20 or so sysbench OLTP runs with other system permutations. The basic test methodology was identical across all trials: run the test for an hour collecting metrics at one second intervals, then drop the first 600 seconds to account for cache warm-up time and process the remainder. After the optimal configuration had been identified, we rebuilt our largest and most critical servers and rolled out the changes into production.

From 5000 QPS to 26000 QPS: scaling MySQL performance without scaling hardware

Let's take a look at the impact of these changes on some basic sysbench OLTP tests via the p99 response times and throughput metrics at 16 and 32 threads for several different configurations.  

Here is what each of the numbers represent:

CURRENT:                    3.2 kernel and standard MySQL configuration

STOCK:                          3.18 kernel with standard MySQL configuration

KERNEL:                        3.18 kernel with a few IO/memory sysctl tweaks

MySQL:                          3.18 kernel with an optimized MySQL configuration

KERN + MySQL:         3.18 kernel with tweaks from #3 and #4

KERN + JE:                    3.18 kernel with tweaks from #3 and jemalloc

MySQL + JE:                3.18 kernel with MySQL config from #4 and jemalloc

ALL:                                 3.18 kernel with #3,  #4 and jemalloc

When we enable all of the optimizations, we find we can achieve roughly 500 percent more read and write throughput at both 16 and 32 threads while simultaneously reducing p99 latency by over 500ms in both directions. On the read side, we go from approximately 4100 – 4600 QPS to just over 22000 – 25000, depending on concurrency. On the write side, we go from approximately 1000 QPS to 5100 – 6000 QPS. These are massive gains in headroom and performance achieved with just a few simple changes.

Of course, all the synthetic benchmarks in the world don't mean much if they don't translate into real-world results. The graph below shows latency on our primary clusters from both the client and server perspective from several days before the upgrades were pushed until several days after all the masters were upgraded. The process took just a week to complete.

The red line represents client-perceived latency, and the green represents server-measured latency. From the client perspective, p99 latency dropped from a highly-variable 15-35ms with outliers over 100ms down to a pretty flat 15ms with outliers at 80ms or less. Server-measured latency also declined from a wavy 5-15ms to a basically-flat 5ms, with a daily 18ms spike due to system maintenance. Furthermore, since the beginning of the year, our peak throughput on these clusters has increased about 50 percent, so not only are we handling considerably more load (still well under our estimated capacity), we're doing it with much better and more predictable throughput. And, in what can only be termed good news for everyone who enjoys sleeping through the night, the number of pageable incidents related specifically to system performance or general server overload dropped from over 300 in March down to less than 10 in April and May combined.

For more information, including the before-and-after details of what our MySQL and OS configurations look like, check out the slides from “All Your IOPS Are Belong To Us,” my talk from the 2015 Percona Live MySQL Conference/Expo, and stay tuned for more insights on how we get the most out of MySQL, Redis and other storage technologies.

Ernie Souhrada is a database engineer on the SRE team, part of the Cloud Engineering team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

We use Hadoop/MapReduce batch jobs extensively to process content and activity streams. The home feed is a prime example where we employ such batch jobs to compute signals and features to create a personalized feed of interesting Pins for every Pinner. While this batch approach is effective and scalable, it’s not responsive to recent activity. RoadRunner, a stream compute infrastructure, was born to address the need for near real-time feedback cycle.

Introducing RoadRunner

RoadRunner processes a stream of activities (repins, searches, clicks, etc.) as they happen and aggregates them in near real-time. From these we derive meaningful estimates of the quality and category of content, interests of Pinners and more. These signals are made available to our search and home feed ranking models that determine which Pins are most likely to appeal to a Pinner. RoadRunner also preserves a lot of temporal information about the activity stream. For example, we can discern how quality signals vary over time which allows us to understand what content is popular on weekends, what’s trending or what has changed recently.

Under the hood

RoadRunner is built on top of Apache Storm for streaming computation, HBase for storage and Finagle for serving. It crunches about 10 billion events daily at a peak rate of ~30K/sec with end-to-end latency of just a few seconds (p99 < 10s).

We needed a storage technology to support low latency high QPS increments and very low latency reads/scans (look up hour by hour metrics for a Pin over the last seven days). HBase was a great fit as it supports atomic increments and scans, and it’s horizontally scalable and widely used throughout our engineering teams. Our HBase cluster uses SSDs to meet our latency requirements. The HBase keys are a combination of objectID and time window ID to expire old data, capture timeseries and support aggregation over dynamic ranges. This design allows for the merging of results from both batch systems and roadrunner.

We also spent time tuning the storm topology. To keep the storm topology free flowing, we use a future pool for the blocking updates to HBase. This worked better than increasing the number of bolts that were responsible for performing updates. We built an end-to-end audit system that pumps events for test objects and checks the service for expected updates in metrics in just a few seconds. This periodic check helps make sure the entire system is healthy.

Using RoadRunner data

The data from RoadRunner is combined with data from nightly aggregates to generate feature vectors that go into the home feed ranking models. There’s some overlap between the features computed by RoadRunner and those computed by the Hadoop jobs. While both compute estimates of the Pin quality, the estimates from Hadoop are over the lifetime of a Pin (or at least until the nightly job was run), and estimates from RoadRunner are over the last several days. Both have strengths and weaknesses. The RoadRunner estimates are fresher and more flexible which allows us to incorporate time of day- or day of week-based variation into the models. However, they’re more noisy since they’re based on less data and are less complete. Our models treat the batch and real-time estimates as two completely different features. The training procedure we use doesn’t require the features to be independent, and so this works well for us.

We built and launched RoadRunner for search and home feed first. Since, we’ve found several additional use cases, from personalization of new user in-product guides to monetization products. We’ve started work on a version 2 to support this vast array of needs. We envision the next version to be a self-service platform that allows you to easily declare features and also apply lambda architecture for improved fault tolerance.

Stay tuned for future blog posts that discuss training and evaluation of models that use these dynamic features, and what’s new in version 2.

Ramki Venkatachalam and Mukund Narasimhan are software engineers on the Data Team and the Recommendations Team, respectively.

Acknowledgements: RoadRunner is part of a long-term strategic effort to enable streaming compute solutions at Pinterest. It was developed in collaboration with many folks from the Platform, Discovery and Cloud teams.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

Earlier this week, we announced buyable Pins, a simple and secure way to buy your favorite products on Pinterest. When building buyable Pins, we focused on making the technology easy and fun to use from a mobile device, but even more important was building security right into the feature.

How does it work?

We’ll be working with our primary partner Stripe, as well as Braintree, two industry-leading payment providers who’ve been protecting people’s information for years. We’ll be working with both Stripe and Braintree to vault credit cards.

Once a Pinner enters their credit card information in the Pinterest app, we send it via an encrypted channel to the providers, who store it inside a secure vault. The merchant then charges the credit card from the vault through their existing payment processor and lets Pinterest know that the order was successful.

Security and trust are incredibly important to us. We chose to work with these providers for their experience and the trusted relationships they have with the payment processors who work with the world’s best merchants.

This is just the beginning. With this technology and design, we’ll be able to work with merchants of all sizes and help Pinners everywhere discover and buy the products they love.

Announcing buyable Pins to Pinners and partners at Pinterest HQ.

Stay tuned for more posts from the buyable Pins engineering team. If you’re interested in tackling new engineering challenges as we roll out buyable Pins, join our team!

Wendy Lu is an engineer on the iOS team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

Discovery on Pinterest is all about finding things you love, even if you don’t know at first what you’re looking for. The Visual Discovery engineering team is tasked with building technology that will help people to continue to do just that, through features that can understand the objects in a Pin’s image to get an idea of what a Pinner is looking for.

Over the last year we’ve been building a large-scale, cost-effective machine vision pipeline and stack with widely available tools with just a few engineers. We faced two main challenges in deploying a commercial visual search system at Pinterest:

As a startup, we needed to control the development cost in the form of both human and computational resources. Feature computation can become expensive with a large and continuously growing image collection, and with engineers constantly experimenting with new features to deploy, it’s vital for our system to be both scalable and cost-effective.  

The success of a commercial application is measured by the benefit it brings to the user (e.g., improved user engagement) relative to the cost of development and maintenance. As a result, our development progress needed to be frequently validated through A/B experiments with live user traffic.

Today we’re sharing some new technologies we’re experimenting with, as well as a white paper, accepted for publication at KDD 2015, that details our system architecture and insights from these experiments and makes the following contributions:

We present a scalable and cost-effective implementation of a commercially deployed visual search engine using mostly open-source tools. The tradeoff between performance and development cost makes our architecture more suitable for small-and-medium-sized businesses. 

We conduct a comprehensive set of experiments using a combination of benchmark datasets and A/B testing on two Pinterest applications, Related Pins and an experiment with similar looks, with details below.  

Experiment 1: Related Pin recommendations

It used to be that if a Pin had never before been saved on Pinterest, we weren’t able to provide Related Pins recommendations. This is because Related Pins were primarily generated from traversing the local “curation graph,” the tripartite user-board-image graph evolved organically through human curation. As a result, “long tail” Pins, or Pins that lie on the outskirts of this curation graph, have so few neighbors that graph-based approaches do not yield enough relevant recommendations. By augmenting the recommendation system, we are now able to recommend Pins for almost all Pins on Pinterest, as shown below.

Figure 1. Before and after adding visual search to Related Pin recommendations.

Experiment 2: Enhanced product recommendations by object recognition

This experiment allowed us to show visually similar Pin recommendations based on specific objects in a Pin’s image. We’re starting off by experimenting with ways to use surface object recognition that would enable Pinners to click into the objects (e.g. bags, shoes, etc.) as shown below. We can use object recognition to detect products such as bags, shoes and skirts from a Pin’s image. From these detected objects, we extract visual features to generate product recommendations (“similar looks”). In the initial experiment, a Pinner would discover  recommendations if there was a red dot on the object in the Pin (see below). Clicking on the red dot loads a feed of Pins featuring visually similar objects. We’ve evolved the red dot experiment to try other ways of surfacing visually similar recommendations for specific objects, and will have more to share later this year.

Figure 2.  We apply object detection to localize products such as bags and shoes. In this prototype, Pinners click on objects of interest to view similar-looking products.

By sharing our implementation details and the experience of launching products, we hope visual search can be more widely incorporated into today’s commercial applications.

With billions of Pins in the system curated by individuals, we have one of the largest and most richly annotated datasets online, and these experiments are a small sample of what’s possible at Pinterest. We're building a world-class deep learning team and are working closely with members of the Berkeley Vision and Learning Center. We’ve been lucky enough to have some of them join us over the past few months.

If you’re interested in exploring these datasets and helping us build visual discovery and search technology, join our team!

Kevin Jing is an engineering manager on the Visual Discovery team. He previously founded Visual Graph, a company acquired by Pinterest in January 2014.

Acknowledgements: This work is a joint effort by members of the Visual Discovery team, David Liu, Jiajing Xu, Dmitry Kislyuk, Andrew Zhai, Jeff Donahue and our product manager Sarah Tavel. We’d like to thank the engineers from several other teams for their assistance in developing scalable search solutions. We’d also like to thank Jeff Donahue, Trevor Darrell and Eric Tzeng from the Berkeley Caffe team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

Among the things a developer worries about most, deploy is near - or at - the top of the list. A deployment is often the first time a new code change runs in the production environment. A dependable and straightforward deploy tool is a crucial part of any developer’s arsenal.

A deploy system should support the following functionalities:

Rollback. This is the most important feature of any deploy tool. Having a time machine to go back to a certain previous state is priceless.

Hotfix. There are times when rollback is either impractical or hard, and so a hotfix is easier to perform and faster to deploy with higher priority over the regular ones.

Rolling deploy. Deploy shouldn’t interrupt service, but if absolutely necessary, the impact has to be minimal. It’s important to halt the deployment if a certain number of servers have failed to upgrade or a service SLA is violated.  

Staging and testing. Deploying to production directly has higher risks than deploying to a staging environment or canary to verify things work first. Often times engineers don’t follow this best practice because of the overhead of creating a staging environment and integrating it with their tests. A good deploy system minimizes such overhead.

Visibility. Make sure it’s easy to find out which code changes are available to deploy and the number of hosts running new and older versions. It’s also important to easily track which code change was introduced when and by whom, as well as the critical metrics and alarms status during a deployment.

Usability. A simple user interface is key for the above functionalities.

Introducing Teletraan

Teletraan is our internal deploy system for supporting all of the above functions (named after a character from the famous Transformer TV series). It was built by a small group of development tools engineers on the Cloud Engineering team that drives reliability, speed, efficiency and security for the site and infrastructure.

Design overview

Teletraan follows the traditional client-server model with MySQL as the backend data storage.

Deploy agents are daemons running on all the hosts and interact with Agent Service periodically to get the latest instructions. During a deployment, an agent downloads and extracts service build tar along with specific deploy scripts, and executes them. These deploy scripts include PRE/POST-DOWNLOAD scripts, PRE/POST-RESTART scripts and the RESTART script itself, and are responsible for stopping and starting services.

Teletraan Workers perform jobs in the background, such as transitioning deploy states based on deploy progress and performing auto deploys based on schedule.

Teletraan Service provides APIs support for Deploy Board and any RESTful calls. It’s responsible for most deploy-related actions, including deploy and rollback. It also creates and maintains service deploy configuration, answers deploy and agent status queries, enforces permission control and more.

Advanced features

In addition to the core functionalities listed above, Teletraan also supports several advanced features:

Pause and resume. It comes in handy when a developer wants to double check  something before the code is fully deployed to the cluster.

Qualification. Once configured, a successful deploy will trigger an acceptance test to qualify it. An accepted deploy could be used for future promote or auto deploy to the next stage

Auto Deploy. Automatically promote builds from one stage to another whenever a new build is available or based on cron job-like schedule settings. An auto deploy could be paused, rollbacked or overridden by the system automatically upon failures.

Teletraan has helped us move faster and ship code easier. We want to share these deploy tools with the world, and are planning to open-source Teletraan later this year. Keep an eye on the blog for updates.

Baogang Song is an engineering lead on the Internal Development Tools team, which is part of the Cloud Engineering team at Pinterest.

Acknowledgements: Teletraan was built by Jinru He, Nick DeChant and Baogang Song from the Internal Development Tools team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

Every day, millions of people use Pinterest to save creative ideas for recipes, articles, places to travel, products and more. Today we’re announcing the beta version of the Pinterest Developers Platform, a suite of APIs for developers to build apps and integrations that bring Pins to life.

On Pinterest, people have saved over 50 billion Pins. As a developer, you’ll have the chance to build something that makes those Pins a reality, like apps for ordering ingredients from a recipe Pin or booking trips based on travel Pins.

When Pinners sign in to your app with Pinterest, you can:

Build a personalized, curated experience based on their boards and Pins

Let people easily create multiple Pins or boards to get more of your content distributed across Pinterest

We’d love to hear your app ideas—take a look at our technical overview and apply for beta access. We’re looking forward to working with developers (only in the U.S. to start).

Josh Inkenbrandt is a product manager on the Developer Platform team.

Every day, tens of millions of people discover and save Pins on Pinterest, making the Pinning flow one of the most important features. It’s so important that we’re constantly and carefully making updates to it to ensure ease-of-use and fast load times.

As an active Pinner, I have more than 20 boards where I’m curating collections of Pins. Before, I often found myself scrolling through the drop down list of my many boards trying to find the right board for each pin. This was especially challenging on mobile where only a few boards appeared on the screen at a time. The data showed I wasn’t the only one having this frustrating experience, because active Pinners have 24 boards on average.

As a former Pinterest intern (class of 2013), I started building a smart board picker for iOS and Android in the Pinning flow during a Make-a-thon (our version of a hack-a-thon). The goal was to provide suggested boards based on the match of the content of the Pin and a Pinner’s boards to make Pinning easier.

The magic

We have a rich collection of metadata for the content on our service. For most Pins, we know exactly what they're about, such as category (like travel), and which keywords are most important based on descriptions written by tens of thousands of Pinners. The same applies to boards. Using these rich data we maintain, we concluded we could predict which board a Pinner intends to save to.

When a Pinner repins, we run a scoring algorithm using the Pin against all the Pinner’s boards and return the best-scored board(s) based on content match. The scoring algorithm uses a linear combination of the following features:

Category match: whether the Pin’s category is matched with the board’s category. The Pin and board’s category information is represented as a unit vector, where each dimension tells the probability of the Pin or the board falling in this category. The match is represented as the dot product of the two category vectors.

Keyword match: we use keywords to describe a Pin or board. These textual terms place each object in a high dimensional space. We measure closeness in that space by inner product of the two sparse vectors of keywords from the Pin and the board.

Activeness of the board: how frequent new Pins are added to the board is also considered in the scoring algorithm. A board that has been abandoned for a long time should be penalized in favor of more active boards.

Figure 1: Smart board picker behind the scenes

Tuning the user experience

Accuracy isn’t the only factor that makes a predictive feature successful. Getting the right user experience is even more important, especially for a feature embedded in the essential Pinning flow. A small mistake could ruin the Pinning experience for millions. Given we were introducing a small latency for fetching board suggestions in the Pinning flow, we needed to keep the impact of the additional latency as low as possible in order to make the repin experience better.

Unfortunately, we didn’t get it right the first try. Our first version on iOS showed the original board picker first (with the three most recently picked boards in the Quick Picks section) while fetching the board suggestions, and then swapped them in the Quick Picks section after they were fetched. Although the numbers showed Pinners were spending less time on the repin flow and quickly picking up this feature, we noticed there was a strange increase in the number of Pinners who edited or deleted Pins. We learned the swapping of boards in Quick Picks was providing a bad user experience (see in the animation below). Those who were trying to Pin to the recent boards would accidentally save to the wrong board when the swapping happened right before they tapped on the board.

Figure 2: First try of smart board picker

To make a better user experience, we added a small spinning circle, blocked the area reserved for suggested boards while fetching them and kept the Pinning flow active. We still show the most recently picked board at the first slot before the two predictions, and Pinners can scroll down a list of all their boards if they don’t want to wait for suggestions. Putting Pinners first is always our priority, and this design successfully reduced the accidental Pinning to wrong boards while still achieving our goal for this feature.

Figure 3: Improved version of smart board picker

Insights

There are several interesting insights from the experiment:

Making the Pinning flow easier to use increases repins and repinners: our tests show the smart board picker reduces abandons on the Pinning flow by 4 percent and increases overall repinners by 1 percent.

Suggesting correct boards in the Pinning flow reduces the efforts of re-organizing Pins later: our data confirms that the smart board picker reduces the number of Pinners who create boards by 2 percent, who edit Pins by 4 percent and who delete pins by 3 percent. In other words, Pinners are Pinning to the correct boards!

The smart board picker is helpful in the core Pinning flow: this feature increased the number of Pinners using boards in Quick Picks by 15 percent and reduced the average time spent on the Pinning flow by 10 percent. 

Come try the smart board picker on the latest iOS or Android apps if you haven’t already!

Yuchen Liu is a software engineer on Recommendations team at Pinterest.

Acknowledgements: Smart board picker was built in collaboration with Bin Liu on iOS platform and with Nikki Shah on Android platform. 

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

Our top priority is to create a great user experience in everything we build, and across platforms. As part of that, each day we work to reduce the effort it takes to use Pinterest. So when Apple announced support for shared web credentials, we were excited to make it easier for Pinners to log in.

Shared web credentials allows an iOS app to read and write passwords to the same store that Safari uses on iOS and Mac. For example, if a Pinner signed up for Pinterest on their Mac, they could download our iPhone app and immediately login without needing to remember or tap in their credentials. Magic!

How To

Before you can start using the APIs, there are two requirements to get iOS to trust your app with shared web credentials.

Add your domain to “Associated Domains” in the Capabilities section of your Xcode project:

Sign a JSON file and host it at https://<yourdomain.com>/apple-app-site-association. Note the https. The file should contain a dictionary that lists your approved app identifier and is signed to ensure it has not been tampered with. The listing below shows an example JSON file formatted for reading: Listing 1-1 Server-side web credentials:

{ "webcredentials": { "apps": [ "YWBN8XTPBJ.com.example.myApp"] } }

We used our pinterest.com certificate and key to sign the JSON file (more on the requirements of the certificate here). To sign the certificate, use the commands below: Listing 1-2 Signing the credentials file:

echo '{"webcredentials":{"apps":["YWBN8XTPBJ.com.example.myApp"]}}' > json.txt cat json.txt | openssl smime -sign -inkey pinterest.com.key -signer pinterest.com.pem -certfile intermediate.pem -noattr -nodetach -outform DER > apple-app-site-association

As a final step, host the produced file on your website under https://example.com/apple-app-site-association. This process should be relatively straightforward, however there are a couple of gotchas. First, be sure you have credentials in iCloud Keychain that match your domain. Without them the API methods will fail with an error (-25300, No matching items found). If the user clicks “Not Now” there will be no error, but zero credentials returned. Also, if you don’t get everything set up correctly the first time, you’ll want to delete the app between tests. Apple verifies your domain app link when your app is installed, so relaunching the app after fixing the issue won’t result in a new check.

Now, to request the user’s credentials in the app, just make a simple function call:

SecRequestSharedWebCredential(NULL, NULL, ^(CFArrayRef credentials, CFErrorRef error) { if (error) { //handle error } else { if ([(__bridge NSArray *)credentials count] > 0) { NSDictionary *dictionary = [(__bridge NSArray *)credentials objectAtIndex:0]; NSString *accountName = dictionary[(__bridge id)(kSecAttrAccount)]; NSString *password = dictionary[(__bridge id)(kSecSharedPassword)]; } else { // No accounts } } });

Design

Imagine you’ve just signed up for Pinterest on the web and then download our iOS app. Upon loading the app, you’re offered to immediately log in with your credentials, which is the ideal experience.

However, you don’t want to annoy users who have no interest in using the feature. So if the user clicks “Not Now,” offer to stop showing the dialog every time they’re on the login screen.

To create the most seamless experience, integrate shared web credentials in other parts of your app.

When the user logs in successfully, call SecAddSharedWebCredential to update the user’s credentials so that they can use them when they visit your website later.

If you have a password change feature, be sure to also update credentials in the keychain as above.

Finally, make sure to have a button to login with the credentials on your login screen.

Results

After introducing support for shared web credentials we saw modest increase in engagement of our Pinners, which made the small effort well worth the time. Encouraging people to use features like iCloud Keychain will help users create more secure passwords without having to remember them, which is a win for everyone involved.

Garrett Moon is an iOS engineer on the Mobile team.

Acknowledgements: Thanks to Amine Kamel, senior security engineer, for providing the how-to on signing JSON files properly.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

A good portion of the startups I meet and advise want to use the newest, hottest technology to build something that’s cool, but not technologically groundbreaking. I have yet to meet a startup building a time machine, teleporter or quantum social network that would actually require some amazing new tech. They have awesome new ideas with down-to-earth technical requirements, so I kept wondering why they choose this shiny (and risky) new stuff when all they need is a good ol’ trustworthy database. I think it’s because many assume that building the latest and greatest needs the latest and greatest!

It turns out that’s only one of three bad reasons (traps) why people go for the shiny and new. Reason two is people mistakenly assume older stuff is slow, not feature rich or won’t scale. “MySQL is sluggish,” they say. “Java is slow,” I’ve heard. “Python won’t scale,” they claim. None of it's true.

The third reason people go for shiny is because older tech isn’t advertised as aggressively as newer tech. The younger companies needs to differentiate from the old guard and be bolder, more passionate and promise to fulfill your wildest dreams. But most new tech sales pitches aren’t generally forthright about their many failure modes.

In our early days, we fell into this third trap. We had a lot of growing pains as we scaled the architecture. The most vocal and excited database companies kept coming to us saying they’d solve all of our scalability problems. But nobody told us of the virtues of MySQL, probably because MySQL just works, and people know about it.

Through the gauntlet, two of the most important lessons I learned building Pinterest were:

Don’t be the biggest. If you’re the biggest user of a technology, your challenges will be greatly amplified.

Keep it simple. No matter what technology you’re using, it will fail.

After about a year of fast, sleep-defying scaling at Pinterest, we had MySQL, Memcache, MongoDB, Redis, Cassandra, Membase and Elastic Search. Everything was on fire and breaking in their own special ways. We wanted to simplify and get rid of all the fancy stuff, but we also wanted something that would scale to the moon with us. It was time to start our grand re-architecture. To help guide us and our choices, we built a set of questions to apply to every different technology:

Does the technology meet your needs? Solidify your requirements. Do you need simple lookups, ordered lookups and/or lookup by geography? Graph walking? One technology may not be able to support all of your requirements, but it’s nice when it does.

How well does it scale? Some technology is designed for massive scale and some is not. You may have to layer a little scale magic on top. With scale comes more complexity and less agility. Keep that in mind, and avoid scaling prematurely.

Is the cost justified? Consider the support contract or licensing and whether or not you can get by without a support contract. If you’re an angel funded startup, ideally you’d like to spend your money on the lights and Ramen, and not on a mainframe.

How mature is the technology? Maturity of a product is the most important question you can ask next to basic requirements. This is where I’ll spend the lion share of this post.

Maturity is natural

The harder and more passionately people push on a technology, the faster they will run across bugs or performance problems, fix them and hopefully contribute fixes back for the whole community to use.

Maturity can come a hell of lot faster if the technology is simple. Getting a five line Python program to print “Hello, World” to be bug free should take 30 person-seconds, whereas a program to handle world-wide distributed banking transactions will take many person-years to mature due to the complexity that must be hammered out.

So, let’s equation-ify this. Maturity increases with blood and sweat, but comes slower with more complexity.

Maturity = (Blood + Sweat) / Complexity

(I really wanted to throw in some e, i, and π into the equation, but just couldn’t justify it. Yet…)

As a technology hardens, collaboration occurs, understanding gets deeper and a wealth of knowledge is built out. As the technology itself gets more stable and beautiful, documentation and simplification occur and the frontiers and boundaries are tested and widened.

You don’t want to be on the wrong end of the maturity equation. There be dragons there:

Hiring will be more difficult Try searching on Google for MySQL admin and Cassandra admin. Try walking out into a busy San Francisco walkway and yell out that you need a MySQL admin. You’ll find one. Hbase, unlikely.

You’ll find minimal community It’s 2 a.m. and you’re getting a weird error message “48fe1a is opaque” (an actual error message I got from Membase). What the f!*k does that mean? Crap, there are there no answers anywhere on Google. Conversely, I can’t remember the last time a MySQL question I had wasn’t already answered along with somebody calling the questioner a nOObXor.

You’re more likely to fail, possibly catastrophically We had an unexpected loss of data on nearly every technology we used at one time or another, except MySQL. Even on the worst days, when the hard drive crashed, we still managed to find a script somebody wrote to do magic voodoo to get our MySQL data out again and live another day. Other technologies left us dead in our tracks because nobody had encountered the same problems, or they hadn’t taken the effort to dive deep to recover their data or they hadn’t contributed the fix back to the community. Incidentally, I’m super thankful we never trusted the golden copy of our data to any other system except MySQL in those early days.

Sometimes you have to be on the bad end of this maturity ratio. For instance, if you HAVE to have a Flux Capacitor for your time machine, recognize that you won’t be able to hire for it easily. There will be minimal community online to help you debug why you only went back in time 100 years and not 1,000 years. Support may not be able to help you understand why Marty McFly is now stuck in a supernova.

If you’re on the frontier, you’ll hit new bugs and issues that the rest of the world has never seen. They’ll be 10x harder to debug and will likely require a depth of knowledge that goes outside the comfort zone of your current engineers. You’ll have to dig in, push hard and learn fast. I send you my virtual hugs and admiration. I’ve been there. It will be tough. Blog what you find, collaborate and communicate.

If you’re starting or growing a company, and your scale is smaller than huge, consider maturity to be your most important factor aside from basic requirements. Ask yourself — does MySQL sufficiently meet my needs? If so, use it. If you’re wondering if MySQL will be fast enough, the answer is YES. Even better than fast, MySQL’s performance will be consistent.

Last Remarks

So I’ve wailed away on a bunch of technologies, but I seem to have a near-romantic thing for MySQL. I’d like to take a moment to mention that MySQL, while mature, does not solve all your problems. Sometimes you’ll have to venture away from the comforting warming glow of maturity.

Cartesian Distance If you need to search for nearby points in two dimensions, storing coordinates as Geohashes in MySQL would work well (here’s an XKCD comic to help). Three dimensions would probably also work well. But if you need large N-dimensional search spaces, I don’t know of a good way to store and retrieve them in MySQL efficiently. You might find yourself needing to store N-dimensional points if, for instance, you have created a model that produces feature vectors for some input and you want to see if two inputs are similar. Classic examples include determining if two images are similar but not exactly the same. For these sorts of situations, consider building a distributed RP/KD tree (would love to collaborate! Email me!).

Speed of writes MySQL delivers full write consistency. If you’re willing to trade off “full” for “eventual” consistency, your writes can be much faster. HBase, Cassandra and other similar technologies write to an update log incredibly fast, at the expense of making reads slower (reads must now read the stored info and walk the update log). This is a nice inversion, because it’s easier to cache reads and make them fast.

FIFOs, such as feeds My biggest complaint about MySQL is that it’s still living in 1994 (with baggy pants and the.. erk.. Macarena). Many uses of databases back then needed relational queries. There were no social networks. MySpace wouldn’t come around for another nine years! And so MySQL is built out of trees and has no good notion of queues. To insert into a B-tree is an O(lg(N)) operation (assuming happy balance). But today, social networks are a major force on the internet, and they depend heavily on queues. We want uber fast O(1) enqueuing! My suggestion is to not use MySQL for feeds. It’s too much overhead. Instead, consider Redis, especially if you’re still a small team. Redis is super fast and has lists with fast insertion and retrieval. If you’re a larger company and can hire the folks to maintain it, consider HBase. It’s working well for our feeds.

Logs For the love of everything holy, don’t store logs in MySQL. As mentioned in the previous paragraph, MySQL stores things in trees. Logs should not live in trees (that’s a weird thing to say...). Send your logs to Kafka, then use Secor to read from Kafka and ship up to S3. Then go nuts with EMR, Qubole or your map-reduce platform du jour.

Scale beyond one box If you’re in this position and you’ve optimized all queries (no joins, foreign keys, distinct, etc.), and you’re now considering using read slaves, you need to scale to beyond one MySQL server. MySQL won’t do this for you out of the box, but it’s not hard. And we have a solution, which you’ll be able to read about and learn how we did this once I finish writing the blog post. In the meantime, bug me at [email protected], and I’ll get you up and running now.

Marty Weiner is an engineering manager at Pinterest. Follow his adventures on Twitter and Pinterest. Interested in joining his team? Check out our careers site.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

The benefits of using a data-driven approach to product development are widely known. Most companies understand the benefits of running an A/B experiment when adding a new feature or redesigning a page. While engineers and product managers have embraced a data-driven approach to product development, few think to apply it to backend development. We’ve applied A/B testing to major infrastructural changes at Pinterest and have found it extremely helpful in validating those changes have no negative user-facing impact.

Bugs are simply unavoidable when it comes to developing complex software. It’s often hard to prove you’ve covered all possible edge cases, all possible error cases and all possible performance issues. However, when replacing or re-architecting an existing system, you have the unique opportunity to prove that the new system is at least as good as the one it’s replacing. For rapidly growing companies like Pinterest, the necessity to re-architect or replace one component of our infrastructure happens relatively frequently. We rely heavily on logging, monitoring and unit tests to ensure we’re creating quality code. However, we also run A/B experiments whenever possible as a final step of validation to ensure there’s no unintended impact on Pinners. The way we run the experiment is pretty simple: half of Pinners are sent down the old code path and hit the old system and the other half use the new system. We then monitor the results to make sure there’s no impact across all our key metrics for Pinners in the treatment group. Here are the results of three such experiments.

2013: A new web framework

Our commitment to A/B testing infrastructural changes was forged in early 2013 when we rewrote our web framework. Our legacy code had grown increasingly unwieldy over time, and its functionality was beginning to diverge from that of our mobile apps, because it ran through completely independent code paths. So we built a new web framework (code-named Denzel) that was modular and composable and consumed the same API as our mobile clients. At the same time we redesigned the look and feel of the website.

When it came time to launch, we debated extensively whether we should run an experiment at all, since we were fully organizationally committed to the change and hadn't yet run many experiments on changes of this magnitude. But when we ran the experiment on a small fraction of our traffic, we discovered not only major bugs in some clients we hadn't fully tested but also that some minor features we hadn't ported over to the new framework were in fact driving significant user engagement. We reinstated these features and fixed the bugs before fully rolling out the new website, which gave Pinners a better experience and allowed us to understand our product better at the same time.

This first trial by fire helped us establish a broad culture of experimentation and data-driven decision-making, as well as learn to break down big changes into individually testable components.

2014: Pyapns

We rely on an open-source library called pyapns for sending push notifications via Apple’s servers. The project was written several years ago and wasn’t well maintained. Based on our data and what we’d heard from other companies, we had concerns about its reliability. We decided to test out using a different library called PyAPNs, which seemed better written and better maintained. We set up an A/B experiment, monitored the results and found that there was a 1 percent decrease in our visitors with PyAPNs. We did some digging and couldn’t determine the cause for the drop, so we eventually decided to roll back and stick with pyapns.

2015: UserService

We’ve slowly been moving towards a more service-oriented architecture. Recently we extracted a lot of our code for managing users and encapsulated it into our new UserService. We took an iterative approach to building the service, extracting one piece of functionality at a time. With such a major refactor of how we handle all user-related data, we wanted to ensure nothing broke. We set up an experiment for each major piece of functionality that was extracted, for a total of three experiments. Each experiment completed successfully showing no drop in any metrics. The results have given us strong confidence that this new UserService is at parity with the previous code.

We’ve had a lot of success with A/B testing our infrastructure. It’s helped us identify when changes have caused a serious negative impact that we probably wouldn’t have noticed. When they go well, they also give us the confidence that a new system is performing as expected. If you’re not A/B testing your infrastructure changes, you really should be.

John Egan is a growth engineer and Andrea Burbank is a data scientist

Acknowledgements: Dan Feng, Josh Inkenbrandt, Nadine Harik and Vy Phan for their help in running the experiments covered in this post.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

Pinterest hosts more than 30 billion Pins (and growing) with rich contextual and visual information. Tens of millions of Pinners (users) interact with the site every day by browsing, searching, Pinning, and clicking through to external sites. The home feed, a collection of Pins from the people, boards and interests followed, as well as recommendations including Picked for You, is the most heavily user-engaged part of the service, and contributes a large fraction of total repins. The more people Pin, the better Pinterest can get for each person, which puts us in a unique position to serve up inspiration as a discovery engine on an ongoing basis.

The home feed is a key way to discover new content, which is valuable to the Pinner, but poses a challenging question. Given the ever increasing number of Pins from various sources, how can we surface the most personalized and relevant Pins? Our answer is Pinnability.

Pinnability is the collective name of the machine learning models we developed to help Pinners find the best content in their home feed. It's part of the technology powered by smart feed, which we introduced last August, and estimates the relevance score of how likely a Pinner will interact with a Pin. With accurate predictions, we prioritize those Pins with high relevance scores and show them at the top of home feed.

The benefits of Pinnability

Before launching Pinnability a few months ago, all home feed content from each source (e.g., following and Picked For You) was arranged chronologically, without taking into account which Pins people may find to be most interesting. In other words, a newer Pin from the same source always appeared before an older Pin. This simple rule is easy to understand and implement, but it lacked the ability to effectively help Pinners discover Pins that really interest them, because a low-relevance Pin could very well appear before a high-relevance one (see Figure 1).

With Pinnability launched, the candidate Pins for home feed are scored using the Pinnability models. The scores represent the personalized relevance between a Pinner and the candidate Pins. Pins in home feed are prioritized by the relevance score as illustrated in Figure 2.

Powering Pinnability with machine learning

In order to accurately predict how likely a Pinner will interact with a Pin, we applied state-of-the-art machine learning models including Logistic Regression (LR), Support Vector Machines (SVM), Gradient Boosted Decision Trees (GBDT) and Convolutional Neural Networks (CNN). We extracted and tested thousands of textual and visual features that are useful for accurate prediction of the relevance score. Before we launch a model for an online A/B experiment, we thoroughly evaluate its offline performance based on historical data.

Figure 3 summarizes the three major components of our Pinnability workflow, namely training instance generation, model generation and home feed serving.

Training instance generation

The basis of the Pinnability training data is the historical Pinner interaction with home feed Pins. For example, after viewing a Pin in home feed, a Pinner may choose to like, repin, click for a Pin closeup, clickthrough, comment, hide, or do nothing. We record some of the "positive actions" and "negative action" as training instances. Naturally the number of Pins viewed is often much larger than the number of Pins in which the Pinner made a positive action, so we sample the positive and negative instances at different rates. With these defined, we test thousands of informative features to improve Pinnability’s prediction accuracy.

Our unique data set contains abundant human-curated content, so that Pin, board and user dynamics provide informative features for accurate Pinnability prediction. These features fall into three general categories: Pin features, Pinner features and interaction features:

Pin features capture the intrinsic quality of a Pin, such as historical popularity, Pin freshness and likelihood of spam. Visual features from Convolutional Neural Networks (CNN) are also included.

Pinner features are about the particulars of a user, such as how active the Pinner is, gender and board status.

Interaction features represent the Pinner’s past interaction with Pins of a similar type.

Some features are subject to transformation and normalization. For instance, log transformation is applied to many “count features” such as the number of Pins a Pinner owns for regression-friendly distributions.

The major challenge we faced in developing a robust training data generation pipeline was how to cope with the large data scale. We built MapReduce pipelines to generate the training instances, each representing a Pinner/Pin interaction. A training instance contains three parts of information:

Meta data (Pin ID, Pinner ID, source of the interaction, timestamp, etc.) for data grouping when we want to train and analyze a Pinnability model with a subset of training instances, such as following and Picked For You (PFY) models.

Target value to indicate whether a Pinner has taken a positive action after viewing the Pin. We can train separate models that optimize different positive actions such as repins and clickthroughs.

Feature vector that contains the informative signals for interaction prediction.

Pinnability model generation

In training Pinnability models, we use Area Under the ROC Curve (AUC) as our main offline evaluation metric, along with r-square and root mean squared error. We optimized for AUC not only because it is a widely used metric in similar prediction systems, but also because we’ve observed strong positive correlation between the AUC gain from offline testing and an increase in Pinner engagement in online A/B experiment. Our production Pinnability model achieves an AUC score averaging around 90 percent in home feed.

We experimented with multiple machine learning models, including LR, GBDT, SVM and CNN, and we use AUC score in 10-fold cross-validation and 90/10 train-test split settings with proper model parameters for evaluation. We observed that given a fixed feature set, the winning model always tends to be either LR or GBDT for Pinnability. For online A/B experimentation, we prioritize models based on offline AUC scores.

Among the thousands of features we added to the training instances, we select features that significantly increase our offline AUC metric as candidates for online A/B experiments. Given the large amount of candidate features, we often test new features in smaller groups, such as recency, Pin owner quality and category match features. The A/B experiments we conducted compare Pinner engagement between the control group using production features and the treatment group using the new experimental features. If the results are positive, we evaluate the extra data size and latency impact on our servers before adding the new features to our production Pinnability models. We iterate quickly on new features supported by our robust training instance generation, model training and evaluation pipelines. In order to monitor models’ on-going performance, we keep a small holdout user group that is not exposed to the Pinnability models. Comparing the engagement difference between the holdout and enabled groups provides valuable insights about Pinnability’s long-term performance.

Currently we only use offline batch data to train our Pinnability models. This poses a potential issue in that we’re not utilizing the most recent data to dynamically adjust model coefficients in serving. On the other hand, we tested and confirmed that the model coefficients do not change substantially when trained on different batches of data separated by several days, so the benefits of online model adjustment are subject to further evaluation.

We’re also exploring ways to apply online training with real-time instances to augment our offline training pipeline so our models are calibrated immediately after we gather new home feed activity data. Online training poses new challenges both algorithmically to our machine learning pipeline and systematically to our home feed serving framework.

Home feed serving

Home feed is powered by our in-house smart feed infrastructure. When a new Pin is repinned, smart feed worker sends a request to the Pinnability servers for the relevance scores between the repinned Pin and all the people following the repinning Pinner or board. It then inserts the Pins with the scores to the pool that contains all followed Pins. PFY Pins are inserted into the PFY pool with the Pinnability relevance score in a similar fashion.

When a user logs on or refreshes home feed, smart feed content generator materializes the new content from the various pools while respecting the relevance scores within each pool, and the smart feed service renders the Pinner’s home feed that prioritizes the relevance scores.

Pinnability outcome

We continue to refine Pinnability and have released several improvements to-date. With each iteration, we’ve observed significant boosts in Pinner engagement, including an increase in the the home feed repinner count by more than 20 percent. We’ve also observed significant gains in other metrics including total repins and clickthroughs.

Given the importance of home feed and the boost in Pinner engagement, Pinnability continues to be a core project in building our discovery engine. We’ve also begun to expand the use of our Pinnability models to help improve our other products outside home feed.

We’re always looking for bright engineers to join Pinterest to help us solve impactful problems like Pinnability.

Yunsong Guo is a software engineer on the Recommendations team Acknowledgements: Pinnability is a long-term strategic project being developed in collaboration with Mukund Narasimhan, Chris Pinchak, Yuchen Liu, Dmitry Chechik and Hui Xu. This team, as well as people across the company, helped make this project a reality with their technical insights and invaluable feedback.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

Pinner safety is a top priority for us, and so earlier this year we joined the growing list of websites that are fully HTTPS. As we build trust with Pinners, it significantly improves security in one fell swoop. Migrating to HTTPS presented a number of expected - and unexpected - engineering challenges, starting with finding the right CDN provider partner.

Expected challenges

We identified and mitigated many technical challenges in the discovery process of the migration. One of the biggest challenges was working with our CDN providers, which support HTTPS and our certificates. We also knew that CDN image distribution over HTTPS can potentially cost significantly more. Other technical challenges included:

Surfacing hard coded HTTP URLs and functions in source files

Performance impact

Older browser support

Referral header removal from HTTPS to HTTP sites

Mixed content warnings (broken lock in browser bar)

Unexpected challenges

Once we felt comfortable enough to start testing, we launched a test in the UK where we have an active Pinner community. Our tests showed there was an insignificant impact on SEO and little effect on any one browser. It wasn’t until we cast a larger net to a percentage of our global audience did we see the following other challenges:

Missed CDN content that broke the Pin It button for several hours

Not all sitemap files were updated to point to HTTPS domains

An unknown Safari issue

Tackling challenges

Although we anticipated a number of challenges, we were able to tackle those unexpected ones with a lean and fast-moving team. Here’s how we did it:

Broken “Pin It” button. We were able to quickly mitigate by a swift DNS change to a new CDN provider.

Referral header remover issues. We used a meta referrer header to support HTTPS tracking to HTTP sites.

Unknown Safari issue. Our UK experiment provided data that showed a small percentage of users had problems logging in after the migration. We pinpointed this to Safari users, which allowed us to start investigating the root cause.

In addition, having multiple CDN providers that supported HTTPS gave us options for performance as well as commercial leverage.

In the end, we enhanced the privacy of Pinners by enabling encryption while also hindering exploitation by way of man-in-the-middle attacks, session hijacking, content injection, etc. This also paved the way for future products that may require HTTPS to launch. Finally, the move to HTTPS resulted in a 10 percent (max) increase in signups a day, because we were able to remove the redirect flow from HTTP to the HTTPS signup page.

We will continue our journey towards HTTPS with further enhancements including HTTP Strict Transport Security (HSTS), which will prevent SSL stripping. We also plan to work with Chromium to preload our domain to prevent SSL stripping on a user’s first visit to Pinterest.

Introducing our paid bug bounty program

Prior to the HTTPS migration, we were hesitant to open a paid bug bounty program because of a number of known vulnerabilities associated with being only HTTP. Now that a number of gaps have been closed as a result of the migration, we’re happy to announce that we’ve upgraded the program with payouts results, with a 10x increase in reports since launching the paid program. We highly encourage the whitehat hacker community to use our program and report bugs, which helps us keep Pinners safe and increase our security posture.

If you’re interested in working on security engineering challenges like this, join our team!

Paul Moreno is the security engineering lead on the Cloud team.

Migrating to HTTPS wasn’t a smooth process. It took several members of various teams to pull off, and there were a number of moving parts. Special thanks to engineers Amine Kamel, Chris Danford, Danilo Stefanovic and Anna Majkowska for their hard work making Pinterest a safer place.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

As we continue to build in a fast and dynamic environment, we need a workflow manager that’s flexible and can keep up with our data processing needs. After trying a few options, we decided to build one in-house. Today we’re open-sourcing Pinball, which is designed to accommodate the needs of a wide range of data processing pipelines composed of jobs ranging from simple shell scripts to elaborate Hadoop workloads. Pinball joins our other open-sourced projects like Secor and Bender, available on our Github.

Building highly customizable workflow manager

Development of new product features is often constrained by the availability of data that powers them. The raw data, typically originating from logs, is sliced and diced, merged and filtered across multiple dimensions before it reaches the shape suitable for ingestion by downstream applications. The process of data transformation is often modelled as a workflow, which, in abstract terms, are directed graphs of nodes representing processing steps and edges describing the run-after dependencies.

Workflows can be arbitrarily complex. In realistic settings it’s not uncommon to encounter a workflow composed of hundreds of nodes. Building, running and maintaining workflows of that complexity requires specialized tools. A Bash script won’t do.

After experimenting with a few open-source workflow managers we found none of them to be flexible enough to accommodate the ever-changing landscape of our data processing solutions. In particular, current available solutions are either scoped to support a specific type of job (e.g. Apache Oozie optimized for Hadoop computations) or abstractly broad and hard to extend (e.g. monolithic Azkaban). With that in mind, we took on the challenge of implementing a highly customizable workflow manager build to survive the evolution of the data processing use cases ranging from execution of basic shell commands to elaborate ETL-style computations on top of Hadoop, Hive and Spark.

Pinball is used by all of our engineering teams, and handles hundreds of workflows with thousands of jobs that process almost three petabytes of data on a daily basis in our Hadoop clusters. The largest workflow has more than 500 jobs. Workflows generate analytics reports, build search indices, train ML models and perform a myriad of other tasks.

Platform vs. end product

Pinball offers a complete end-to-end solution ready to use right out of the box. At the same time, its component-wise design allows for easy alterations. The workflow management layer of Pinball is built on top of generic abstractions implementing atomic state updates.

Conceptually, Pinball’s architecture follows a master-worker (or master-client to avoid naming confusion with a special type of client that we introduce below) paradigm where the stateful central master acts as a source of truth about the current system state to stateless clients. Clients come in different flavors ranging from workers taking care of job execution, to scheduler controlling when a workflow should run and UI allowing users to interact with the system, to command line tools. All clients talk the same language (protocol) exposed by the master, and they don’t communicate directly. Consequently, clients are independent and can be easily replaced with alternative implementations. Because of the flexibility of Pinball’s design, it’s a platform for building customized workflow management solutions.

While customization is possible, it’s worth emphasizing that Pinball comes with default implementation of clients allowing users to define, run and monitor workflows.

Workflow life cycle

Workflow is defined through a configuration file or a UI workflow builder, or even imported from another workflow management system. Pinball offers a pluggable parser concept allowing users to express their workflows in a format that makes most sense to them. Parser translates an opaque workflow definition to a collection of tokens representing workflow jobs in a format understandable by Pinball. (Read more about Pinball’s features.)

Workflow gets deployed through a command line tool or a UI component. Deployment invokes the parser to extract schedule token from the workflow configuration and stores it in the master. A schedule token contains metadata such as the location of the workflow config, the time at which the workflow should run, the recurrence of executions and an overrun policy. The policy describes how the system should behave if the previous run of the workflow hasn’t finished by the time a new execution is due. Example policies allow aborting the currently running workflow, starting another workflow instance in parallel to the running one or delaying the workflow start until the previous run finishes.

When the time comes, the scheduler uses the information stored in the schedule token to locate the workflow config, parse it and generate job tokens representing individual jobs in that workflow. Job tokens are posted to the master under a unique workflow instance ID. Workflow instances are controlled independently of one another giving the user the flexibility to run multiple instances of the same workflow in parallel.

Job tokens are claimed and executed by idle workers. A job is described by a command line that the worker runs in a subprocess. The output of the subprocess is captured and exposed in the UI. Pinball interprets specially formatted log lines as values to be exposed in the UI or passed on to downstream jobs. This allows us to directly embed a link to a Hadoop job tracker page in the Pinball UI or propagate parameters from an upstream job to its downstream dependents.

If any post-processing is needed on job failure (e.g. one may want to remove partial output), Pinball offers the ability to attach arbitrary cleanup command to job configuration. Cleanups are guaranteed to run even if the worker that claimed the job died in the middle of its execution.

Failed jobs may be retried automatically or manually. Users can choose any subset of jobs to retry by interacting with the workflow diagram. Bulking of actions significantly improves the usability when working with larger job hierarchies.

When the workflow instance finishes (either failing or succeeding), optional email notifications are sent out to workflow owners.

Workflow configuration and job templates

To end users, workflow manager is often a black box that employs faerie magic to schedule and execute their jobs, but the workflow itself needs to be defined in one way or another. While designing Pinball, we made a conscious choice to not make the configuration syntax part of the system core in order to give developers a lot of flexibility to define workflow configurations in a way that makes the most sense in a given setting. At the same time, we wanted to offer a complete package with a low barrier to entry. So, we decided to include a simplified version of the parser and job templates that we use in our open-source release.

Out of the box, we support a Python-based workflow configuration syntax. We also provide a number of job templates for configuring simple shell scripts as well as more elaborate computations on the Hadoop platform. We offer a native support for EMR and Qubole platforms with some power features such as embedding of job links in the Pinball UI and cleaning up resources after failed jobs. We also propose the notion of a condition that allows users to model data dependencies between jobs (think of a job being delayed until the data it needs becomes available).

Give our projects a try and let us know what you think! If you’re interested in working on projects like this from the inside, join our team.

Pawel Garbacki is a software engineer on the Monetization team. Mao Ye, Changshu Liu and Jooseong Kim are software engineers on the Data team.

Acknowledgements: Thanks to Krishna Gade, Mohammad Shahangian, Tongbo Huang, Julia Oh and Roger Wang for their invaluable contributions to the Pinball project.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

We have a lot of important and common data that’s not modified frequently but accessed at a very high rate. One example is our spam domain blacklist. Since we don’t want to show Pinners spammy Pins, our app/API server needs to check a Pin's domain against this domain blacklist when rendering the Pin. This is just one example, but there are hundreds of thousands of Pin requests every second, which generates enormous demand for access to this list.

Existing Problem

Previously, we stored this kind of list in a Redis sorted set which provided us with easy access to keep the list structure in a time sorted order. We also have a local in-memory and file-based cache that’s kept in sync via polling the Redis host for any updates. Things went well in the beginning, but as the number of servers and size of the list grew, we began to see a network saturation problem. In the five minutes after the list was updated, all the servers tried to download the latest copy of data from a single Redis master causing the network to saturate on Redis master and resulting in a lot of Redis connection errors.

We identified a few potential solutions:

Spread the download of this data over a longer period. But for our use case, we wanted the updates to converge within a few minutes at most.

Shard the data. Unfortunately, since this data is a single list, sharding it would add more complexity.

Replicate this data. Use a single Redis master with multiple Redis slaves to store this data and randomly pick a slave for reads. However, we weren't confident about Redis replication (we were running v2.6). Moreover, it wouldn’t be cost effective since most of the time (when the data is not updated) these Redis boxes will be idle due to client side caching.

Solution

As each of the above solutions has its own shortcoming, we asked ourselves, how would we design a solution if we were building from ground up?

Formalizing the requirements of the problem:

Frequent read access (>100k/sec) and rare updates (several times a day, at most).

Quick (within one minute, or several at most) converge the updates across all boxes. Ideally, a push-based model instead of clients polling for updates.

We engineered a solution by combining the solutions to the smaller problems:

Cache the data in-memory so that high read access won’t be a problem.

Use Apache ZooKeeper as a notifier when updates are made.

This is conceptually similar to the design of ZooKeeper resiliency, but if we stored the entire data in one ZooKeeper node, it would still cause a huge spike in network traffic on the ZooKeeper nodes during an update. Since ZooKeeper is distributed, the load would be spread across multiple ZooKeeper nodes. Yet, we didn’t want to burden ZooKeeper unnecessarily as it’s a critical piece of our infrastructure.

We finally arrived at a solution where we use ZooKeeper as the notifier and S3 for the storage. Since S3 provides very high availability and throughput, it seemed to be a good fit for our use case in absorbing the sudden load spikes. We call this solution managed list aka config v2.

Config v2 at work

Config v2 takes full advantage of the work we have already done, except that the source of truth is in S3. Further, we added logic to avoid concurrent updates and to deal with S3 eventual consistency. We store a version number (that’s actually a timestamp) in ZooKeeper node which also will be a suffix of the S3 path to identify the current data.

If a managed list’s data needs to be modified, a developer has the option to change it via an admin web UI or a console app. The following steps are executed by the Updater app on save:

First, grab a Zk lock to prevent concurrent write to the same managed list.

Then, compare the old data with the one in S3 and only upload the new data to S3 if it matches - Compare And Swap update. This prevents dirty writes while a previous update is converging.

Finally, write the version to Zk node and release the Zk lock.

As soon as the Zk node’s value is updated, ZooKeeper notifies all its watchers. In this case, triggering the Daemon processes on all servers to download the data from S3.

How we grappled with S3’s consistency model

Amazon’s S3 gives great availability and durability guarantees even under heavy load, but it’s eventually consistent. What we needed was “read after write” consistency. Fortunately, it does give “read after create” consistency in some regions*. Instead of updating the same S3 file, we create a new file for every write. And yet, this introduces a new problem of synchronizing the new S3 filename across all the nodes. We solved this problem by using ZooKeeper to keep the filename in sync across all the nodes.

Introducing Decider

When a new feature or service is ready for launch, we gradually ramp up traffic in the new code path and check to make sure everything is good before going all in. This resulted in the need to build a switch that can allow a developer to decide how much traffic should be sent to the new feature. Also, this traffic ramp-up tool (aka “Decider”) should be flexible enough so that developers can add new experiments and change the values of existing experiments without requiring a re-deploy to the entire fleet. In addition, any changes should converge quickly and reliably across the fleet.

Earlier Solution

Every experiment is a ZooKeeper node and has a value [0-100] that can be controlled from the web UI. When the value is changed from the web UI, it’s updated in the corresponding node, and ZooKeeper takes care of updating all the watchers. While this solution worked, it was plagued with the same scaling issues we previously experienced since the entire fleet was directly connecting to ZooKeeper.

Our Decider framework consisted of two components: a web-based admin UI to control the experiments and a library (both in Python and Java) that can be plugged in where branch control is needed.

Current Solution

Once we realized the gains of managed list, we built managed hashmap and migrated values of all Zk nodes containing the experiments. Essentially, the underlying managed hashmap file content is a json dump of the hash table that contains experiment names as the keys and an integer [0-100] as the value.

API

def decide_expermiment(experiment_name): return random.randrange(0, 100, 1)

How this is used in code:

if decide_experiment("my_rocking_experiment"): // new code else // existing code

Another use case of Decider: dark read and dark write

We use the terminology “dark read” and “dark write” when we duplicate the production read or write request and send it to a new service. We call it dark because the response from the new service doesn’t impact the original code path whether it’s a success or failure. If asynchronous behavior is needed then we wrap the the new code path in gevent.spawn().

Here’s a code snippet for dark read:

try:

if decider.decide_experiment("dark_read_for_new_service"): new_service.foo() except Exception as e: log.info("new_service.foo exception: %s" % e)

*In the rare event that S3 returns “file not found” due to eventually consistency, the daemon is designed to refresh all the content every 30 mins, and those nodes will eventually catch up. So far, we haven’t seen any instances where the nodes got out of sync for more than a few minutes.

If you’re interested in working on engineering challenges like this, join our team!

Pavan Chitumalla and Jiacheng Hong are software engineers on the Infrastructure team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.