Richard Nelson

By James Collins, Sr. Director, Developer Platforms and Services, and St. John Johnson, Principal Engineer

Continuous Delivery enables software development teams to move faster and adapt to users’ needs quicker by reducing the inherent friction associated with releasing software changes. Yahoo’s engineering has modernized as it has embraced Continuous Delivery as a strategy for improving product quality and engineering agility. All our active products deliver from commit to production with full automation and this has greatly improved Yahoo’s ability to deliver products.  

Part of what enabled Yahoo to make Continuous Delivery at scale a reality was our improved build and release tooling. Now, we are open sourcing an adaptation of our code as Screwdriver.cd, a new streamlined build system designed to enable Continuous Delivery to production at scale for dynamic infrastructure.

Some of the key design features of Screwdriver have helped Yahoo achieve Continuous Delivery at scale. At a high level these are:

Making deployment pipelines easy

Optimizing for trunk development

Making rolling back easy

Easy deployment pipelines: Deployment pipelines that continuously test, integrate, and deploy code to production greatly reduce the risk of errors and reduce the time to get feedback to developers. The challenge for many groups had been that pipelines were cumbersome to setup and maintain. We designed a solution that made pipelines easy to configure and completely self-service for any developer. By managing the pipeline configuration in the code repository Screwdriver allows developers to configure pipelines in a manner familiar to them, and as a bonus, to easily code review pipeline changes too.

Trunk development: Internally, we encourage workflows where the trunk is always shippable. Our teams use a modified GitHub flow for their workflows. Pull Requests (PRs) are the entry point for running tests and ensuring code that entered the repository has been sufficiently tested. Insisting on formal PRs also improves the quality of our code reviews.

To ensure trunks are shippable, we enable functional testing of code in the PRs. Internally, this is a configuration baked into pipelines that dynamically allocates compute resources, deploys the code, and runs tests. These tests include web testing using tools like Selenium. These dynamically-allocated resources are also available for a period after the PR build, allowing engineers to interact with the system and review visual aspects of their changes.

Easy rollbacks: To allow for easy code rollbacks, we allow phases of the pipeline to be re-run at a previously-saved state. We leverage features in our PaaS to handle the deployment, but we store and pass metadata to enable us to re-run from a specific git SHA with the same deployment data. This allows us to roll back to a previous state in production. This design makes rolling back as easy as selecting a version from a dropdown menu and clicking “deploy.” Anyone with write access to the project can make this change. This helped us move teams to a DevOps model where developers were responsible for the production state.

The successful growth of Screwdriver over the past 5 years at Yahoo has today led to Screwdriver being synonymous with Continuous Delivery within the company. Screwdriver handles over 25,000+ builds per day and 12,000+ daily git commits as a single shared entrypoint for Yahoo. It supports multiple languages and handles both virtual machine and container-based builds and deployment.

Screwdriver.cd’s architecture is comprised of four main components: a frontend for serving content to the user, a stateless API that orchestrates between user interactions and build operations, the execution engines (Docker Swarm, Kubernetes, etc.) that checkout source code and execute in containers, and the launcher that executes and monitors commands inside the container.

The diagram below shows this architecture overlaid with a typical developer flow.

To give some context around our execution engines, internal Screwdriver started as an abstraction layer on top of Jenkins and used Docker to provide isolation, common build containers, etc. We used features provided by Jenkins plugins to leverage existing work around coverage and test reports. However, as Screwdriver usage continued to climb, it outgrew a single Jenkins cluster. So in order to grow to our needs, we added capabilities in Screwdriver that allowed us to scale horizontally while also adding capabilities to schedule pipelines across a number of Jenkins clusters. As we scaled Screwdriver, we used less from Jenkins and built more supporting services utilizing our cloud infrastructure. The open-source version is focused on Kubernetes and Docker Swarm as our primary supported execution engines.

In the coming months we will expand our offering to match many of the features we have internally, including:

Mechanism to store structured build data for later use (last deployed version, test coverage, etc.)

Built-in metric collecting

System-wide templates to enable getting started quickly

Log analysis to provide insights to developers

In a letter today to James Clapper, Director of National Intelligence (DNI), Yahoo is formally urging that the U.S. government provide its citizens with clarification around national security orders they issue to internet companies to obtain user data.

While the letter makes specific reference to recent allegations against Yahoo, it is intended to set a stronger precedent of transparency for our users and all citizens who could be affected by government requests for user data. As we’ve said before, recent press reports have been misleading; the mail scanning described in the article does not exist on our systems.

By Joe Francis and Matteo Merli, Yahoo Platforms

Pub-sub messaging is a very common design pattern that is increasingly found in distributed systems powering Internet applications. These applications provide real-time services, and need publish-latencies of 5ms on average and no more than 15ms at the 99th percentile. At Internet scale, these applications require a messaging system with ordering, strong durability, and delivery guarantees. In order to handle the “five 9’s” durability requirements of a production environment, the messages have to be committed on multiple disks or nodes.

At the time we started, we could not find any existing open-source messaging solution that could provide the scale, performance, and features Yahoo required to provide messaging as a hosted service, supporting a million topics. So we set out to build Pulsar as a general messaging solution, that also addresses these specific requirements.

Pulsar is a highly scalable, low latency pub-sub messaging system running on commodity hardware. It provides simple pub-sub messaging semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication.

Using Pulsar, one can set up a centrally-managed cluster to provide pub-sub messaging as a service; applications can be onboarded as tenants. Pulsar is horizontally scalable; the number of topics, messages processed, throughput, and storage capacity can be expanded by adding servers to the pool.

Pulsar has a robust set of APIs to manage the service, namely, account management activities like  provisioning users, allocating capacity, accounting usage, and monitoring the service. Tenants can administer, manage, and monitor their own domains via APIs. Pulsar also provides security via a pluggable authentication scheme, and access control features that let tenants manage access to their data.

Application development using Pulsar is easy due to the simple messaging model and API. Pulsar includes a client library that encapsulates the messaging protocol; complex functions like service discovery, as well as connection establishment and recovery, are handled internally by the library.

Architecture

At a high level, a Pulsar instance is composed of multiple clusters, typically residing in different geographical regions. A Pulsar cluster is composed of a set of Brokers and BookKeepers (bookies), plus ZooKeeper ensembles for coordination and configuration management.

A Pulsar broker serves topics. Each topic is assigned to a broker, and a broker serves thousands of topics. The broker accepts messages from writers, commits them to a durable store, and dispatches them to readers. The broker also serves admin requests. It has no durable state. The broker has built-in optimizations; for example, it caches the data in order to avoid additional disk reads when dispatching messages to clients as well as replication clusters. Pulsar brokers also manage the replicators, which asynchronously push messages published in the local cluster to remote clusters.

Apache BookKeeper is the building block for Pulsar’s durable storage. BookKeeper is a distributed write-ahead log system, a top-level Apache project that was originally developed at and open-sourced by Yahoo in 2011. BookKeeper has an active developer community with contributors across the industry. Using the BookKeeper built-in semantics, Pulsar creates multiple independent logs, called ledgers, and uses them for durable message storage. Bookkeeper hosts, called bookies, are designed to handle thousands of ledgers with concurrent reads and writes. BookKeeper is horizontally scalable in capacity and throughput; from an operational perspective we can elastically add more bookies to a Pulsar cluster to increase capacity.

By using separate physical disks (one for journal and another for general storage), bookies are able to isolate the effects of read operations from impacting the latency of ongoing write operations, and vice-versa. Since read and write paths are decoupled, spikes in reads – which commonly occur when readers drain backlog to catch up – do not impact publish latencies in Pulsar. This sets Pulsar apart from other commonly-used messaging systems.

Managed Ledger represents the storage layer for a single topic. It is the abstraction of a stream of messages, with a single writer, and multiple readers, each with its own associated cursor position, the offset of the reader in the message stream. A single managed ledger uses multiple BookKeeper ledgers to store the data. Cursor positions are maintained in per-cursor ledgers.

A Pulsar cluster runs a ZooKeeper (another top-level Apache project open-sourced by Yahoo in 2008) ensemble used for coordinating assignment of topics among brokers, and storing BookKeeper metadata. In addition, Pulsar runs a Global ZooKeeper ensemble to store the provisioning and configuration data. At Yahoo, we have presence in multiple regions and our users create global topics that are replicated between these regions. The Global Zookeeper ensemble keeps provisioning and configuration data consistent globally. We can tolerate higher write latencies on these writes (e.g.: ~150ms latency for configuration writes).

The load balancer is a distributed service that runs on the brokers, to make sure the traffic is equally spread across all available brokers. Since Pulsar brokers have no durable state, topics can be redistributed within seconds.

Messaging Model

The Pulsar topic is the core of the system; applications and components communicate by publishing to and consuming from the same topic. Topics are created dynamically as needed when a producer (writer) starts publishing on it; and topics are removed when not in use.

Subscriptions are created automatically when a consumer (reader) subscribes to the topic. A subscription persists until it is deleted, and receives all messages published during its lifetime. Common messaging semantics (like JMS Topic or Queue) are available as subscription modes; an exclusive subscription is equivalent to a “topic,” and a shared subscription is equivalent to a “queue.”

Performance

Pulsar is designed for low-publish latencies at scale. Our typical publish latencies on average are well below 5ms. With SSD as the bookie journal device, Pulsar can achieve 99 percentile latencies of 5ms with two guaranteed copies and total ordering.

The latency remains within the acceptable range until the throughput reaches the limit of the disk IO capacity.

Pulsar supports partitioned topics, which can further increase the per-topic throughput.

Pulsar at Yahoo

Pulsar backs major Yahoo applications like Mail, Finance, Sports, Gemini Ads, and Sherpa, Yahoo’s distributed key-value service.

We deployed our first Pulsar instance in Q2 2015. Pulsar use has rapidly grown since then, and as of today, Yahoo runs Pulsar at scale.

Deployed globally, in 10+ data-centers, with full mesh replication capability

Greater than 100 billion messages/day published  

More than 1.4 million topics

Average publish latency across the service of less than 5 ms

As Pulsar use grows at Yahoo, we have been scaling the service horizontally. Most of the challenges we faced were with JVM GC impacting publish latencies, and reducing failover times when the number of topics on a broker went up to tens of 1000s (now 40,000). This led to significant changes to the Pulsar broker and to BookKeeper.

Looking to the Future

We are actively engaged in pushing the scale and reliability boundaries of Pulsar further. Current improvements being worked on include:

Migrate topic between brokers in under 1 sec, from 10 sec

Improve 99.9%ile publish latencies to 5ms

Provide additional language bindings for Pulsar

Conclusion

Pulsar is a highly scalable pub-sub messaging system, production-ready and battled tested at Yahoo. We are glad to make Pulsar available as open source under Apache License Version 2.0. Detailed instructions and documentation are available at Yahoo’s Github repository. Our goal is to make Pulsar widely used and well integrated with other large-scale open source software, and we welcome contributions from the community to make that happen.

Additional Links

Pulsar Github project page: https://github.com/yahoo/pulsar

Earlier this year when Apple shared our PLUS7 app as an example of excellence to a global audience in their keynote address, it was the culmination of an engaging journey for the engineering team.

When Apple announced the 4th generation Apple TV, we knew it would be a perfect fit for expanding our PLUS7 offering. The Apple TV’s simple yet elegant experience is great for designing in a way that allows users to get straight to the content they want.

Building upon our knowledge of iOS and existing Apple frameworks, the process to develop a tvOS app was fast and really enabled our engineers to push the envelope in terms of redefining the PLUS7 viewing experience on a larger screen.

Swift

With Swift still evolving fairly heavily, choosing to go with it is not without some risk and overhead. On evaluation, we decided that it was mature enough to use in a production application and that it would be the way forward for us. Although we could have reused Objective-C code from our iOS implementation, we further decided to start from scratch and our own code in the tvOS build is 100% Swift. We now believe this decision was a good one; even our test builds were remarkably stable and there are much fewer lines of code than its iOS counterpart. Additionally, we can now use our arguably better Swift code in future versions of our iOS releases.

The Build

We initially had a prototype version of the app which was built in spare time and had no input from the design team. Ultimately, it was this version that was iterated on as the build progressed. One of the main issues we encountered was 3rd party SDKs not being tvOS or bitcode compatible. With a bit of prodding, we were able to resolve most of these, with the large exception of the SDK we use for video advertising.

Advertising

On iOS and Android we use Google’s IMA SDK for video advertising, which is not compatible with tvOS at this time. Obviously, launching PLUS7 for Apple TV without video advertising is not a viable strategy for our business. So we decided to implement the VAST spec ourselves, in Swift. We had a look at some (old) open source VAST implementations which unfortunately turned out not to be suitable. At first glance a VAST implementation looks fairly straightforward.

However, it turns out that it’s much more complicated than at first glance. A video contains “cuepoints”, which indicate when advertising slots are to be played. There are three types of ad sets; pre-rolls (played before a stream), mid-rolls (several per video, in the middle of the stream) and post-rolls (played after the stream). Each cuepoint maps to a “pod” of ads. A pod is made up of a number of ads. So the first preroll might contain one 15 second ad. The first mid-roll pod may contain 4 ads of various lengths (between 15 and 60 seconds), the next one 5 ads of various lengths and so on.

When the video starts, we make one query which returns each element in each pod. However, each ad is a recursive ad call to an ad server which you have to keep querying until you get an actual creative to play. The first ad server may not have the content, but will defer to another ad server, and so on. Additionally, each ad server that is traversed through will add its own tracking URLs for each event that may occur during playback, for example each quartile that is played, play, pause, stop etc events. Further, it’s possible that following a particular path down ad servers may result in an infinite loop (!), or a playback error, in which case specific rules apply.

It is vital that the implementation of the VAST specification behaves correctly, as our business revenue depends on it. Most of the time spent on development for our 1.0 release went into this effort.

Video player

Playing content along with video ads (which are separate video streams) while providing a seamless experience to the users is not that straightforward. Each video ad is fetched ahead of time, stored in a list and passed to the video controller provided by tvOS API. The content is paused and resumed after each ad sequence in a different controller. Seeking forward over one or more “cuepoints” means that we have to play the last set of ads in a pod and so on: the user might get to the post rolls too if they are patient enough.

As much as we can, we fetch required data ahead of time. As soon as we have enough information to play a preroll, we do, and then go and do the rest of the requests for specific ads so that users aren’t waiting for this to happen every time an ad has to play.

UI

The Apple TV experience has some unique features due to the nature of the remote. Our first iteration had multiple tabs of grid views, and a master/detail view. It worked, but had the obvious drawback of having to excessively scroll vertically to simply navigate around. After feedback (and using the app ourselves), we simplified the navigation dramatically and created multiple horizontal scrolling collection views on the main screen, and a “Search” screen for quickly filtering for known content.

Result:

Furthermore, we were honoured to be featured on the screen during Tim Cook’s recent keynote: