Nine Fives

> When this happened it caused a great deal of churn within the network as all of our aggregated links had to be re-established, leader election for spanning-tree had to take place, and all of the links in the network had to go through a spanning-tree reconvergence. This effectively caused all traffic between access switches to be blocked for roughly a minute and a half. > > [...] When the network froze, many of our fileservers which are intentionally located in different racks for redundancy, exceeded their heartbeat timeouts and decided that they needed to take control of the fileserver resources. They issued STONITH commands to their partner nodes and attempted to take control of resources, however some of those commands were not delivered due to the compromised network. When the network recovered and the cluster messaging between nodes came back, a number of pairs were in a state where both nodes expected to be active for the same resource. This resulted in a race where the nodes terminated one another and we wound up with both nodes stopped for a number of our fileserver pairs.

> In the course of our troubleshooting we discovered that our aggregation switches were missing a number of MAC addresses from their tables, and thus were flooding any traffic that was sent to those devices across all of their ports. Because of these missing addresses, a large percentage of our traffic was being sent to every access switch and not just the switch that the destination devices was connected to. During normal operation, the switch should "learn" which port each MAC address is connected through as it processes traffic. For some reason, our switches were unable to learn a significant percentage of our MAC addresses and this aggregate traffic was enough to saturate all of the links between the access and aggregation switches, causing the poor performance we saw throughout the day. > > [...] > > We need to be more mindful of tunnel-vision during incident response. We fixated for a very long time on the idea of a bridge loop and it blinded us to other possible causes. We hope to begin doing more scheduled incident response exercises in the coming months and will build scenarios that reinforce this.

> Though we cannot be absolutely certain, we believe that the combination of our growing number of users and the loss of the database cache that resulted when we resized the database caused a sudden, sharp increase in lock contention within mysql. The issues were compounded by the fact that Amazon uses proprietary technology to power the file system that RDS runs on, but doesn’t offer documentation about how this technology works during a resize operation. Further, the lack of root access to our database made the cause of the problems more difficult for us to understand.

> Monday's migration caused higher load on the database than our operations team has previously seen during these sorts of migrations. So high, in fact, that they caused Percona Replication Manager's health checks to fail on the master. In response to the failed master health check, Percona Replication manager moved the 'active' role and the master database to another server in the cluster and stopped MySQL on the node it perceived as failed. > > At the time of this failover, the new database selected for the 'active' role had a cold InnoDB buffer pool and performed rather poorly. The system load generated by the site's query load on a cold cache soon caused Percona Replication Manager's health checks to fail again, and the 'active' role failed back to the server it was on originally. [...] > Upon attempting to disable maintenance-mode, a Pacemaker segfault occurred that resulted in a cluster state partition. ... In the second, single-node cluster, node 'c' was elected at 8:19 AM, and any subsequent messages from the other two-node cluster were discarded. As luck would have it, the 'c' node was the node that our operations team previously determined to be out of date. ... As a result of this data drift, inconsistencies between MySQL and other data stores in our infrastructure were possible. ... Consequentially, some events created during this window appeared on the wrong users' dashboards. Also, some repositories created during this window were incorrectly routed. [...] > In summary, three primary events contributed to the downtime of the past few days. First, several failovers of the 'active' database role happened when they shouldn't have. Second, a cluster partition occurred that resulted in incorrect actions being performed by our cluster management software. Finally, the failovers triggered by these first two events impacted performance and availability more than they should have.

> Shortly after this, our DNS provider, GoDaddy, experienced a totally unrelated outage.

In past EBS incidents, it was very rare for volumes to be damaged even if they sustained downtime, so they were very fast to recover. We were therefore not fully prepared for a recovery effort of this magnitude, and it took several hours to automatically restore the affected databases.

2012-06-30 05:25 UTC: I finish configuring the replacement Tarsnap server and start the process of regenerating its local state from S3. The first phase of this process involves reading millions of stored S3 objects; unfortunately, these reads were performed in sequential order, triggering a worst-case performance behaviour in S3. As a result, this phase of recovery took much longer than I had anticipated; unfortunately, the design of the code meant that changing the order in which objects were read was not something I could do "on the fly".

... As the power and systems returned, a large number of ELBs came up in a state which triggered a bug we hadn’t seen before. The bug caused the ELB control plane to attempt to scale these ELBs to larger ELB instance sizes. This resulted in a sudden flood of requests which began to backlog the control plane. At the same time, customers began launching new EC2 instances to replace capacity lost in the impacted Availability Zone, requesting the instances be added to existing load balancers in the other zones. These requests further increased the ELB control plane backlog. Because the ELB control plane currently manages requests for the US East-1 Region through a shared queue, it fell increasingly behind in processing these requests; and pretty soon, these requests started taking a very long time to complete.

The first root cause is related to the streaming data API which connects the dyno manifold to the routing mesh. On the dyno management side, an engineer was performing a manual garbage collection process which created an unusual record in the data stream. On the routing side, the subprocess of the router which handles the incoming stream could not parse this record.

On Tuesday (Feb 21), we deployed a code change to the dyno management agent which runs on the machines which comprise our dyno manifold. This change introduced a latent problem with creating extra ("phantom") dynos, which only became visible under unusual load conditions that would occur that weekend.

If a clean VM (one in which no customer code has executed) times out its GA connection three times in a row, the HA decides that a hardware problem must be the cause since the GA would otherwise have reported an error. The HA then reports to the FC that the server is faulty and the FC moves it to a state called Human Investigate (HI).

On Saturday we had a rough night. There were problems routing HTTP traffic to customer applications, to varying degrees, for more than four hours. ... Upon further investigation, we discovered that we were being attacked by a very large scale distributed denial of service (DDoS) attack. This attack was sending multiples of our normal production traffic levels to our HTTP routing system which was unable to handle the load.

The hot spare for the database was not known to be in a usable state due to missing information in the management UI. We are improving our monitoring to monitor the status of the hot spare to ensure that it is in a usable state. We are also improving the documentation around failing over to the hot spare since it was also not immediately obvious to the responding engineers what the procedure was.

First, our EC2 management service (which handles API requests to RunInstance, CreateVolume, etc.), has servers in each Availability Zone. The management servers which receive requests continued to route requests to management servers in the affected Availability Zone. Because the management servers in the affected Availability Zone were inaccessible, requests routed to those servers failed. Second, the EC2 management servers receiving requests were continuing to accept RunInstances requests targeted at the impacted Availability Zone. Rather than failing these requests immediately, they were queued and our management servers attempted to process them. Fairly quickly, a large number of these requests began to queue up and we overloaded the management servers receiving requests, which were waiting for these queued requests to complete. The combination of these two factors caused long delays in launching instances and higher error rates for the EU West EC2 APIs.

At approximately 03:30 Pacific time, it was determined that Heroku was the target of a distributed denial of service (DDoS) attack. The method of the DDoS was a SYN flood [1]. Our network engineers applied a variety of host-based techniques to reduce the effect of the attacks. A combination of tweaked networking configuration and some quickly built tools for maintaining system firewall rules helped us get the affected servers to the point where they were no longer crashing, but we were still seeing very high levels of network packet loss to the nodes -- users often had to reload multiple times in order to get a web page to load.

After the initial sequence of events described above, the degraded EBS cluster had an immediate impact on the EBS control plane. When the EBS cluster in the affected Availability Zone entered the re-mirroring storm and exhausted its available capacity, the cluster became unable to service “create volume” API requests. Because the EBS control plane (and the create volume API in particular) was configured with a long time-out period, these slow API calls began to back up and resulted in thread starvation in the EBS control plane. The EBS control plane has a regional pool of available threads it can use to service requests. When these threads were completely filled up by the large number of queued requests, the EBS control plane had no ability to service API requests and began to fail API requests for other Availability Zones in that Region as well.

In this case, the EC2 outage lasted a total of about 12 hours. In the afternoon on Thursday, we were able to begin starting new instances en masse and we believed we'd be just an hour or two away from recovery. The majority of applications were back up on Thursday afternoon, but it took us much longer to recover the remaining ones.