43 Seconds of Darkness, 24 Hours of Damage: GitHub’s 2018 MySQL Failover and the Split-Brain Nobody Could Undo

37 minute read

Published:

中文版 Chinese version

This is the sixth entry in the 🔥 Incident Recap series.

Start with the disproportion that frames the whole incident: 43 seconds of network partition, 24 hours and 11 minutes of degraded service. The network was back in under a minute. The database was not back for a day. On October 21, 2018, routine maintenance to replace failing 100G optical equipment cut connectivity between GitHub’s US East Coast network hub and its primary US East Coast data center for 43 seconds. In that window, the automated failover system did what it was built to do: it decided the primary was gone and promoted replicas on the US West Coast. The application tier followed, and writes began flowing to the other side of the country. The problem was not that the failover fired. The problem was that a few seconds of writes on the East Coast had never reached the West Coast — and once both coasts held writes the other one lacked, there was no safe way back. The system had two primaries’ worth of truth and no way to merge them.

That is the shape of this incident, and it is one this series has not covered yet. The Meta BGP outage (entry #1) locked engineers out of their own data centers. The Cloudflare regex outage (entry #2) was a single bad pattern amplified by eleven converging causes. The Fastly outage (entry #3) was one valid config change waking a latent bug in shared code. The AWS us-east-1 outage (entry #4) was a feedback loop that fed on its own retries. Knight Capital (entry #5) was a deployment that woke dead code and a rollback that spread the fault. This one is a sixth shape: an automation that behaved exactly as configured, a failover that succeeded, and a recovery that took a day anyway — because the hard part was never restoring availability. It was restoring correctness, and correctness had to be rebuilt from backups, terabyte by terabyte, while the site served stale data to the whole world.

One note on sources before the story. This article is written against GitHub’s own postmortem, the “October 21 post-incident analysis” published on the GitHub Blog on October 30, 2018, and against GitHub’s contemporaneous “October 21 Incident Report” from the night itself. Every technical claim comes from those two documents unless explicitly marked as analysis. GitHub’s postmortem is unusually candid about the decisions its engineers made during the incident, including the ones that prolonged it — and the most useful parts of this article are the places where the postmortem says, in plain language, that a painful choice was deliberate.

Background: what MySQL holds at GitHub, and what Orchestrator does

The first thing to get straight is what actually broke, because the popular summary — “GitHub went down for a day” — is wrong in the interesting way. Git itself never went down. GitHub’s own incident report, published during the outage, says so directly: the incident “only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Git repository data remains unaffected and has been available throughout the incident.” Cloning, pushing, and pulling kept working. What broke was the metadata layer around the repositories: pull requests, issues, authentication, background processing — everything the website shows you about a repository, as opposed to the repository itself.

That metadata lives in MySQL. GitHub runs multiple MySQL clusters of different sizes, from hundreds of gigabytes to nearly five terabytes, with up to dozens of read replicas per cluster, using functional sharding — different clusters for different kinds of data. Applications direct writes to the relevant primary for each cluster and delegate most reads to a subset of replicas. The topology of those clusters — which server is primary, which are replicas, how replication flows — is managed by Orchestrator, an open-source tool that GitHub uses for MySQL cluster management and automated failover. Orchestrator is built on Raft for consensus: a group of Orchestrator nodes elect a leader, and that leader decides when a primary has failed and which replica should be promoted.

Two properties of this setup matter for everything that follows. First, replication between the primary and its replicas is asynchronous in the normal sense that matters here: a write acknowledged on the primary is not guaranteed to have reached every replica at the same instant. There is always a small window — usually milliseconds, sometimes seconds — in which the primary holds writes its replicas do not yet have. Second, Orchestrator’s configuration determines where a promoted primary is allowed to live. As the postmortem puts it, Orchestrator “can implement topologies that applications are unable to support,” so its configuration has to align with what the application tier can actually live with. That sentence is the whole incident in miniature: the failover system and the application tier had different ideas about which topologies were acceptable, and nobody had noticed, because the disagreement had never been exercised.

Timeline (all times UTC, October 21–22, 2018)

TimeEvent
Oct 21, 22:52Routine maintenance to replace failing 100G optical equipment cuts connectivity between the US East Coast network hub and the primary US East Coast data center. Connectivity is restored in 43 seconds. During the partition, Orchestrator begins leadership deselection under Raft; nodes in the US West Coast data center and the US East Coast public cloud form a quorum and begin failing clusters over to West Coast primaries. The East Coast database servers hold a brief period of writes not replicated to the West Coast.
Oct 21, 22:54Internal monitoring begins generating alerts of numerous faults. Engineers respond and triage.
Oct 21, 23:02First responders determine the replication topologies of numerous database clusters are in an unexpected state. Querying the Orchestrator API shows a topology including only US West Coast servers.
Oct 21, 23:07The responding team manually locks internal deployment tooling to prevent additional changes.
Oct 21, 23:09The site is placed into yellow status, automatically escalating to an active incident and alerting the incident coordinator.
Oct 21, 23:11The incident coordinator joins; two minutes later the status is changed to red.
Oct 21, 23:13The problem is understood to affect multiple database clusters. Database engineering is paged. The West Coast clusters have already ingested nearly 40 minutes of application writes, and the unreplicated East Coast writes prevent replicating new writes back to the East Coast. GitHub decides the 30+ minutes of West Coast data rules out every option except failing forward.
Oct 21, 23:19Jobs writing metadata about things like pushes are stopped. Webhook delivery and GitHub Pages builds are paused — a deliberate partial degradation to protect data already received.
Oct 22, 00:05The incident team begins a plan: restore from backups, synchronize replicas in both sites, fall back to a stable serving topology, then resume queued jobs. Backups run every four hours and are stored remotely in public-cloud blob storage.
Oct 22, 00:41Backup restoration for all affected MySQL clusters has been initiated and is being monitored.
Oct 22, 06:51Several clusters have completed restoration from backups in the East Coast data center and begun replicating new data from the West Coast. Larger clusters are still restoring. The status page is updated with a two-hour estimated time of recovery, linearly interpolated from replication telemetry.
Oct 22, 07:46GitHub publishes a blog post with more context — with extra effort, because GitHub uses GitHub Pages internally and all Pages builds have been paused for hours.
Oct 22, 11:12All database primaries are established in the US East Coast again. Dozens of read replicas are still multiple hours delayed behind the primary, so users keep seeing inconsistent data. Replication catch-up is following a power-decay function, not the linear trajectory the estimate assumed.
Oct 22, 13:15Approaching peak traffic, replication delays are increasing rather than decreasing. GitHub begins provisioning additional MySQL read replicas in the East Coast public cloud to spread read load so replication can catch up.
Oct 22, 16:24Once replicas are in sync, GitHub conducts a failover to the original topology. Status is deliberately kept red while the accumulated backlog is processed.
Oct 22, 16:45Over five million hook events and 80,000 Pages builds are queued. As processing resumes, about 200,000 webhook payloads that have outlived an internal TTL are processed and dropped. Processing is paused and the TTL is increased.
Oct 22, 23:03All pending webhooks and Pages builds have been processed and the integrity and proper operation of all systems confirmed. Site status is updated to green. Total: 24 hours and 11 minutes of degraded service.

The cascade: 43 seconds, two primaries’ worth of truth, and a day of rebuilding

graph TD; A["Oct 21 2018 22:52 UTC: maintenance replaces failing 100G optics"] -->|"loss of connectivity"| B["Network partition: East Coast hub cut off from primary data center"]; B -->|"restored in 43 seconds"| C["Connectivity returns"]; B -->|"during the partition"| D["Orchestrator loses its leader in the East"]; D -->|"Raft quorum forms in the West"| E["Orchestrator promotes primaries in the US West Coast"]; E -->|"app tier follows the new topology"| F["Writes flow to West Coast primaries"]; C -->|"East Coast primaries still hold writes"| G["A few seconds of East Coast writes never replicated West"]; G -->|"both sides now hold writes the other lacks"| H["Split-brain: divergent writes on both coasts"]; H -->|"no safe failback exists"| I["Failing back would discard one side's writes"]; I -->|"GitHub chooses to fail forward"| J["Cross-country latency makes service unusable for many"]; J -->|"recovery means rebuilding from backup"| K["Multi-terabyte restores from remote blob storage"]; K -->|"replicas hours behind"| L["Read replicas serve stale and inconsistent data"]; L -->|"backlog grows"| M["Over 5 million webhooks and 80,000 Pages builds queue"]; M -->|"TTL expires on some payloads"| N["About 200,000 webhook payloads dropped"]; K -->|"23:03 UTC Oct 22"| O["All backlogs processed: 24 hours 11 minutes"];

The diagram has a shape worth noticing: the fault itself is at the top and takes two boxes. Everything below it is recovery — and recovery is most of the diagram, because recovery is most of the incident.

Mechanism 1: a failover that fired on a transient partition — and had no undo

The first mechanism to understand is why a 43-second partition caused a failover at all. A network partition is the one event a consensus system is designed to reason about, and Orchestrator reasoned about it correctly by its own lights. When the East Coast hub lost connectivity to the primary data center, the Orchestrator nodes that could still talk to each other — in the West Coast data center and in the East Coast public cloud — formed a Raft quorum. A quorum that cannot see the primary concludes, by design, that the primary is gone. Orchestrator then did the thing it exists to do: it promoted replicas to primary, in the West Coast, for the affected clusters. And when connectivity returned 43 seconds later, the application tier immediately began directing write traffic to the new West Coast primaries. The failover was fast, automatic, and — by every measure Orchestrator had — successful.

Here is the causal chain the quality bar for this series demands, stated plainly. The failover had no grace period for a transient partition, and no fencing of the old primary. A partition that lasts 43 seconds is indistinguishable, to a consensus group on the other side of it, from a partition that lasts forever — that is the entire point of a consensus protocol, and also its entire danger. Orchestrator could not know the partition was about to heal, so it acted as if it would not. And when it healed, nothing stopped the old East Coast primaries from still holding writes, because nothing had fenced them: they were alive, healthy, and briefly partitioned, and during those 43 seconds they kept accepting writes that their replicas in the West never received. The failover system treated “unreachable” as “dead.” The database servers treated “unreachable” as “carry on.” Both were right, and the combination produced two live histories.

What this mechanism does not explain, and what it cannot fix. Adding a grace period — “don’t fail over until the primary has been unreachable for N seconds” — would have prevented this exact incident, since the partition lasted 43 seconds. But a grace period is a tradeoff, not a fix: it trades failover speed for partition tolerance, and the right value of N is unknowable in advance, because the next partition might last 44 seconds or four hours. More importantly, a grace period fixes the trigger, not the structure. The structural defect is that the failover system was allowed to create a topology the application tier could not support: primaries a continent away from the applications writing to them. The postmortem names this directly — Orchestrator behaved as configured, but its configuration allowed promotion of database primaries across regional boundaries, which the application tier could not support, and the sudden cross-country latency was a major contributing factor. The fix that actually shipped was not a grace period. It was a configuration change: never promote a primary across a regional boundary again.

Mechanism 2: split-brain — why two sets of writes cannot be merged

The second mechanism is the one that turned a one-minute failover into a one-day outage, and it is the mechanism most often misunderstood about this incident. The common telling is “the failover went to the wrong data center.” The accurate telling is that the failover created a state with no safe reverse.

Here is the precise shape of it. During the 43-second partition, the East Coast primaries accepted a few seconds of writes that were never replicated to the West Coast. The total was small — one of the busiest clusters had 954 writes in the affected window — but small is irrelevant, because the problem is structural, not volumetric. Meanwhile, once the failover completed, the application tier wrote to the new West Coast primaries for roughly 30 to 40 minutes before the responders fully understood the situation. So by 23:13 UTC, each coast held writes the other coast lacked. The West Coast had tens of minutes of new application data. The East Coast had a few seconds of older data that had never left the building.

MySQL replication, in the topology GitHub was running, cannot merge two divergent histories. Replication is a one-way stream: a replica applies the primary’s binary log in order. If you point the West Coast primary at the East Coast as its upstream, it will refuse — or worse, silently conflict — because the East Coast’s log contains writes the West Coast does not have, and the West Coast’s log contains writes the East Coast does not have. There is no “merge” operation for two primaries that have both accepted writes. The only operations available are destructive: pick one side’s history and discard the other’s. Failing back to the East Coast would have meant throwing away 30+ minutes of writes that users had already made and, in many cases, already seen confirmed. Failing forward — staying on the West Coast — meant throwing away the few seconds of East Coast writes, which had to be recovered by hand from binary logs afterward.

That is why the responders, at 23:13 UTC, concluded that the 30+ minutes of data written to the West Coast “prevented considering options other than failing forward to keep user data safe.” Note the phrasing: the decision was not “fail forward because it is faster.” It was “fail forward because every other option deletes user data.” The asymmetry is the whole story: 30 minutes of visible, confirmed writes on one side versus a few seconds on the other. The side with more data wins, and the side with less data has to be reconciled by hand — which is why, at the time the postmortem was published nine days later, GitHub was still doing manual reconciliation of those few seconds of writes, having captured the binary logs from each affected cluster and analyzed which writes could be reconciled automatically and which would require reaching out to users.

What this mechanism cannot fix. Nothing in the recovery fixes the underlying asymmetry of asynchronous replication: there is always a window, however small, in which the primary holds writes its replicas lack. Semi-synchronous replication narrows the window — it waits for at least one replica to acknowledge — but it does not close it, and it does not help when the partition separates the primary from all of its replicas at once, which is what happened here. The durable fix is not a replication mode. It is the recognition that a failover which can create divergent writes is a failover that must be fenced: either the old primary is prevented from accepting writes during the partition, or the promotion is not allowed to cross the boundary that makes the divergence unrecoverable. GitHub’s chosen fix — pinning promotions inside a region — is the second kind.

Mechanism 3: why recovery took a day — backups in the cloud, and replication that decays, not lines

Once the decision to fail forward was made, the recovery plan was straightforward in outline and brutal in execution: restore the affected clusters from backups into the East Coast data center, synchronize replicas on both coasts, fall back to a stable serving topology, and then resume the queued jobs. The plan began at 00:05 UTC on October 22. It finished at 23:03 UTC. Here is why each stage cost what it cost.

The backups were remote, and the clusters were terabytes. GitHub’s MySQL backups run every four hours and are retained for many years — but they are stored remotely, in a public-cloud blob storage service. Restoring a cluster meant transferring multiple terabytes out of that blob storage, then decompressing, checksumming, preparing, and loading large backup files onto newly provisioned MySQL servers. The postmortem is explicit that the transfer was only part of it: decompressing, checksumming, preparing, and loading “took most of the time.” The procedure itself was tested daily at minimum, so the recovery timeframe was well understood — but GitHub had never before needed to fully rebuild an entire cluster from backup, having relied instead on strategies such as delayed replicas. A tested procedure and an exercised procedure are different things, and the first full-scale exercise happened during the incident. Later in the incident, teams found a faster path — restoring directly from the West Coast, to get around the throughput limits of downloading from off-site storage — but that discovery came hours in.

Replication catch-up followed a power-decay curve, not a line. By 11:12 UTC, all primaries were back in the East Coast and the site became far more responsive, because writes were once again directed to a database server in the same physical data center as the application tier. But dozens of read replicas were still multiple hours behind the primary, and read load was spread across a large pool of replicas — so any given request had a good chance of landing on a replica that was hours stale. Users saw inconsistent data: an issue that existed on one page load and vanished on the next. The status page had earlier promised a two-hour estimated recovery, linearly interpolated from replication telemetry. The actual catch-up followed a power-decay function, not a linear trajectory — the remaining lag shrank more slowly as it got smaller — and rising write load as users woke up in Europe and the US made it slower still. By 13:15 UTC, replication delays were increasing instead of decreasing. The responders’ answer was to provision additional MySQL read replicas in the East Coast public cloud, spreading read volume across more servers so that lower utilization per replica let replication catch up. That is the mechanism worth remembering: replica lag is not a function of how far behind you are. It is a function of how much read load each replica carries while it tries to catch up, and adding read capacity is a legitimate recovery action, not just a scaling action.

The incident stayed red on purpose. At 16:24 UTC, once replicas were in sync, GitHub failed over to the original topology. But the site status was deliberately kept red while the accumulated backlog was processed. The postmortem states the reasoning plainly: as a conscious decision to prioritize data integrity over a shorter incident window, service status was kept red until the backlog was fully processed and services had clearly settled back to normal. This is the same choice made at 23:19 UTC the night before, when webhook delivery and Pages builds were paused rather than risking data already received. Twice in one incident, GitHub chose a longer outage over a risk to data. That is a defensible choice and an expensive one, and the postmortem owns it as a choice rather than presenting it as something that merely happened.

Mechanism 4: the backlog, and 200,000 webhooks that outlived their TTL

The final mechanism is the one that turned a database recovery into a platform-wide degradation: the backlog. For most of the incident, GitHub could not serve webhook events or build and publish GitHub Pages sites. By 16:45 UTC on October 22, there were over five million hook events and 80,000 Pages builds queued. Draining a queue that large is not just a matter of throughput; it is a matter of what the queue’s consumers can absorb. Webhooks go to users’ servers and to ecosystem partners — CI systems, chat integrations, deployment pipelines. Releasing five million of them at full speed risks overloading those partners, so the drain had to be balanced against the risk of taking down someone else’s system with a flood of notifications.

And then the queue bit back. As processing was re-enabled, about 200,000 webhook payloads that had outlived an internal time-to-live were processed — and dropped. The TTL existed to bound how long a payload could sit before it was considered stale; the backlog had simply grown older than the bound. On discovering this, GitHub paused processing and pushed a change to increase the TTL for the time being. The detail matters because it is the incident’s second-order failure in miniature: a safeguard (the TTL) designed for normal operations became a data-loss mechanism under backlog conditions, and it fired silently, in the middle of the recovery, in a system that was already the thing being recovered. The postmortem counts the loss honestly — about 200,000 payloads — and the contrast with the database writes is instructive: the database writes were reconciled by hand, binary log by binary log; the webhook payloads were simply gone, because a queue’s TTL is a deletion policy with a timer on it.

Mechanism 5: the estimates were wrong — and the incident broke its own communications

Two smaller mechanisms deserve their own section, because both are failure modes that recur in other incidents in this series.

The recovery estimate was built on the wrong curve. The two-hour estimate published around 06:51 UTC was linearly interpolated from replication telemetry: if lag is shrinking at this rate, it reaches zero in two hours. Replication lag does not shrink linearly. It follows a power-decay function — fast at first, then progressively slower — and the estimate also failed to account for the rising write load as Europe and the US woke up and started working. The result was an estimate that was wrong in the direction that erodes trust: too optimistic, published on the status page, then visibly missed. The postmortem apologizes for the confusion and says GitHub would strive to provide more accurate information in future. The transferable lesson is narrower than “estimate better”: when you extrapolate a recovery curve, name the curve you are assuming, and check whether the system’s own dynamics — rising load, per-replica utilization, backlog growth — bend it. A linear extrapolation of a decaying process is not a rough estimate. It is a wrong model, and it will be wrong in the same direction every time.

The incident disabled the channel GitHub uses to talk about incidents. GitHub publishes its blog on GitHub Pages, and all Pages builds had been paused since 23:19 UTC on October 21 as part of the deliberate degradation. So when the team wanted to publish a blog post with more context at 07:46 UTC on October 22, publishing took additional effort — the postmortem says so directly, and apologizes for the delay, noting that the team had intended to send the communication much sooner. The incident had broken the company’s own incident-communications pipeline. This is the same class of failure as the Meta BGP outage in entry #1, where the network failure took out the tools needed to fix the network. The general form: your communications path, your deployment path, and your monitoring path all run on the system that is down. If the blog is hosted on the platform that is broken, the post about the breakage cannot ship until the platform is fixed. GitHub’s follow-up — accelerating a new status-reporting mechanism that shows the status of different platform components instead of only green, yellow, and red — is a partial answer to this, but the deeper answer is the one the postmortem lists under technical initiatives: serve traffic from multiple data centers in an active/active/active design, so that no single facility’s failure can take out the platform and its own voice at the same time.

What would have stopped it — and what would not have

The postmortem’s own “next steps” name the fixes, and they are worth reading in the order GitHub lists them, because the order reveals what GitHub considered structural and what it considered hygiene.

  1. Pin Orchestrator promotions inside a region. Adjust the Orchestrator configuration to prevent promotion of database primaries across regional boundaries. Leader election within a region is generally safe; the sudden cross-country latency was a major contributing factor. This is the one-line fix that would have prevented the entire incident: if the failover cannot cross a region, a 43-second partition in the East cannot create a primary in the West, and there is no divergent history to reconcile.
  2. Serve from multiple data centers, active/active/active. Accelerate the company-wide initiative, begun weeks before the incident, to support serving GitHub traffic from multiple data centers with N+1 redundancy at the facility level, with the goal of tolerating the full failure of a single data center without user impact. This is the real fix, and the postmortem is honest that it is a major effort that will take some time. Note the relationship between fixes 1 and 2: fix 1 constrains the automation to the topology the application can support today; fix 2 rebuilds the application so that it can support the topology the automation wants to create. One is a guardrail, the other is the road.
  3. Validate failure scenarios before they affect users. Begin a systemic practice of validating failure scenarios, including future investment in fault-injection and chaos-engineering tooling. GitHub calls the incident “emergent behavior” and notes it had not previously seen an internal network partition of this magnitude. The partition was not a failure anyone had tested, because it had never happened — which is precisely the class of event that fault injection exists to produce on purpose.
  4. Better status reporting. Accelerate migration to a new status-reporting mechanism with richer, crisper language and per-component status instead of only green, yellow, and red.

And the honest boundary: none of these fixes addresses the 43-second partition itself. Routine maintenance to replace failing optical equipment caused a loss of connectivity, and the postmortem does not elaborate on why the replacement took the link down. A reader who wants a story about a technician’s mistake will not find one in the postmortem, and inventing one would be filling a gap the primary source deliberately leaves open. The disciplined claim is the structural one: a system that treats a transient partition as a permanent failure will eventually experience a transient partition, and the cost of that event is set not by the partition but by everything downstream of it.

Lessons worth stealing

  1. Your failover system and your application must agree on which topologies are legal. Orchestrator could implement topologies the application tier could not support, and the configuration allowed it to. The postmortem’s phrasing is the lesson: automation configuration has to align with application-level expectations. Audit your failover automation for the same gap — ask what topology it is allowed to create, not just what topology it usually creates, and check whether your application can actually run on each one.
  2. A failover with no fencing is a split-brain waiting for a partition. If the old primary can keep accepting writes while the new primary is being promoted, you have two histories, not one. Fence the old primary, or do not promote. The 954 writes on the busiest cluster were a few seconds of traffic; they cost nine days of manual reconciliation, because a divergent write is not a volume problem. It is a logic problem, and its cost does not scale with its size.
  3. Failing back is not the reverse of failing over. The asymmetry between the two directions is the incident: 30 minutes of confirmed writes on one side made the reverse direction a data-deletion operation. Before you design a failover, design the failback, and ask what state each direction discards. If the answer is “whichever side has less data,” you have a failover that can only move in one direction — and you should know that before the night you need it to move in the other.
  4. A tested restore procedure is not an exercised restore procedure. GitHub tested its backup restore daily and had never needed to fully rebuild an entire cluster from backup. The first full rebuild happened during the incident, and it exposed the real costs: remote blob storage transfer, decompression, checksumming, preparation, loading. Test the restore at full scale, on a clock, with the real storage path — not just the script.
  5. Replica lag is a function of read load, not just of distance behind. Adding read replicas was a recovery action, not a scaling action: spreading read volume lowered per-replica utilization and let replication catch up. When your replicas fall behind, your first question should be what load they are carrying while they try to catch up, not only how far behind they are.
  6. A linear estimate of a decaying process is a wrong model, not a rough one. Replication catch-up followed a power-decay function; the two-hour estimate was linearly interpolated and was wrong in the optimistic direction. Name the curve you are assuming before you publish the number, and re-check it when the load profile changes underneath you.
  7. Your incident-communications path must survive the incident. GitHub’s blog runs on GitHub Pages, and Pages builds were paused as part of the degradation — so the post explaining the outage could not ship on time. The same failure appears in entry #1 of this series, where the network outage took out the tools needed to fix it. Put your status page, your blog, and your incident tooling on infrastructure that does not share fate with the system they describe.
  8. A TTL is a deletion policy with a timer — check what it deletes under backlog. About 200,000 webhook payloads were dropped because they outlived an internal TTL during the recovery. The TTL was a reasonable safeguard under normal conditions and a silent data-loss mechanism under backlog conditions. Every TTL, retention window, and expiry in your system has a backlog scenario; find it before the backlog does.
  9. Choosing data integrity over uptime is a choice — say so, and own the cost. GitHub paused webhooks and Pages builds at 23:19 UTC and kept the status red at 16:24 UTC the next day, twice choosing a longer outage over a risk to data. The postmortem presents both as deliberate decisions, with reasons. If your incident process makes the same choice implicitly — by freezing everything until someone says go — it is still a choice, but one nobody has thought through. Make it explicit, in advance, so the 3 a.m. version of the decision is the one you already agreed to.

What the postmortem does not say

This is the article’s honesty ledger. GitHub’s postmortem, for all its candor, does not disclose: the exact number of database clusters affected; why the optical-equipment replacement took the link down, beyond the fact that it was routine maintenance to replace failing equipment; how many users were ultimately contacted about the unreplicated writes, or the final outcome of the manual reconciliation, which was still in progress when the postmortem was published nine days later; or the per-cluster write counts beyond the single example of 954 writes on one of the busiest clusters. The contemporaneous incident report, published on the night itself, says “no data was lost” — a statement the postmortem nine days later softens to “no user data was lost; however manual reconciliation for a few seconds of database writes is still in progress.” The two statements are consistent if the reconciliation eventually succeeded, but the postmortem does not confirm that it did. This article reports both statements as they were published, and does not claim a final reconciliation outcome the primary source does not state.

The contrast with the other entries in this series is worth noting. Knight Capital published no postmortem at all — the SEC published one for it. GitHub published its own, nine days after the incident, with the timeline, the decisions, the wrong estimate, and the 200,000 dropped webhooks all in it. The postmortem is also unusually specific about what it got wrong in its own response, including the estimate and the delayed blog post. That specificity is the reason this incident is a useful case study rather than a cautionary tale: the company that writes down exactly how its automation outran its application is the company whose next failover is less likely to do the same.

Sources

  • GitHub Engineering (Jason Warner), “October 21 post-incident analysis,” GitHub Blog, October 30, 2018 (updated December 19, 2021). https://github.blog/news-insights/company-news/oct21-post-incident-analysis/ — primary source for everything in this article unless noted: the 43-second partition and the failing 100G optical equipment (22:52 UTC); Orchestrator’s Raft leadership deselection and the West Coast quorum; the unreplicated East Coast writes and the impossibility of safe failback; the full timeline from 22:54 UTC on October 21 through 23:03 UTC on October 22; the 24 hours and 11 minutes of degraded service; the decision to fail forward and the 30+ minutes of West Coast writes; the pause of webhooks and Pages builds at 23:19 UTC; the backup-restore plan and the remote public-cloud blob storage; the two-hour estimate and its linear interpolation; the power-decay replication catch-up; the additional read replicas in the East Coast public cloud; the 16:24 UTC failover to the original topology and the deliberate decision to keep status red; the five million queued hook events, 80,000 queued Pages builds, and ~200,000 dropped webhook payloads; the 954 writes on one of the busiest clusters and the ongoing manual reconciliation from binary logs; the Orchestrator cross-region configuration fix; the active/active/active initiative; and the fault-injection and chaos-engineering commitment.
  • GitHub (Jason Warner), “October 21 Incident Report,” GitHub Blog, October 21, 2018 (updated December 19, 2021). https://blog.github.com/2018-10-21-october21-incident-report/ — the contemporaneous report published during the incident: the network partition and subsequent database failure; the pause of webhook events and other internal processing; the statement that only MySQL metadata (issues, pull requests) was impacted and that Git repository data remained unaffected and available throughout; and the “no data was lost” statement as published on the night itself.
  • Sections labeled as analysis — the grace-period tradeoff argument, the fencing argument, the failover/failback asymmetry reading, the “tested versus exercised restore” distinction, the replica-lag-as-a-function-of-read-load reading, the linear-versus-power-decay estimation critique, and the incident-communications single-point-of-failure reading — are the author’s reasoning from the postmortem’s facts and from standard distributed-systems practice, not GitHub’s statements. Where the postmortem is silent (the number of affected clusters, the reason the optical replacement took the link down, the final reconciliation outcome), the article says so rather than filling the gap.