Article · Engineering

When the Service Stops: Lessons from Outlook, GitHub, and Binance

Caleb Ngeno

By · November 13, 2026

Three disrupted status panels for Outlook, GitHub, and Binance on a yellow warning background, with a broken connection line

This week made the dependency chain visible. On November 12, Microsoft 365 services went down across multiple regions, taking Outlook, Teams, Exchange Online, OneDrive, and Copilot with them and stretching into the next day. GitHub logged another cluster of incidents across Actions, Pages, pull requests, and email delivery. Binance was still reopening withdrawals in phases weeks after a $6.3 billion security incident.

Three companies, three very different failure modes. But the same uncomfortable lesson: modern services do not fail in isolation, and the controls we assume will save us are often the first thing the incident invalidates.

Outlook: capacity is a failure domain

The Microsoft outage began around midday Eastern on November 12. Downdetector recorded more than 130,000 Outlook reports at one point, and the disruption reached users in 74 countries according to the tracker. Outlook, Exchange Online, Teams, Copilot, OneDrive, and parts of Microsoft 365 were affected; Microsoft opened incidents under EX1550543 and MO1550821.

Microsoft attributed the problem to elevated load on a portion of infrastructure across multiple regions. Its updates described a failure that compounded itself: servers returned to service could become unresponsive again once they received traffic. The mitigations were the familiar emergency playbook — pause deployments, move traffic off the affected infrastructure, restart servers, limit which traffic types could reach the sick systems, and rebuild a subset of servers with a previous configuration.

Two details matter more than the outage itself. First, the company said the steps it tried were not working as expected, which is what an incident looks like when the recovery path depends on the same fragile component that failed. Second, no evidence of a cyberattack has been disclosed. This was an availability failure, not a breach — and that distinction should shape how we prepare for it.

The financial footnote: affected Microsoft 365 customers may be able to claim SLA credits, but they have to ask. A credit is not a recovery plan.

GitHub: change is the most common outage

GitHub's month has been a sequence of smaller, sharper failures. Status trackers recorded roughly 25 incidents in the last 30 days, totaling about 34 hours of disruption. The pattern is instructive because the causes are so ordinary:

  • On November 3, Git operations over HTTPS were degraded for hours, with median response time rising from under 100 milliseconds to as high as 5 seconds for about half of traffic.
  • On November 5, a database service storing Actions checks data hit resource exhaustion; roughly 3 percent of workflow runs failed to queue.
  • On November 9, a planned database migration to a new cloud region pushed traffic past the capacity and rate limits configured there, degrading Actions and Checks for a period.
  • Also on November 9, a degraded host had been slowing a Kafka broker since November 7, delaying webhook deliveries and Actions runs.
  • On November 10, Git operations and pull requests degraded for about an hour and 44 minutes.
  • On November 11, Actions, Pages, Webhooks, and Copilot were hit together for roughly half an hour.

None of these reads like a dramatic catastrophe. That is the point. Most outages are not caused by a mysterious adversary; they are caused by a change that was reasonable in isolation, a capacity assumption that was true last quarter, or a single unhealthy host that quietly poisons a shared dependency.

Binance: the pause button that does not pause

Binance is the opposite case: a security incident where the recovery itself became the disruption. On October 29, attackers moved roughly $6.3 billion in digital assets across multiple blockchains, what may be the largest crypto theft to date. Binance halted trading and withdrawals as a precaution, then began reopening withdrawals in phases on November 1. The first day of restored access produced about $3.6 billion in net outflows across 15 networks.

The most important disclosure came later. Binance's chief security officer explained that pausing withdrawals did not stop the attacker's transfers, because the authorization used to move funds existed on-chain, outside Binance's withdrawal system. Stopping it required a separate on-chain transaction to revoke the permission. Because the hot wallet used multiparty computation, no single private key was available to revoke it immediately, and containment took hours.

Read that again. The emergency stop was not wired to the thing that needed stopping. In a well-designed system, the ability to revoke access must be independent of the access path itself, pre-authorized, tested, and fast. Binance is now working with outside experts and plans a security research fund; those are the right moves after the fact. The lesson for everyone else is to find that gap before an incident finds it for you.

Fitch's assessment was that the market absorbed the shock better than earlier exchange failures: Bitcoin fell about 6 percent in the hours after the hack while daily volume rose 67 percent to roughly $40.9 billion. Market resilience is real. It is not the same as operational resilience.

The shared pattern

IncidentWhat failedWhat it reveals
Microsoft 365Capacity and health checks under elevated loadRecovery depended on the component that was already unhealthy
GitHubChange, capacity, and a degraded hostSmall configuration and infrastructure debts compound into user-visible outages
BinanceOn-chain authorization and revocationThe control plane could not stop the data plane

Three conclusions follow.

First, availability is a chain, not a property. A service is only as reliable as its identity provider, its DNS, its control plane, and the operator's ability to change any of them safely.

Second, recovery is a separate system from production. If your failover, revocation, or restore path shares a dependency with the thing that failed, you do not have a recovery path. You have a hope.

Third, transparency is part of the product. Users can forgive a outage that is clearly communicated. They do not forgive silence, contradictory updates, or a status page that says "investigating" for six hours.

Recommendations to curb the next disruption

These are not theoretical. Each one maps to something that failed this week.

1. Treat capacity as a failure domain

Model the loss of a region, a zone, and a control-plane component, then test whether the remaining capacity can absorb the shift. Microsoft's "elevated load" is what happens when it cannot. Keep headroom for failover, not just for average traffic, and make load shedding a designed behavior rather than an emergency improvisation.

2. Separate the control plane from the data plane

GitHub's migration and Binance's revocation both failed at the seam between the system that makes decisions and the system that moves data. Rate limits, capacity checks, and revocation paths must be provisioned and tested independently of the workload they govern. If the only way to stop traffic is through the service that is already saturated, the design is wrong.

3. Make revocation a first-class, pre-authorized operation

For cloud credentials, OIDC trust, API keys, and on-chain permissions: define who can revoke, how, and how fast without waiting on a committee or a single key holder. Test it quarterly. Binance's multiparty setup was a security strength that became a containment weakness because the revoke path was not designed for speed.

4. Stage every change and verify capacity before you shift traffic

Most of GitHub's incidents this month trace to changes: a migration, a configuration event, a database service under load. Canary the change, verify the new region's limits before cutover, and keep an automated rollback that does not require the same engineers who are debugging. A change freeze during an active incident is table stakes; a change freeze that only exists on paper is not.

5. Design graceful degradation, including offline paths

Email is not a single product; it is mailboxes, calendars, identity, and mobile clients. When Outlook is down, the useful question is not "is the service up?" but "which workflows still work?" Provide alternate channels for critical communication, cache read paths, queue writes, and give users a clear, honest view of what is degraded.

6. Make health checks trustworthy

Returning servers to service that immediately become unresponsive again turns a recovery into a loop. Use synthetic probes from outside the failing domain, apply connection draining and gradual ramp-up, and cap the blast radius of a bad health check. A health check that only knows whether a process is running is not a health check.

7. Write status updates that help

A status page should answer three questions: what is broken, who is affected, and what can users do right now. Commit to an update cadence and keep it even when there is nothing new. "We are investigating" is a start, not a communication strategy.

8. Rehearse recovery, not just failure

Teams practice failover. Fewer practice the messy middle: partial recovery, backlog drain, duplicate processing, and support volume. Run game days that include the control plane, the status page, and customer support. Measure time to detect, time to mitigate, and time to full recovery — and report them, even when they are embarrassing.

9. Know your contracts

SLA credits exist, but they are prorated, service-specific, and usually require a claim. Read the commitment before you need it, keep evidence of impact, and do not confuse a credit with a continuity plan. For critical dependencies, negotiate exit paths and data portability while the relationship is healthy.

10. Map the dependency chain you actually have

Most organizations cannot name the services their revenue path depends on beyond the first layer. Map identity, email, source control, CI, DNS, payments, and the vendors behind them. Then ask the uncomfortable question for each: if this disappeared for six hours, what would we do? If the answer is "wait," that is a finding.

A seven-day checklist

  • List the top ten external services your business cannot operate without.
  • For each, record the owner, the failure mode, the manual workaround, and the expected recovery time.
  • Test one revocation path end to end.
  • Verify that a non-production environment can be rebuilt from backups without the production control plane.
  • Draft a status-page template that includes user actions, not just engineering status.
  • Confirm who can request an SLA credit, and how.
  • Schedule one game day for the next 30 days.

The bottom line

Outlook, GitHub, and Binance are not the same kind of failure, and that is exactly why they belong in the same conversation. One was capacity, one was change, one was control. All three became user-visible disruptions because recovery depended on something that was already compromised.

The goal is not zero outages. It is outages that stay small, recover predictably, and are communicated honestly. That is an engineering property, not a press release. The organizations that get it right build the revoke button before they need it, test the failover before the region dies, and treat the status page as part of the product.

Work with TechGaetano

Resilience is cheaper to fund before an outage than after one. TechGaetano works with teams to map the dependencies they cannot afford to lose, design revocation and recovery paths that do not depend on the system that failed, and turn incidents like these into controls that hold under pressure.

If you want a second set of eyes on the assets your business cannot afford to lose — cloud, identity, source control, or on-chain — start a conversation with TechGaetano and protect them today.

Sources

Caleb Ngeno

About the author

Caleb Ngeno

Lead Developer

Founder and lead developer with expertise in full-stack development and cloud solutions.

Share this article

Work with TechGaetano

Need senior engineering firepower?

TechGaetano partners with teams on architecture, delivery, and the hard technical decisions that shape the next stage of a product.

Work with TechGaetano

Frequently asked questions

What is When the Service Stops?

This week made the dependency chain visible. On November 12, Microsoft 365 services went down across multiple regions, taking Outlook, Teams, Exchange Online, OneDrive, and Copilot with them and stretching into the next day. GitHub logged another cluster of in.

How should a team get started with When the Service Stops?

Start with the problem and its constraints, then choose the smallest tool that solves it. Build a small proof, measure the result, and expand only when the evidence supports it.

What should teams avoid when working on When the Service Stops?

The main risks are choosing tools by fashion, optimizing for the demo instead of production, and skipping measurement before expanding a technical bet.

Further reading

Related posts