Cascading Failure
One component's failure taking down its dependents, amplified by retries and blocked callers instead of being contained.
A cascading failure is one component's failure spreading through a distributed system. Anything that depends on the broken component fails as well, breaking its own dependencies.
This can happen in two ways.
The first is the synchronous call chain. Service A calls B, and B calls C. When C goes down, B's requests hang until they time out, holding connections. B slows down or runs out of resources, and now A has the same problem. The failure propagates upstream, and users see errors from top-level services, even though a single downstream service is the cause.
The second is retry amplification. A retry is a good way to handle a temporary error, but during an outage, every caller retrying at once multiplies the load. It gets worse with retries in batch jobs: a loop that makes 500 calls and fails on the last one re-runs all 500 on the next attempt. A couple of retries turn one stuck operation into thousands of calls aimed at a service that was already struggling. The failing service gets even more traffic.
References
- Synchronous vs Asynchronous Architecture: Describes the usual path to cascading failures: a team builds more microservices than people, one service in the middle stops working, and all requests in the platform start failing. Discusses asynchronous communication as the way out when reversing the microservices decision is off the table.
- AMA #1: Clean Architecture, Learning, Event-Driven, Go: Covers cascading failures in chains of synchronous service calls, where every service must handle errors and propagate them up the stack. Instead of increasing timeouts, consider asynchronous messaging, or question whether you need that many services at all.
