The engineering team at Stripe recently described how they automated database incident recovery by modeling their global infrastructure as a graph. Using graph search algorithms together with state machines, the team computes and executes remediation plans automatically.

According to the authors, the system now dynamically adapts to different MongoDB shard layouts, reducing database-related pager alerts by about 30%. This translates to 200 fewer pages per year, while eliminating an estimated 12 days of unhealthy shard states annually.

Stripe found that its original hard-coded, plugin-based remediation system did not scale well, as fragile dependencies, complex multi-failure scenarios, layout-specific logic, and unhandled intermediate states frequently required manual intervention. The authors acknowledge:

In one six-month window, the control plane paged operators 124 times for misconfigured shards, and 32 times for single-node-down scenarios complicated by additional health issues. Critical operations like index builds and planned maintenance were blocked for an average of one hour per incident.

Stripe addressed these limitations by modeling its MongoDB infrastructure as a graph, in which nodes represent infrastructure components, edges capture their relationships, and node attributes describe their current state.

Removing the previous hard-coded remediation sequences, Stripe now relies on graph traversal to identify valid recovery paths, allowing the same remediation logic to adapt automatically to different database layouts and evolving infrastructure.

Source: Stripe blog.

Stripe initially used breadth first search (BFS) to find valid remediation paths, but later adopted Dijkstra's algorithm to prioritize lower-cost recovery plans, reducing unnecessary operations while preserving correctness:

Because Dijkstra's algorithm explores paths to all reachable states rather than only the goal state, we can also get partial remediation. When no complete path exists, the algorithm returns the path to the least misconfigured state.

Rather than embedding recovery logic in fixed workflows, Stripe models remediation as composable rules with explicit state transitions, enabling the planner to combine operations dynamically as infrastructure evolves. According to the article, the team plans to extend the framework beyond failure recovery to automate topology changes and blue-green deployments, and to orchestrate planned maintenance alongside reactive healing. The team concludes:

For teams managing complex distributed infrastructure, this pattern of state machine modeling, simulation-based planning, and runtime pathfinding offers a compelling alternative to accumulating ever-more-specific runbooks. Runbooks encode known recovery procedures; a state machine discovers novel ones.

Stripe is not the only large software company investing in automated infrastructure operations. Uber recently described its declarative, self-healing Odin platform, while Meta has detailed AI-assisted tooling to accelerate incident response. Scott MacVicar, head of developer infrastructure at Stripe, writes on LinkedIn:

Operating a global database fleet means accepting that hardware degradation and unhealthy shards are daily occurrences. At scale, the challenge isn't just fixing issues. It is doing so without burning out your on-call engineers.

The engineering team at Stripe has also recently published articles on fund segregation in Stripe Connect and event notification handlers for processing thin events.