Following this, quarterly failovers were performed, until a fault in the failover tooling caused a 47 minute outage in May 2020, highlighting that the failover tooling did not itself fail safely. The disaster readiness team began building tools to make performing frequent failovers possible, and ran their first formalized failover in 2019. Some earlier design tradeoffs - such as using asynchronous replication of MySQL data between regions and using caches to scale databases - forced a rethink of the disaster readiness plan. Making the metadata stack resilient proved to be a difficult goal to achieve. The architecture for block storage - named Magic Pocket - allows block data to be served from multiple data centers in an active/active configuration, and part of this new resilience work involved making the metadata service more resilient, and eventually also active/active too. The overall architecture of Dropbox's systems involves a system to store files ( block storage), and another system to store the metadata about those files. The improvement was measured as a Recovery Time Objective (RTO) - a standard measure from Disaster Recovery Planning (DRP) for the maximum time a system can be tolerably down for after a failure or disaster occurs. The recovery time from an outage at San Jose was considered to be far in excess of what was desired - hence initiating a project to improve this in case of a significant disaster - such as an earthquake at the nearby San Andreas Fault. With a dependency on their San Jose data center, Dropbox ran a multi-year project to engineer away this single point of failure in their metadata stack, culminating in a deliberate and successful switch-off of the San Jose data center.ĭropbox had moved away from AWS for storing data, but were still heavily centralised and dependent on their San Jose data center. Dropbox has published a detailed account of why and how they unplugged an entire data center to test their disaster readiness.
0 kommentar(er)
