Rafał Grembowski: Multi-regional high availability in AWS cloud | JDD 2023

### Overview In this talk presented at JDD 2023, Rafał Grembowski, an architect at Huuuge Games, discusses implementing multi-regional high availability in Amazon Web Services (AWS). He outlines the evolutionary steps of application redundancy—from single-machine processes to multiple servers, dedicated data centers, Availability Zones (AZ), and ultimately multi-region setups designed to withstand catastrophic infrastructure failures such as natural disasters or regional blackouts. While multi-AZ deployment handles common failures with reasonable effort, the speaker's organization took on the engineering challenge of an active-active multi-region architecture driven by both team interest and real-world disaster warnings (e.g., hurricane-related blackout risks on the US East Coast).

### Key Topics - **Active-Active vs. Active-Passive Architectures:** The trade-offs between running hot-standby failover setups and true active-active configurations where traffic and data replication flow across both regions simultaneously. - **AWS Regional Isolation:** Challenges arising because AWS regions are completely isolated units with separate VPCs, subnets, Kubernetes clusters, load balancers, and security credentials. - **Traffic Routing and Service Discovery:** Managing public user traffic using AWS Global Accelerator and private server-to-server traffic through internal networking and Route 53 Proximity routing while avoiding DNS caching bottlenecks. - **Data Replication and Capacity Planning:** Utilizing DynamoDB global tables for bidirectional multi-master replication, handling conflicts (last-write-wins), and factoring cross-region replication traffic into Read and Write Capacity Units (RCU/WCU). - **Custom Failover Health Checks:** Building custom health check microservices in Python to validate critical dependent services (such as DynamoDB and EC2/Kubernetes) rather than relying on trivial HTTP 200 responses. - **Cost Implications and Operational Benefits:** Evaluating financial impacts (data transfer, Global Accelerator costs, doubled storage) alongside infrastructural flexibility, canary deployments at the infrastructure layer, and reduced on-call stress.

### Technical Details - **Kubernetes & Load Balancing:** Kubernetes worker nodes must stay in the same region as their control plane, preventing multi-region clusters. Likewise, AWS load balancer target groups cannot cross regional boundaries, exposing endpoints in multiple places. - **DNS Limitations & Global Accelerator:** Public DNS caching and long TTL propagation (up to two days across cascades) prevent rapid failover via standard DNS edits. For public client traffic, AWS Global Accelerator provides a static anycast IP that routes transparently to backend endpoints across regions without DNS propagation delay. Internal server-to-server traffic relies on Route 53 with latency/proximity routing over private networks to satisfy corporate security policies. - **Database Layer:** DynamoDB was selected as the managed storage engine capable of bidirectional cross-region replication. Conflicts are resolved via timestamps (latest write wins). When provisioning RCU and WCU, engineers must allocate capacity for both local application operations and incoming replicated writes. - **Health Monitoring:** Because AWS does not offer an out-of-the-box regional health score, the team developed a custom Python endpoint. It probes upstream dependencies (e.g., listing DynamoDB tables) before returning healthy status to Route 53 or Global Accelerator. Centralized monitoring was transitioned from regional Grafana/Prometheus/Alertmanager instances to Datadog for unified cross-region metric analysis. - **Cost Drivers:** Compute (EC2) costs do not simply double if overall traffic is distributed (e.g., 70/30 split), but DynamoDB storage is paid in both regions, and Global Accelerator introduces significant data transfer premiums (roughly 3x to 4x higher standard egress).

### Takeaways Multi-region active-active architectures require solving significant distributed systems problems—including data synchronization, traffic rerouting without DNS caching latency, and end-to-end health checking. While not necessary for every project due to cost and complexity, multi-region setups offer superior fault tolerance against catastrophic outages, enable infrastructure-level canary releases, reduce user latency via geographic routing, and give on-call engineers high operational confidence.