Post Mortems for 4 Years of Remote Execution

### Overview This talk presents postmortems from four years of operating remote execution services at EngFlow (referred to phonetically in parts of the transcript as Eno/Enpl). Remote execution allows build clients like Bazel to upload files, download build artifacts, and execute actions on worker clusters via a cloud API server backed by remote storage. The speaker emphasizes that incident response does not end when an immediate issue is mitigated by restarting workers or servers; rather, true learning begins with a structured postmortem assessing impact, root cause, and actionable resolutions across technical, process, and human factors.

### Key Topics - Technical postmortems: structuring post-incident reviews around impact assessment, root cause analysis, and preventative actionable resolutions. - Distributed system RPC deadlocks: risks of cyclic dependency graphs and connection/stream exhaustion over HTTP/2. - Memory exhaustion and flow control: proxying high-throughput data between fast internal cloud networks and slow external clients without linked backpressure. - Third-party software integration and operations: monitoring blind spots, over-aggressive autoscaling, logging performance traps, and escalation paths in Goma deployments. - Guardrails and human error: managing massive queue buildup caused by misconfigured or broken client builds generating massive bursts of failing actions.

### Technical Details Four specific production incidents were analyzed:

1. **Worker Deadlock via Cyclic RPC Calls (2021)**: Customers experienced hanging builds where workers stopped responding to RPC calls, leaving 200 developers unable to build for 20 minutes until workers were restarted. Analysis revealed workers were initiating outgoing RPC calls to other workers in a chain with cycles. Because each worker maintained only a single outgoing HTTP/2 connection with a finite number of streams, mutual recursive calls caused stream exhaustion and deadlock. Action items included strictly enforcing an acyclic call graph across services and configuring RPC timeouts (though the speaker noted action execution can naturally run for an hour, limiting timeout effectiveness).

2. **API Server OOM Crashes from Missing Flow Control (2020)**: Cloud API servers occasionally spiked to 100% memory and crashed when proxying large files (such as 10GB Docker images). The API server pulled data rapidly across fast datacenter links from workers or storage, while the downstream client (Bazel) was on a slower network. Because the RPC library's automatic flow control was bypassed when combining async/Futures with proxying, incoming and outgoing streams lacked linked backpressure, causing buffers to allocate until memory was exhausted. Resolving this required explicit end-to-end flow control integration and adding flow control considerations to design review templates.

3. **Intermittent Slow Goma Builds (2022)**: For 15 days, a customer experienced intermittent slowness due to Goma falling back or queuing. Two technical issues caused this: a newly rolled-out autoscaler that aggressively scaled down under active load/queuing while scaling up very slowly, and a custom frontend refactor containing accidental verbose logging that saturated CPUs at 100% (hidden because logging pipelines dropped oversized messages). From a process perspective, Goma monitoring had become noisy and ignored, and the customer had communicated through chat rather than paging/escalating. Solutions included tuning monitoring, clarifying customer escalation pathways, and proactively raising internal incidents from chat inquiries.

4. **Queue Starvation from Faulty Client Actions**: A customer pool hit maximum queue age (causing actions to queue for hours and eventually drop after an internal one-hour age limit) when a single build dispatched 172,000 compile actions that timed out due to source code errors, consuming massive cluster capacity. Mitigations included introducing guardrails, reducing action timeouts, implementing quotas, detecting abnormal loads, and creating mechanisms to cancel misbehaving builds.

### Takeaways Outages and incidents in distributed systems are rarely isolated technical bugs; they usually expose weaknesses in design reviews, testing incentives, operational monitoring, or human communication. Organizations running remote execution and distributed infrastructure should ensure RPC call graphs remain acyclic, strictly manage proxy backpressure/flow control, invest in understanding third-party tools before hosting them in production, maintain clean alert signals, and implement guardrails and clear escalation channels.