Lightning Talk: Hotspot detection and handling in FoundationDB

### Overview This lightning talk focuses on identifying, diagnosing, and mitigating read and write hotspots in FoundationDB (FDB). A hotspot occurs when there is a high volume of writes or reads directed at a small set of blocks or partitions on a storage node or team of nodes. The speaker explains the core internal metrics used to detect these conditions, how FoundationDB's default prioritization affects node performance under load, and practical strategies for resolving hotspot issues by combining storage-level metrics with application-level context.

### Key Topics - **Definition of Hotspots**: Concentrated reads or writes across a small number of partitions or storage nodes. - **FoundationDB Storage Metrics**: Roles of non-durable versions (NDV) in the version tree, the storage queue mutation size, and Ratekeeper throttling. - **Hotspot Detection Signatures**: Distinguishing write hotspots from read hotspots using concurrent metric changes. - **Mitigation Techniques**: Adjusting internal task priorities, introducing application-level throttling, and monitoring access patterns by merging partition stats with application context.

### Technical Details - **Internal Metrics & Behavior**: In FoundationDB, a storage node retains five seconds of non-durable data within a version tree. The *storage queue* tracks the size of these mutations, while *non-durable versions* (NDV) represents the number of versions not yet persisted to SQLite. When the storage queue increases, the Ratekeeper decreases system rate. When NDV rises too high, system stability degrades. - **Detecting Write vs. Read Hotspots**: - *Write Hotspots*: Both the storage queue and NDV rise simultaneously on a specific process. - *Read Hotspots*: Only the NDV increases significantly while the storage queue does not rise as much. This occurs because FoundationDB prioritizes reads over writes; under heavy read loads, storage nodes stay occupied servicing reads, delaying persistence and causing non-durable versions to accumulate. - **Handling Hotspots**: - *Priority Adjustment*: A code modification was implemented to raise the priority of write tasks relative to reads, preventing write operations from being starved. - *Application Throttling*: Application-level, fine-grained throttling was introduced to restrict reads and writes for poorly performing or misbehaving applications. - *Data Pattern Analysis*: While FoundationDB tracks partition statistics (shards) without application context, application layers have context and can use APIs to retrieve partition stats (extended to include storage and port information). Combining these datasets reveals where data access flows across storage nodes. - *Telemetry Pipeline*: Per-transaction stats are buffered in memory and flushed to disk (or sampled under high throughput). This data is ranked by shard/partition and loaded into an analytical store (Snowflake in the speaker's implementation, with FDB itself being considered by the FDB team) to correlate partition-level reads/writes with application entities and identify extreme skew, such as partitions seeing near a gigabyte per minute.

### Takeaways - Distinguishing between write and read hotspots in FDB can be determined by observing whether storage queue and NDV climb together or NDV climbs alone. - Rebalancing read/write task priorities and applying application-level throttling provide immediate operational relief. - Deep understanding of application data access patterns is critical; merging FoundationDB partition statistics with application context allows engineering teams to identify skewed access and implement fixes such as caching.