A ZooKeeper Layer for FoundationDB

### Overview In this talk, software enthusiast Paul introduces a project that implements an Apache ZooKeeper compatibility layer on top of FoundationDB. ZooKeeper is widely used across distributed systems for storing system configurations, handling service discovery, and building synchronization primitives such as leader election, mutexes, and locks. While ZooKeeper scales reads well for small datasets, its write throughput does not scale horizontally, and adding nodes can actually degrade write performance. To overcome these limitations, the speaker engineered a stateless layer that translates ZooKeeper's semantics and data model onto FoundationDB's horizontally scalable, strictly serializable key-value store.

### Key Topics The presentation focuses on analyzing ZooKeeper's core capabilities and re-implementing them over FoundationDB through three main functional components: - **Data Model and Consistency**: Mapping ZooKeeper’s hierarchical filesystem-like node structure (Znodes) and its sequential consistency model onto FoundationDB. - **Ordered Watches**: Replicating ZooKeeper's strict watch notification ordering semantics using FoundationDB's simpler key-watch mechanism and versionstamps. - **Session State and Ephemeral Nodes**: Managing client heartbeats, session tracking, and automatic deletion of ephemeral nodes without introducing dedicated stateful background services.

### Technical Details To construct the stateless layer, the speaker addressed three primary technical challenges:

1. **Data Model Mapping**: ZooKeeper models data as a hierarchical tree of paths with small data blocks and metadata. Instead of implementing a custom hierarchy, the layer leverages FoundationDB's existing folder layer to map Znode paths to key subspaces and serialize native ZooKeeper objects. FoundationDB's strict serializability provides stronger consistency guarantees than ZooKeeper's sequential consistency, resolving ordering challenges out of the box.

2. **Replicating ZooKeeper Watches**: ZooKeeper allows clients to watch paths for creation, deletion, or modification events and guarantees that event order strictly mirrors update order. Furthermore, clients must receive watch events before reading updated data. Because FoundationDB watches only notify when a key value changes without detailed event types or ordering guarantees, the layer implements an indexed per-client event log. When mutations occur, FoundationDB versionstamps order the events within the transaction. A FoundationDB watch key is used simply as a signal for the client to retrieve its ordered pending events from the log. Read requests also consult the event log to prevent race conditions where clients might read updated values before their watch notifications trigger.

3. **Session Management and Periodic Cleanup**: Ephemeral Znodes must be deleted when a client disconnects. While native ZooKeeper handles this with an in-memory background thread, the layer stores session information and heartbeats directly in FoundationDB subspaces, using versionstamps for session IDs. Sessions are indexed by expiration time so that expired sessions can be identified via range reads. To coordinate cleanup across multiple stateless layer instances without a dedicated daemon, instances participate in a lightweight per-second leader election using FoundationDB transaction conflicts on an election subspace. The winning instance executes range reads for expired sessions and performs the idempotent cleanup of ephemeral Znodes.

### Takeaways The implementation operates as a functional proof of concept capable of running all recipes in Apache Curator, the standard library for ZooKeeper synchronization primitives. While not yet recommended for production use, the project demonstrates that complex distributed coordination semantics and stateful mechanisms like ordered watches and session heartbeats can be mapped cleanly into a stateless architecture backed by FoundationDB.