GeeCON 2023: Rafal Foltynski, Nick Palmer - Determinedly deterministic and how it can help you

### Overview In this GeeCON 2023 presentation, Rafal Foltynski (Raf) and Nick Palmer from financial exchange LMAX present the architecture, design principles, and performance benefits of deterministic systems. Given the high-throughput, low-latency requirements of institutional financial trading—operating with sustained rates of hundreds of thousands of orders per second and strict price-time priority execution—handling shared state across concurrent threads or distributed microservices introduces severe complexity and latency jitter. The speakers explain how isolating business logic into single-threaded, deterministic execution pipelines backed by ring buffers, journaling, and snapshotting delivers predictability, fault tolerance, and extreme performance.

### Key Topics - **Definition of Determinism**: A deterministic system is one where a given initial state and input sequence will always yield the exact same result with zero randomness, enabling reliable replayability. - **Architectural Antipatterns**: In standard microservice architectures (e.g., separate gateway, user service, and matching engine), state is smeared across services and updated asynchronously, breaking deterministic replay and complicating ordering. - **Core Requirements for Determinism**: Three fundamental rules must be followed: record all incoming inputs, update state using only a single writer, and produce no non-reproducible side effects. - **Ring Buffers & The Disruptor**: Using ring buffers (such as the LMAX Disruptor) enforces strict event ordering and sequencing, permitting efficient inter-thread communication and fast replay. - **Journaling & Snapshots**: Events are journaled to disk before execution ("if it didn't journal, it didn't happen"). Snapshotting in-memory state and buffer sequence positions avoids replaying history from the beginning of time during recovery. - **Verification via Binary Comparison**: Determinism can be formally verified by taking a base snapshot, replaying recorded journals, taking a subsequent snapshot, and comparing them for binary equivalence.

### Technical Details & Pitfalls - **Common Determinism Breakers**: - *Wall-clock time*: Calling `System.currentTimeMillis()` introduces nondeterminism. Instead, events must be timestamped once upon entry into the ingress ring buffer, using that timestamp as the single source of truth. - *Unordered collections*: Iterating over structures like `HashMap` produces non-deterministic iteration order across snapshot reloads. - *Side effects & blocking I/O*: Direct database queries, RPC calls, HTTP requests, or JMX manipulation pollute the core processing path. Any persistence or database storage must be pushed downstream to asynchronous, idempotent consumers. - **Single-Writer Performance Optimization**: - Lock-free single-writer execution eliminates lock contention and synchronization overhead (contrasting multi-threaded locks or atomic constructs verified via JCStress). - Isolating CPUs and pinning the single application thread via tools like `hwloc` (`lstopo`), `cset`, or `tuned` prevents kernel rescheduling and reduces latency jitter, crucial for sub-80-microsecond responses. - Ring buffers provide natural back-pressure when thread capacity is saturated. - **Replication & Scaling**: - Replicated state machines clone the sequenced input events to backup replicas before processing, allowing live replicas to run concurrently without duplicate external outputs. - When a single thread reaches CPU capacity, systems scale through state partitioning (sharding) along low-cardinality discriminators (e.g., independent order books per currency pair). - Backward compatibility in event schemas is maintained using protocol versioning (e.g., Protobuf or custom versioned serializers) while retaining old deserializers.

### Takeaways Designing core stateful services as deterministic, single-writer state machines significantly reduces architectural complexity by eliminating locks and concurrency bugs. Decoupling I/O and persistence to downstream consumers allows single threads pinned to dedicated cores to process massive workloads with minimal jitter. Furthermore, combining sequenced journaling with periodic snapshots provides seamless fault tolerance, active-passive replication, and the ability to reproduce and debug exact production issues locally.