Building a Global CyberGrid on Cassandra

### Overview In this talk, Charles Herring, CTO and co-founder at WitFoo, details the architecture and operational philosophy behind building a global cyber grid using Apache Cassandra. The platform aggregates, processes, and coordinates cybersecurity telemetry across hundreds of organizations, ingesting hundreds of gigabytes to tens of terabytes of data daily per organization. Designed to serve diverse personas—including Security Operations Center (SOC) analysts, compliance auditors, insurers, and law enforcement—the system federates data and intelligence across distributed clusters deployed on varied fabrics ranging from Raspberry Pis to multi-cloud and private data centers.

### Key Topics - **Project Scope and Multi-Persona Coordination**: Ingesting massive telemetry streams (syslog, network infrastructure logs, AWS VPC Flow Logs, endpoint agents, and external APIs) to coordinate crime reporting, compliance auditing, and incident response across clusters. - **Dynamic Node Provisioning**: Utilizing a local daemon/agent that coordinates with a central service called "WitFoo Library" to retrieve seed nodes, cluster secrets, and Docker images (such as Cassandra, Kafka, or NLP modules) based on rack, data center, and license key metadata. - **Predestination of Data**: Moving away from the conventional "data lake" approach of storing raw logs to query later. Instead, the team maps out the entire life cycle of every signal at ingestion—defining its downstream transformations, access patterns, and expiration. - **Schema Design and Hotspot Prevention**: Creating Cassandra tables with composite primary keys composed of a partition key (`orgId` and a time-based `partition` UUID) and a clustering key (time UUID of insertion), storing the payload in an unindexed JSON object. - **Application-Level Indexing and Map-Reduce Fetching**: Avoiding native Cassandra secondary indexes to prevent non-linear scaling issues, opting instead for custom partition summaries and targeted partition-level fetches. - **Graph and NLP Processing Pipeline**: Structuring data pipelines through Apache Kafka topics into NLP semantic frames and contextual relationship graphs rendered with Cytoscape.js.

### Technical Details - **Pipeline Components**: Raw ingest streams into an initial Kafka topic, is consumed by an in-memory natural language processor using semantic frames to extract artifacts, flows into a subsequent topic consumed by a graph processor, and writes aggregated units of work and analytics to Cassandra. - **Partitioning and Sizing**: To eliminate partition hotspots, inserting code counts the number of rows written to a partition. Once a maximum threshold is reached, that partition key is retired. - **Indexing and Querying Strategy**: Rather than indexing 100 to 500 fields natively in Cassandra, an array of extracted values (e.g., client IPs, usernames) is stored inside a partition summary row in a separate `partitions` table. When executing queries (such as searching for a username across a time window), the system scans the partition summaries, identifies matching partition candidates, executes full partition fetches, and reduces the results client-side. - **Compression and TTLs**: Disk and network I/O are the primary bottlenecks, making aggressive compression highly beneficial for overall performance. Time-to-live (TTL) settings are configured at insert time per cluster and per object type, taking advantage of Cassandra's native deletion and default compaction mechanisms without heavy operational overhead. - **Data Federation**: Upstream and downstream cluster communication relies on standard REST calls passing JSON representations of incidents, threat intelligence, and search jobs, inserting cross-cluster records cleanly using unique `orgId` and `partition` keys without collision.

### Takeaways Designing distributed systems on Cassandra for cybersecurity requires adapting the data pipeline directly to Cassandra's operational strengths. By enforcing a philosophy of "predestination of data," avoiding native Cassandra secondary indexes in favor of partition summaries, preventing hotspots through explicit row counting, and relying on JSON data exchange with aggressive compression and TTLs, organizations can reliably scale multi-tenant, multi-cluster cyber data grids across heterogeneous hardware.