Beyond Human Scale: AI and Automation as Force Multipliers in Cyber Threat Intelligence

### Overview In this presentation, members of the AWS Security Incident Response team share lessons learned and architectural methodologies developed while building a hybrid AI security tool for threat intelligence and data analysis. Facing head count constraints when manually analyzing massive volumes of AWS CloudTrail logs across customer security incidents, the team designed a solution that pairs traditional deterministic programming with an AI enhancement layer to extract threat actor actions, reconstruct kill chains, and correlate trends across cases.

### Key Topics - **Operational Discovery**: Identifying repetitive tasks and clear input boundaries across weekly team workflows to establish an automation roadmap. - **Consumer vs. Creator Personas**: Shifting from passive consumer AI tasks (such as code generation or summarization) to creating custom AI-enabled systems and pipelines. - **Threat Technique Catalog for AWS (TTC)**: An AWS-specific extension of the MITRE ATT&CK framework used to categorize and map observed threat actor API calls. - **Decision Criteria for Automation**: Evaluating factors such as output criticality, downstream impact, accuracy versus throughput, acceptable error tolerances, token costs, and data privacy.

### Technical Details - **Deterministic Logic Layer**: Raw CloudTrail logs are initially ingested and processed using traditional Python recursive depth-first search (DFS) algorithms with parallel processing. This layer isolates threat actor activity from benign activity deterministically rather than relying directly on an LLM. - **Graph Transformation & Technique Mapping**: Filtered CloudTrail CSV logs are converted into a graph data structure. Nodes represent individual API calls classified using the Threat Technique Catalog (e.g., initial access, discovery, impact). Paths are traversed from impact actions backward to initial access points. - **Embedding-Based Pruning and Semantic Ranking**: Candidate paths contain noise and benign operations. The team generates vector embeddings for each API call and computes similarity against the impact action to prune irrelevant or read-only calls. Remaining paths are semantically ranked by relevance. - **LLM Synthesis & Validation**: Only after thorough programmatic filtering and ranking is an LLM invoked to select and synthesize final kill chains and summaries. Output quality is validated by comparing LLM-generated summaries against documented incident ticket summaries, combined with custom validation models and human-in-the-loop review. - **Performance**: The hybrid pipeline demonstrated the capability to process roughly 990 million records in under 20 to 25 minutes, identifying shared patterns across hundreds and thousands of customer security cases.

### Takeaways - **Hybrid Architecture Over Pure AI**: Directly feeding raw logs to LLMs or autonomous agents is cost-prohibitive, slow, and prone to non-deterministic errors that trigger noisy downstream alerts. Deterministic layers provide precision and scale, while AI handles contextual synthesis. - **Realistic Data Sets**: Models trained purely on synthetic or red team datasets struggled in production because the data was too clean; real-world messy data is vital for handling noise. - **Trust Determines Adoption**: Incident responders will abandon tools if outputs lack transparency. Explaining tool inputs and high-level logic empowers analysts to trust and validate the results. - **POC vs. Production Scale**: Prototype designs must account for token costs, infrastructure limits, and system permissions from day one, including conducting threat modeling to restrict tool access.