2024 - John Gillis - Let's build an AI Security Research Team using Python

### Overview John Gillis, an automation engineer at Adobe on the Adobe Cyber Defense Center (ACDC) team, presents an architectural approach to building an AI security research team in Python to handle repetitive Tier 1 SOC analyst tasks. Tier 1 analysts often perform monotonous triage—evaluating initial detections, querying VirusTotal or Shodan, executing web searches, and drafting investigation summaries for Tier 2 analysts. Gillis details his progression from using conversational multi-agent frameworks to designing a structured, assembly-line architecture that passes JSON objects sequentially between specialized agents.

### Key Topics - **Limitations of Conversational Multi-Agent Frameworks**: Gillis initially experimented with CrewAI, where a manager agent directs sub-agents via direct messaging. While easy to set up workflows, this chat-based model proved slow, expensive due to extensive token consumption for inter-agent dialogue, and prone to hallucinations or looping. - **The Assembly-Line Architecture**: Rather than treating multi-agent workflows as conversational "group projects," Gillis advocates an assembly-line model. Each agent performs a highly specific function, ingests a structured JSON object, and passes an enriched JSON output downstream. - **Function Calling and Tool Integration**: Agents interact with external security utilities (such as Shodan, VirusTotal, and custom search scrapers) via function calling, where schemas dictate parameter passing and the local application executes the function on behalf of the LLM. - **Multi-Model Cost and Context Optimization**: Balancing performance and cost by leveraging high-capability models (like GPT) for planning and large-context, inexpensive models (like Gemini 1.5 Pro) for large-scale summarization.

### Technical Details The assembly-line pipeline operates across sequential stages: 1. **Step 1 & 1.5 (Detection Ingestion and Decoding)**: Ingests raw detection alerts (containing file names, hashes, IPs, or Base64-encoded PowerShell). An agent converts the detection into a structured JSON object. Non-LLM tasks, like decoding Base64 strings, are handled by native Python functions rather than LLMs. A subsequent prompt analyzes and breaks down the de-obfuscated command line. 2. **Step 2 (Investigation Planning)**: A more powerful GPT model ingests the enriched JSON and generates a step-by-step investigation plan. Each step defines a title, description, target IOC, IOC type, and the designated tool, removing guesswork for downstream agents. 3. **Step 3 (Tool Execution via Function Calling)**: An agent iterates through the research plan steps, triggering external tools via function calling. In this pattern, the LLM determines arguments based on provided JSON schemas, while the local Python runtime executes the actual API requests and returns the output to the LLM. Tools used include Shodan (IP reputation/geolocation), VirusTotal (hash analysis), and a custom "advanced search" tool that generates Google dorks, scrapes the top three results into markdown using Firecrawl, and summarizes the content. 4. **Step 4 (Final Synthesis)**: The full investigation log is passed into Gemini 1.5 Pro (utilizing its large context window) to synthesize the findings into a concise, Tier 1 research report for Tier 2 analysts.

### Takeaways - **Favor Structured Data Over Freeform Agent Chat**: Passing strict JSON schemas between agents significantly reduces hallucination and token waste compared to conversational agent-to-manager chat models. - **Deterministic Code for Deterministic Tasks**: Simple operations like decoding Base64 should remain in native code (e.g., standard Python functions) rather than delegating to an LLM. - **Scalability**: Systems can scale horizontally (adding pipeline steps/agents) or vertically (adding tools to an individual agent). - **Development Strategy**: Gillis recommends starting development with the most intelligent available models to establish a reliable baseline before optimizing costs or experimenting with smaller or open-source models (such as Llama).