AI in the human loop: GenAI in security service delivery

Preeti Ravindra examines how large language models perform when placed directly inside security operations workflows. Many SOC teams adopt AI copilots to boost analyst productivity, yet few measure how these tools change outcomes. Earlier benchmarks often ran outside real analyst workflows, and prior studies used narrow, vendor-native incidents, so this work tests across more data sources and tools within normal SLA limits.

The experiment covered two tasks: drafting key findings for incident reports, and orienting an analyst dropped into an active response. Outputs were generated automatically instead of through repeated prompting. Because analysts collaborate closely, a standard split would leak between groups, so the team used a switchback design that alternated conditions over time. Outputs were scored on editing time, quality, hallucination, contradiction and coverage, using automated checks against investigation ground truth.

Quality and editing time moved together, so better drafts meant less rework. A tuned model lost some general capability and became counterproductive, while base models helped. Missed coverage proved a bigger problem than hallucination, since models often dropped supplied information. The analysis also exposed data gaps where analysts' own findings were absent from model inputs. Practical takeaways include checking whether a task can become simple classification and measuring how much a tool changes workflow and feedback burden.