Wednesday

This recording is a block of three back-to-back talks. The first, from Sophos data scientists, looks at why security detectors fail on rare behavior that labeled training data never covers. The team ran isolation forest, k-means distance and PCA reconstruction error over unlabeled command-line and file data to pick the most unusual samples, had a large language model label them, and retrained the detectors. On a harder test set, the command-line detector's ROC AUC rose from about 0.6 to about 0.9. The same approach carried over to RTF and HTML classifiers, using a dual-prompt method where two opposing judges had to agree before a label was kept. No single anomaly method won on every dataset.

The second talk reports on AI agents used by students in cyber defense and incident response competitions, with all teams given the same logged, audited access. Most teams used agents like search engines and gave thin context, while red teams that had prepared agent frameworks did better. Defending cost far more than attacking, and AI-written incident reports contained confident fabrications. The takeaways were to supply context up front, validate outputs, and authorize actions in advance.

The third talk, by Kora Gortney, frames social engineering as a set of influence patterns such as urgency, authority and reciprocity, and tests them against two chatbots in the audience. A bot with an added knowledge base did not reliably refuse more often than a plain baseline, so explicit knowledge alone did not change results much. The talk argues that detecting these patterns with tooling could reduce the load on people.