Slop Spotting, Using Rules to Detect AI Slop for Bug Bounty

Open-source projects are being flooded with AI-generated vulnerability reports. The speakers point to curl, which ended its HackerOne bug bounty in early 2026 after a surge of plausible-looking submissions. These cited real functions and convincing proofs of concept, yet described behaviour the code did not have. Checking each one by hand consumes scarce maintainer time and makes life harder for honest researchers.

The talk introduces Slop Spotting, a triage method that treats a static analysis rule as a validity test. An LLM turns each report into a Semgrep rule and revises it in a loop until the rule matches the report's proof of concept. The rule is then run against the affected and fixed versions of the code. A rule with matches in the vulnerable version suggests a real issue. A rule that finds nothing, or cannot be written at all, points to hallucinated code or a contextual bug that static analysis cannot see. In a small curl experiment, most genuine reports produced rules that matched only the vulnerable code, while most slop returned zero hits. One overly broad rule produced 38 findings, and reports closed as not-a-vulnerability were hard to classify, since that call depends on a project's risk appetite.

The speakers stress the limits: the sample was small, Semgrep's C support was weak, and dynamic bugs resist static rules. Their takeaway is a fast yes-or-no signal for triage, and a plea for bounty programmes to publish their outcomes so the community can learn how triage is changing.