NeuroCTI - A Custom LLM For CTI: Benchmarking, Successes, Failures & Lessons Learned

The talk starts from the appeal of language models for cyber threat intelligence. They can pull structured information out of free-form reports, such as ATT&CK technique mappings and knowledge graphs, in ways that regular expressions cannot. The speaker argues that sending sensitive CTI to commercial cloud services exposes more than the text itself, which makes a case for running models locally.

The middle section covers the practical work. Fine-tuning with low-rank adapters can run on consumer GPUs if the base model is small enough, but benchmarking proved harder than training. A gamified labeling workbench was built to collect human ratings of summaries and entity markings, and an attempt to use an LLM as a stand-in rater did not work. Converting public report collections from PDF was messy, so a commercial model was used to clean about 10,000 reports for roughly 38 dollars, and a 7B Mistral base model was tuned on the result. A translation benchmark was used to justify choosing Mistral over other options.

The closing part describes integration with MISP. The model is served behind a REST API, so MISP can fetch a report, extract indicators and tags, and attach them to an event, with a human reviewing output because of hallucinations. The code, models and tooling were published for reproducibility. The speaker's takeaway is that the community should build shared datasets and local models instead of depending on sending its data to cloud providers.