Hackers Can Help: Open Technical Problems in Investigative Journalism

Brandon Roberts, a self-taught data journalist who works with newsrooms, opens by arguing that many technical projects aimed at news miss the mark. He points to bias detectors and troll-hunting tools that tend to be vague or unreliable, while records-request tooling, a practical need for reporters, is scarcely present in public code repositories.

He then walks through the usual data journalism pipeline: gathering material through scraping or public records requests, pulling structured information out of PDFs and scanned pages, and assembling datasets. Each stage has gaps. Scrapers on government sites break on JavaScript, pop-up alerts and slow page loads; PDF extraction tools work only on clean layouts; OCR struggles with large leaks such as the Panama Papers archive and with handwritten complaint logs; and campaign finance forms vary from outlet to outlet, so no single template fits. He notes that much existing machine learning serves commercial problems rather than these.

He closes by urging technical people to build shareable tools and datasets, and to partner with local newsrooms and academics, while being transparent about bias and mindful of budgets and exclusivity terms. He lists groups such as NICAR, OpenNews and Big Local News as entry points for getting involved.