Open Source Tools To Empower Ethical and Robust AI Systems

### Overview Presented by Alberto Rodriguez and Miguel Fontana (crediting Vicente, who co-developed the initial talk delivered at KubeCon), this session surveys the landscape of open-source tools and frameworks designed to secure artificial intelligence systems across their lifecycle. As generative AI and large language models (LLMs) transition from experimental pilots to production applications, organizations face amplified risks ranging from prompt injection to unintended database deletion or unauthorized script execution. The speakers argue that open-source tools are vital for independent verification, democratizing safety access, and preventing "evaluation gaming" where model creators are the sole evaluators.

### Key Topics The speakers structure AI security tooling into three core categories across the software development life cycle (SDLC): - **Pre-deployment / Evaluation Tools:** Assessing models during training and prior to deployment using red teaming, benchmark suites, and probe-based evaluation. - **Runtime Tools:** Intercepting interactions during live inference via input/output filters, guardrails, and real-time monitoring. - **Knowledge and Governance Frameworks:** Industry guidance, threat taxonomies, and operational adoption roadmaps.

### Technical Details The presentation details several open-source tools, data sets, and frameworks: - **Runtime Guardrails & Filters:** - *LLM Guard (Protect AI):* A runtime Python library that filters inputs and outputs against jailbreaks and injections. Notably, it detects and obfuscates PII (e.g., emails) before dispatching requests to inference providers, restoring original data upon response receipt (noting this can impact streaming responses). - *Guardrails AI:* Provides an open-source hub of reusable input/output validators (supporting custom Python validators or judge models such as Llama Guard) to block or monitor risky outputs. - **Pre-deployment & Evaluation Engines:** - *Garak (Nvidia):* A probe-based vulnerability scanner for LLMs that runs repeated test prompts and detectors to identify toxic output or policy failures across non-deterministic responses. - *Promptfoo:* An evaluation and red-teaming tool focused on model comparison and custom metrics/datasets with graphical reporting. - *Giskard:* Scans models for hallucinations, bias, and prompt injection, and includes the RAG Evaluation Toolkit (RAGET) to generate targeted test suites for retrieval-augmented generation applications. - **Toolkits and Specialized Models:** - *Purple Llama (Meta):* Includes *Llama Guard* (input/output moderation model evaluating 13 MLCommons hazard categories plus code interpreter abuse), *Prompt Guard* (detects prompt injections and jailbreaks), and *Code Shield* (a static analysis library inspecting assistant code generation). - *Adversarial Robustness Toolbox (IBM / Linux Foundation):* Evaluates traditional machine learning classifiers against evasion/adversarial noise attacks (e.g., video classification manipulation). - *AI Fairness 360 (IBM / Linux Foundation):* Implements bias metrics, group/individual fairness estimators, and disparate impact detectors across the training lifecycle. - **Governance Frameworks:** - *OWASP GenAI Security Project:* Evolves the LLM Top 10 into governance checklists, red-teaming intelligence, and an AI security solutions landscape catalog. - *FINOS AI Governance Framework (Linux Foundation):* Caters to financial institutions with detailed risk catalogs and mitigation strategies.

### Takeaways The speakers observe that while dependency deprecation and data set rot remain practical challenges, the open-source AI security ecosystem is maturing rapidly with active corporate backing and community contributions. Engineers can now mix and match open-source tools to secure the entire SDLC—from pre-deployment red teaming to runtime guardrailing—enabling organizations to look beyond vendor system cards and independently verify AI safety.