Who You Gonna Call? Taming OpenClaw's Rogue AI Agents With OpenTelemetry and Tetragon

### Overview In this presentation, Henrik Rexed (introduced as Henri Gratzer in the transcript), Cloud-Native Advocate at Dynatrace and CNCF Ambassador, addresses the rapid explosion and critical operational challenges of OpenClaw (also referred to as OpenCloak/OpenGoat). While OpenClaw acts as an AI gateway connecting chat channels (e.g., WhatsApp, Telegram) with various LLMs and automated tools, its swift adoption has introduced two massive operational hurdles: runaway token costs and severe security vulnerabilities. The talk outlines concrete mechanisms to monitor and mitigate both using OpenTelemetry (OTel) observability and Tetragon eBPF kernel enforcement.

### Key Topics - **OpenClaw Architecture & Token Inefficiencies**: OpenClaw builds prompt context by assembling four layers: `soul.md`, `agent.md`, installed skills, and session memory/conversation history. Because these components are injected on every turn, a short user prompt can easily consume 60,000+ tokens per interaction. Unchecked session histories, unbounded skills, and automated heartbeats exacerbate token consumption. - **Cost Mitigation Strategies**: Optimizations include model tiering (reserving high-cost models like Opus for complex tasks and local models for coding), capping tokens via configuration, issuing commands like `/compact`, `/reset`, and `/new`, replacing heartbeats with scheduled cron jobs, using output-minimizing skills (such as "caveman"), and offloading context to dedicated memory tools (e.g., Mem Palace, Agent Memory). - **Observability with OpenTelemetry**: Monitoring token usage, system health, and tool execution requires telemetry. Users can utilize either the built-in CLI `OTel diagnosis` plugin (which produces 42 metrics and basic execution traces) or Henrik's custom OTel OpenClaw plugin. The latter integrates with an OpenTelemetry Collector, follows OpenTelemetry semantic conventions, captures granular tool execution data/bytes, and correlates host metrics (CPU, memory, disk). - **Security Risks & Ecosystem Vulnerabilities**: Security audits by CrowdStrike, Palo Alto Networks, Trend Micro, and Token Security uncovered major flaws in OpenClaw deployments. Risks include unauthenticated WebSocket exposures leading to credential and token leakage (over 1.5 million API tokens leaked; 135,000 instances exposed), prompt injection over public channels like WhatsApp, bypasses of command authorization, and malicious extensions on ClawHub (such as 341 malicious skills found, with 335 linked to a single actor). - **Runtime Security Enforcement with Tetragon**: Because rogue skills and prompt injections ultimately trigger operating system and kernel events (spawning processes, modifying permissions, reading credential files, making outbound network calls), Linux eBPF provides deep visibility and blocking capabilities. Using CNCF Tetragon tracing policies (via Kprobes, trace points, Uprobes, and LSM hooks), operators can detect and block malicious OpenClaw behavior.

### Technical Details - **Tetragon Policies**: Henrik implemented nine custom Tetragon policies targeting OpenClaw runtime anomalies: tracking attempts to read sensitive credential files, detecting unexpected privilege changes, blocking destructive commands, intercepting shell command injection, spotting malicious package installations, and restricting outbound network connections. For network egress, a Kprobe on `tcp_connect` inspects socket arguments; if an outbound connection targets an unapproved external IP outside the local subnet, Tetragon triggers `sigkill` to terminate the process before packets leave the system. - **Telemetry Pipeline**: The monitoring setup funnels OTLP traces, metrics, and logs from OpenClaw, log output from Tetragon via the collector's `filelog` receiver, and system performance via the `hostmetrics` receiver into an OpenTelemetry Collector, which normalizes and forwards security events and token metrics to Dynatrace.

### Takeaways Running autonomous AI agent gateways like OpenClaw in production or home labs without controls exposes operators to extreme API bills and system compromise. Operators should run OpenClaw on Linux, inspect and limit skills, continuously track token usage using OpenTelemetry, and deploy eBPF-based runtime enforcement like Tetragon to detect and terminate suspicious process execution and unauthorized network egress at the kernel layer.