Monitoring: A New Approach - Tom King, The Linux Foundation

### Overview In this presentation, Tom King discusses the evolution of open-source monitoring systems, comparing legacy tools to modern architectures. Drawing from four decades of experience in the embedded space and his practical work managing build infrastructure for the OpenEmbedded project, King outlines the core objectives of monitoring: alerting, metric collection, and visualization. He contrasts monolithic and RRDtool-based systems with a modular approach that mirrors traditional Unix philosophy—composing small, specialized daemons for collection, time-series storage, and visualization.

### Key Topics - **Goals of Monitoring**: Effective alerting systems should notify the right people according to severity, avoid waking up system administrators for non-actionable events, and ideally alert before resources crash. Metric collection tracks parameters such as packet throughput, drops (useful for detecting DDoS activity), CPU utilization, memory leaks, swap usage, and request latency. Visualization enables long-term and short-term trend analysis. - **Legacy Monitoring Stacks**: King reviews Monit (used for process monitoring and automatic restarting, e.g., restarting dnsmasq on OpenWRT), RRDtool-based tools (MRTG, Cacti, Munin), and Nagios Core. While RRDtool solutions provide vertical integration and consistent graphs, they rely on pre-allocated databases, fixed time windows, and circular buffers that cause data loss via aggregation and averaging. Nagios offers distributed monitoring and plugins but suffers from complex configuration. - **The Modern Decoupled Approach**: The modern stack separates the workflow into interchangeable components: - **Collectors**: Tools such as collectd (older, higher overhead) and Netdata (low overhead, ~1–2% CPU, minimal disk impact) gather system and network telemetry (including SNMP). - **Time-Series Databases (TSDBs)**: Systems like Prometheus and InfluxDB store timestamped key-value pairs using proprietary query languages rather than SQL, preserving raw historical fidelity. - **Visualizers & Alerting**: Grafana serves as the de facto visualization dashboard, supporting queries across Prometheus and InfluxDB as well as integrations with alert channels like Slack, Telegram, PagerDuty, and webhooks.

### Technical Details King demonstrates this modern pipeline on an OpenEmbedded build farm. Netdata runs on a central file server (using NFS/iSCSI) and individual builder nodes, capturing I/O wait times, CPU spikes, RAM consumption, and tmpfs usage. Prometheus scrapes Netdata metrics roughly every 15 seconds. King uses Grafana dashboards to inspect real-time metrics during an OE world build—identifying an initial CPU spike during recipe parsing, followed by memory accumulation and distinct I/O spikes when writing build artifacts. Grafana allows adjusting query time windows from the last 15 minutes to historical runs from previous days, revealing why specific build architectures consumed significantly more RAM and ran longer.

### Takeaways Legacy RRD-based monitoring discards granular historical data through averaging and circular buffers. In contrast, adopting a modular architecture consisting of lightweight collectors (Netdata), dedicated time-series databases (Prometheus/InfluxDB), and rich dashboards (Grafana) provides non-destructive data retention, granular zoom into historical events, and flexible notifications across modern communication tools.