GeeCON 2023: Asaf Mesika - OpenTelemetry Metrics: observability’s holy grail

### Overview In this GeeCON 2023 presentation, Asaf Mesika, Principal Engineer at StreamNative, provides an in-depth evaluation of OpenTelemetry Metrics, awarding it a top score compared to traditional libraries. Mesika shares his background in observability at Kotendo, Akamai, and Logz.io, leading up to his work on Apache Pulsar at StreamNative. Pulsar's architecture enables rapid horizontal scaling, geo-replication, tiered storage, and support for up to one million topics. However, monitoring one million topics can create an unmanageable volume of unique time series (up to 100 million). Addressing this high-cardinality monitoring challenge required consolidating Pulsar’s legacy metrics implementations into a modern, unified metrics library, prompting an evaluation of Dropwizard Metrics, Prometheus Java Client, Micrometer, and OpenTelemetry.

### Key Topics - **Limitations of Legacy Metric Libraries:** Dropwizard lacks label/attribute support; Prometheus Java Client is tightly coupled to Prometheus protocol and faces performance bottlenecks under high time-series loads; Micrometer failed to achieve widespread vendor-neutral standard adoption. - **OpenTelemetry Project Architecture:** Comprises four core components across metrics, logs, and traces: API (interfaces for defining and recording telemetry), SDK (implementation including transformations, exports, and auto-instrumentation), OTLP (OpenTelemetry Protocol using Protobuf/gRPC or HTTP/JSON), and the OpenTelemetry Collector (lightweight proxy for receiving, transforming, and exporting data). - **OpenTelemetry Metrics API Concepts:** Core API concepts include Instruments (virtual recording objects), Measurements (recorded numerical values), Attributes (key-value contextual metadata), Meters (named and versioned factories for instruments), and Meter Providers (factory for meters and SDK configuration point). - **Instrument Types:** Core instrument primitives include Counters (monotonic, incrementing only), UpDownCounters (can increment or decrement), Gauges (asynchronous, callback-based values representing independent non-additive measurements such as temperature), and Histograms (recording values like latency to produce statistical bucket distributions).

### Technical Details - **SDK Aggregations and Decoupling:** In OpenTelemetry, instruments act as generic input mechanisms whose values flow into distinct aggregation interfaces (e.g., Sum for counters, Explicit Bucket Histogram for histograms). Buckets and aggregations are decoupled from initial instrument declarations. - **The Power of Views:** Views allow SDK users to customize how instrument data is processed without modifying underlying application or third-party library source code. With Views, operators can: - Override default aggregations (e.g., replacing expensive explicit bucket configurations with cheaper bucket boundaries). - Filter attributes to dramatically reduce memory consumption and cardinality (e.g., dropping source IP addresses from an HTTP latency instrument while preserving HTTP status codes). - Clone instrument data into multiple views with different aggregations simultaneously. - Apply wildcard matching to uniformly reconfigure bucket layouts across all matching instruments (e.g., targeting any instrument ending with `latency`). - **Reading and Exporting Data:** Metric Readers (such as the Prometheus HTTP Server or PeriodicMetricReader) interface with a MetricProducer to pull aggregated metrics from memory. PeriodicMetricReader forwards metrics to MetricExporters (such as OTLP gRPC or OTLP HTTP exporters) on a schedule. - **Advanced SDK Features:** The SDK incorporates Semantic Conventions (standardized attribute naming across logs, traces, and metrics to streamline cross-telemetry correlation) and Temporal Aggregation (enabling Cumulative or Delta aggregation, the latter particularly beneficial for search engines like Elasticsearch/OpenSearch).

### Takeaways - OpenTelemetry Metrics provides a unified, cross-language specification backed by the CNCF and broad multi-vendor support, positioning it as the future standard that will likely deprecate single-vendor or legacy metric frameworks. - The Views mechanism offers unprecedented control over third-party library metrics, allowing operators to prevent out-of-memory errors and curb metric ingestion costs directly at the client side. - Recent performance optimizations driven by low-latency workloads (like Apache Pulsar) have significantly reduced memory allocations, making OpenTelemetry Metrics increasingly viable for high-throughput, latency-critical systems.