LF Live Webinar: Detroit SRE City: Recapping What We Learnt at KubeCon22

### Overview In this Linux Foundation webinar, speakers Ben and Ted review observability and cloud-native developments from KubeCon Detroit 2022. Framing their discussion around an evaluative quadrant measuring novelty against relevance to OpenTelemetry (OTel), the speakers break down community updates, technical challenges, and related projects across the Cloud Native Computing Foundation (CNCF) ecosystem.

### Key Topics - **OTel Unplugged & Community Needs:** At the zero-day community event, end users shared production experiences and interest in client-side Real User Monitoring (RUM) for browsers and mobile devices. Although end users voiced strong demand for a standardized analytical query language, the speakers noted that the Technical Committee (TC) and Governance Committee (GC) decided this falls outside OTel’s current scope to prevent the project from becoming a monolithic, full-stack observability platform with built-in storage layers. - **OpAMP and Dynamic Configuration:** OpAMP (Open Agent Management Protocol) emerged as a central topic. Designed as a control plane for OTel collectors and clients, it allows centralized controllers to update rules, sampling behavior, and configurations dynamically without requiring process restarts. The speakers highlighted how this helps close feedback loops between telemetry usage and ingestion cost. - **OTel Demo Project:** The community demonstrated an instrumented, polyglot microservice e-commerce application (selling telescopes) featuring synthetic failure flags (e.g., memory leaks). This project functions as an integration reference across diverse languages and platforms. - **Profiling & eBPF:** While CPU profiling and eBPF generated substantial buzz, the speakers cautioned against conflating the two. Profiling (such as continuous pprof or Google Web Profiling patterns) primarily targets cost and CPU throughput reduction rather than resolving tail latency. Furthermore, developers frequently seek backtraces or detailed distributed stack traces rather than raw CPU profiling. - **Kubernetes Kubelet Tracing:** The announcement of kubelet tracing introduces native distributed tracing into the container lifecycle and scheduling workflows, representing a milestone five years in the making. - **Log Stability & Semantic Conventions:** OpenTelemetry logging and event specifications reached stability, completing the original foundational roadmap (traces, metrics, logs). The speakers discussed the differences in consensus difficulty between metrics (which embed partial queries, aggregation bucketing, and push/pull semantics) and logging (which standardizes event structure). - **Cilium & Fluent Bit v2:** Moving beyond core OTel, the speakers analyzed Cilium’s eBPF-driven networking and observability, noting the hurdles of inspecting Layer 7 application protocols across encrypted TLS connections (alongside emerging kernel TLS developments). Lastly, they addressed Fluent Bit v2, which expanded to handle metrics, traces, and OTLP, providing an alternative to the Go-based OTel collector.

### Technical Details - **Sampling Strategies:** Tracing planetary-scale systems historically required extreme downsampling (such as Google Dapper's 1-in-1024 or 1-in-10,000 trace sampling). OpAMP provides the mechanism for dynamic sampling and verbosity adjustments to avoid starving low-traffic services while containing network and storage overhead. - **Collector Architecture:** Fluent Bit v2 offers a C++ alternative supporting OTLP, serving as a check and balance against assuming all deployments must use the Go-based OTel Collector, which faces garbage collection overhead in high-throughput workloads.

### Takeaways With the core three pillars (traces, metrics, logs) stabilized, OpenTelemetry's next evolutionary phase focuses on ease of adoption, operational control via OpAMP, semantic conventions, and expanding into client-side RUM while carefully maintaining project boundaries against scope creep.