Lightning Talk: Streaming and Processing Edge Vision Data in Real Time

# Overview Joyce, head of developer relations for Viam (an open-source platform for building physical AI), discusses why computer vision pipelines that work perfectly in controlled environments fail at the edge and presents three design patterns for optimizing edge vision workloads without streaming data to the cloud.

# Key Topics Joyce identifies core obstacles: CV pipelines are slow, expensive to run, and demand substantial compute and bandwidth—challenges magnified in real-world edge deployments. Vision Language Models (VLMs), while impressive, are particularly problematic for real-time actuation; even models designed for local execution like Moonbeam can take several seconds to respond.

Rather than solve problems by deploying larger models, Joyce advocates optimization through three design patterns:

**1. Continuous Local Inference** Run models locally and send only events or metadata upstream. She demonstrates this with retail shelf monitoring: instead of streaming video to headquarters, a lightweight YOLO model runs locally to detect when shelves need restocking. The system sends only event signals when items run low, triggering either local actuation (like a light) or system-wide alerts. Frigate is cited as an open-source example running YOLO for person detection, integrable with Home Assistant, and capable of running completely offline without cloud connectivity.

**2. Sensor Fusion** Use inexpensive sensors as low-power watchdogs to trigger compute-intensive CV operations on demand. Joyce illustrates this with whale protection: environmental sensors detect when a boat enters certain zones or reaches speed thresholds, triggering the camera and model only when needed. This reduces compute costs and false positives while working reliably on spotty satellite internet.

**3. Edge-First Systems** Systems designed to function entirely offline. Joyce's bathroom queue example (QuickQ app at stadiums) runs fully locally on mobile devices, displaying wait times peer-to-peer without cloud dependency. Cloud sync is optional and deferred for non-time-sensitive analytics only.

# Technical Details Across all examples, the stack centers on YOLO running on TensorFlow Lite—optimized for single-board computers and lightweight enough for local execution. Hardware acceleration options include Raspberry Pi AI hats or Coral TPU modules. Data transport uses gRPC for fast typed API calls and WebRTC for low-latency peer-to-peer communication. Viam handles orchestration and integration of models, sensors, and actuators.

# Takeaways • Optimize down, not up: event-driven architectures and lightweight models accomplish more with less at the edge • Send only events and metadata upstream, not raw video • Enable offline operation for remote locations and unreliable connectivity • Use open standards to scale from Raspberry Pi to industrial fleets • Combine sensor data with CV to improve accuracy and reduce false positives