Using Yocto to Build an IoT OS Targetting a Crossover SoC

## Overview Ryan Fairfax, an Engineering Lead at Microsoft, presents three practical approaches for using Yocto (an embedded Linux build system) to develop operating systems for crossover System-on-Chip (SoC) devices. Drawing from three years of work on a Linux kernel-based IoT OS, he explains how to build complex, heterogeneous systems combining application processors with microcontroller cores using Yocto's extensibility.

## What Are Crossover SoCs? Crossover SoCs represent a hybrid approach combining two traditionally separate embedded CPU classes onto one die: - **Application processors** (Cortex-A class): highly capable CPUs with clock speeds in the hundreds of MHz to GHz, memory management units, networking stacks, and internet connectivity. Typically run Linux or Android. Cost: tens of dollars. - **Microcontrollers** (Cortex-M class): feature-limited CPUs designed for real-time, predictable applications (sensors, motors, simple displays). Typically run real-time OSes like Zephyr or ThreadX. Cost: a few dollars, less than 1% networked.

Crossover SoCs merge these capabilities on a single chip (e.g., STM32MP1 with dual Cortex-A7 and Cortex-M4; NXP IMX8 with quad-core CPUs and real-time cores). Market drivers include lower cost of ownership, avoiding complex real-time Linux development, and leveraging mature Linux security and networking stacks.

## Real-World Use Case: Azure Sphere The product referenced runs multiple execution environments on different cores: a Pluton security core (dedicated Cortex-M4 for secure boot, key management, device authentication), a TrustZone-based security monitor on an A7 core, Linux on another A7 core, and customer real-time applications on additional Cortex-M cores. This heterogeneous architecture creates significant build complexity because components target different CPU architectures and run different OSes/firmware.

## Key Challenges Linux and most build systems assume SMP (symmetric multiprocessing)—homogeneous cores where any application can run on any core. Crossover SoCs break this assumption: cores are heterogeneous, peripherals attach to specific cores, and multiple OSes/firmware components must be orchestrated. Standard build systems are not designed for this asymmetric multiprocessing scenario.

## Three Yocto Integration Approaches

**1. Recipe-Level Overrides (Simplest)** Override compiler flags for individual recipes to target different architectures. Example: build a bootloader for Cortex-M4 by explicitly setting compiler flags instead of using Yocto's defaults for Cortex-A7. - Pros: Simple, works for self-contained recipes. - Cons: Doesn't scale; breaks with shared libraries (e.g., serialization libraries built for both firmware and Linux), creates cryptic GCC errors, complicates dependency management.

**2. Custom Classes (Mid-Level)** Leverage Yocto's existing BB_CLASS_EXTEND mechanism to build recipes multiple times for different configurations. Define custom classes (e.g., "security_fw") that override compiler flags, set architecture and toolchain, and specify dependencies. Recipes using these classes build once for each target. - Pros: Builds recipes for multiple targets; opt-in (clear error messages); dependencies and packaging largely work automatically. - Cons: Higher cognitive load for new developers; complex cross-architecture dependency graphs require workarounds (S-State class tweaks). - May require custom GCC/binutils for bare-metal compilation (Linux compiler includes Linux-specific defines that break firmware builds).

**3. Multi-Config (Newest)** Tell Yocto to build multiple "machines" (logical CPUs) simultaneously using separate machine configuration files. Uses verbose but consistent syntax: `multiconfig:machine:recipe` and `mc_depends` for cross-machine dependencies. - Pros: Minimal recipe modifications; continued investment and improvement in Yocto; handles dependencies down to task level. - Cons: Verbose syntax; no guardrails (will attempt invalid configurations and fail cryptically); dependency sharing across architectures is limited.

## Key Lessons Learned 1. **Design for debugging**: Log build target architecture and logical target clearly; hours lost to cryptic GCC errors when assumptions are wrong. 2. **Keep targets isolated**: Prevent accidental library/header leakage between targets (e.g., POSIX headers in firmware accidentally linked into Linux, causing infinite loops on asserts). 3. **Don't assume one compiler**: Arm may have 32-bit microcontrollers and 64-bit Linux CPUs; hybrid architectures (Arm + RISC-V) are emerging. 4. **Keep as much in Yocto as possible**: External Python scripts wrapping bitbake slow builds and complicate debugging; integrating everything into Yocto recipes improves performance and maintainability.

## Conclusion Yocto's extensibility makes it viable for heterogeneous embedded systems despite its original SMP Linux focus. The choice between approaches depends on project scope: recipe overrides for simple cases, custom classes for sophisticated multi-architecture builds, and multi-config for standard Yocto-first workflows. Real-world experience shows that careful design for debugging and target isolation is critical when building systems with multiple OSes and heterogeneous CPU architectures.