Debugging & Tracing and More Between Linux and RTOS on Heterogeneous ARM SoC

### Overview In this presentation, Leo Wang from Aumobio (formerly part of Continental Automotive) discusses practical engineering approaches for debugging, tracing, and diagnostics between Linux and Real-Time Operating Systems (RTOS/AUTOSAR) on heterogeneous ARM Systems-on-Chip (SoCs). Heterogeneous SoCs—typically integrating Cortex-A cores running Linux or Android alongside Cortex-M or Cortex-R cores running an RTOS—are becoming standard in automotive high-performance computers (HPCs), body domain controllers, smart chassis, and cockpit systems. While these SoCs offer advantages over traditional multi-chip, single-PCB architectures (such as reduced power consumption, weight reduction, lower cost, and high resource integration), they introduce significant software integration challenges. Specifically, Board Support Packages (BSPs) from chip vendors often isolate the OS environments and do not account for deep cross-core interaction.

### Key Topics - **Automotive Hardware Evolution:** The shift from single-core and multi-core microcontrollers to multi-chip PCB architectures, and now to heterogeneous SoCs hosting both general-purpose performance cores (Cortex-A) and real-time cores (Cortex-M/R). - **Existing Cross-Core Management Solutions:** Limitations and trade-offs of existing frameworks such as Linux `remoteproc`, Huawei's OpenEuler Embedded MICA framework, and lightweight partitioning/virtualization tools like Jailhouse, especially when RTOS boots first as the primary OS. - **Shared Memory Diagnostics:** Using shared memory mechanisms to bypass slow physical bus interfaces (like CAN or serial) to support rapid OTA updates, crash dump generation, variable inspection, and tracing. - **Safety and Security Isolation:** Implementing robust security boundaries across shared resources using MMUs, ARM TrustZone/Secure World, bus-level access control units, and secure enclaves/islands.

### Technical Details In automotive architectures, RTOS typically boots first and acts as the dominant safety-critical operating system, while Linux initializes later. Troubleshooting RTOS-side issues in production or testing is often constrained by lack of hardware debuggers (like JTAG) and bandwidth limitations of CAN or serial ports. Leveraging Linux on the Cortex-A side allows engineers to monitor and troubleshoot the RTOS side: - **Core Dumps and Snapshots:** Generating runtime or crash core dumps for RTOS via shared memory. Challenges include cache coherency (I-cache, D-cache, and Tightly Coupled Memory / TCM), register state preservation, and unmapped virtual memory (since Cortex-M lacks an MMU). Solutions involve configuring designated non-cached memory buffers, triggering periodic copies to shared areas, or using kernel drivers/remoteproc mechanisms. - **Online Debugging and Calibration:** Rather than traditional GDB breakpoints, runtime debugging focuses on reading and writing memory variables. Linux can interface directly with AUTOSAR calibration standards like XCP over shared memory, replacing slower physical CAN transport layers. - **Tracing and Virtual Terminals:** Implementing tracking logic via shared memory allows fine-grained periodic data sampling that can be modified dynamically and deployed post-SOP via Over-The-Air (OTA) updates. A virtual terminal can be established to inject faults or trigger RTOS-side diagnostic logging during development. - **Security and Access Control:** Access to debugging features must be strictly locked down. Leo notes that `/dev/mem` user-space access is viable only for early proof-of-concepts due to security risks. Production and testing architectures require access controls via ARM Secure World (TEE), bus firewalls/access units, and distinct policy switches between development and Start of Production (SOP).

### Takeaways Heterogeneous ARM SoCs provide high-throughput communication paths through shared memory that can significantly enhance RTOS diagnostics from the Linux environment. However, because debugging features require elevated memory access permissions, system architects must enforce strict isolation boundaries—balancing diagnostic observability during development against automotive functional safety and cybersecurity requirements in production.