Why Won't My CPU Sleep? Debugging CPUIdle Mysteries on ARM SoCs

### Overview Presented by Dhruva (Texas Instruments) and Kevin (BayLibre), this presentation explores the complexities of CPU power management on ARM64 SoCs, focusing specifically on debugging CPUIdle and suspend-to-idle (S2Idle) states. While the fundamental goal of CPUIdle is to save power by idling when no tasks are running, achieving deep idle states involves complex interactions across Linux device trees, kernel frameworks, and ARM Trusted Firmware (TF-A) executing at elevated privilege levels (EL3).

### Key Topics - **Linux CPUIdle and Suspend-to-Idle:** Understanding the role of the scheduler, CPUIdle drivers, and governors in selecting idle states based on predicted sleep duration versus waking latency. Unlike suspend-to-RAM, suspend-to-idle freezes user space, suspends drivers, and keeps IRQs enabled without CPU hotplugging, allowing CPUIdle to select the deepest available state. - **Power State Coordination Interface (PSCI):** The standardized ARM specification governing power management calls between EL1 (Linux) and EL3 (TF-A), including platform-coordinated versus OS-initiated modes and parameter validation. - **Device Tree Configuration:** How idle states, bit-encoded PSCI suspend parameters (both original and extended formats), and state hierarchies (core, cluster, system) are exposed to the kernel. - **Timekeeping Dependencies:** The role of architected ARM timers, clock sources, and clock events in power management transitions.

### Technical Details & Case Studies - **Case Study 1: Missing Device Tree Idle States & Firmware Panics:** On an upstream BeaglePlay board running mainline Linux and Buildroot, suspend-to-idle failed because no idle states were defined in the device tree. The CPUIdle driver failed to probe, defaulting to the architecture-specific `WFI` instruction without calling `tick_freeze`, leaving timers and softIRQs active. After adding idle states to the device tree, the firmware panicked because the PSCI suspend parameter was unrecognized. The issue was resolved by implementing the `validate_power_state` callback in TF-A to map the parameter to a platform retention state. - **Case Study 2: Silent CPUIdle Rejection Due to Local Timers:** When configuring deeper power-down states with the `arm,psci-suspend-param` and `local-timer-stop` flag, the CPU local architected timers are powered down. Linux silently refused to select the state because no alternative wakeup timer existed. The fix required extending an upstream driver for an always-on domain timer to register clock source and clock event providers, enabling Linux to switch to a broadcast timer before entering deep idle. - **Case Study 3: The System Fails to Wake:** In suspend-to-idle on an SoC with a separated main domain and always-on domain, a low-power co-processor (e.g., Cortex-M4 or R5) and timer woke the SoC and prompted TF-A to restore the main domain. However, because no hardware interrupt reached ARM's Generic Interrupt Controller (GIC), Linux saw no pending wakeup event and immediately instructed TF-A to go back to sleep. Fixing this required ensuring an interrupt was properly routed to the GIC to trigger CPUIdle's resume logic. - **Debugging Strategies:** Standard JTAG debugging often disconnects when CPU power domains shut down or inadvertently prevents idle entry because debug stepping acts as an interrupt. Traditional `printk` fails when serial consoles power down. Recommended alternatives include toggling GPIOs monitored with a logic analyzer, using `trace_printk`, and capturing Ftrace traces using `trace-cmd`, Kernel Shark, or Perfetto.

### Takeaways Debugging ARM64 low-power states requires close coordination between Linux kernel drivers, device trees, and vendor-specific firmware hooks in ARM Trusted Firmware. Developers must ensure alternative always-on broadcast timers are registered when local timers stop, correctly encode PSCI parameters according to ARM specs, and ensure wakeup events trigger GIC interrupts. Future improvements include enhancing idle state tracing in visualization tools and enabling shared trace buffers between firmware and kernel spaces.