Embedded Linux Conference 2013 - The Embedded Linux Problem

### Overview In this talk from the Embedded Linux Conference 2013, an Android Kernel Architect at Intel discusses the practical realities and challenges of maintaining, hardening, and migrating Linux kernels for commercial mobile devices (specifically Intel-based phones and tablets). The core problem—termed "the embedded problem"—is that hardening a kernel within a complete software stack for regulated, production-shipping devices takes significantly longer than the Linux kernel community's release cadence. This gap creates massive inertia, making it extremely difficult to track modern kernels, rebase code, or upstream device drivers.

### Key Topics - **The Embedded Problem Defined**: Hardening a board support package (BSP) and full OS stack requires bottom-up stabilization—from silicon and firmware to HAL, middleware, and user space. Because bugs are uncovered late in the testing cycle across tens of millions of dollars in test resources, teams become tethered to older kernels. - **Organizational Friction and Upstream Disconnect**: Development at Intel is distributed across specialized "feature teams" (e.g., audio, graphics, cellular, Wi-Fi, camera, power management). These teams are evaluated and compensated on bug fixes and delivering user-facing features on schedule, not on upstream contributions. Consequently, "Upstream First" breaks down under time-to-market pressure. - **Continuous Integration Challenges**: With over 500 to 1,000 developers committing 50–100 changes per week to a single integration branch managed via Gerrit and Buildbot, multi-branch or tracking-branch workflows become unmanageable for teams accustomed to treating Git like CVS. - **Migration Drivers and the Auditing Problem**: Customers demand kernels no older than 1.5 years for high-volume manufacturing, forcing periodic major jumps between Long Term Support (LTS) releases (e.g., 2.6.35 to 3.0 to 3.4). However, migrating exposes an "auditing problem": verifying whether thousands of bug fixes and features accumulated in the moldy old kernel are properly preserved in the new baseline without retesting thousands of Bugzilla tickets.

### Technical Details - **History Across Intel SoCs**: The speaker recounts managing kernel stacks across successive Intel platforms: Moorestown (forking Moblin on 2.6.31/2.6.32 with AOSP patches for Cisco), Medfield (MeeGo with 2.6.35 Android cherry-picks for Froyo/Gingerbread), Clover Trail and an unannounced SoC on 3.0 (Ice Cream Sandwich), and Bay Trail / Merrifield moving to 3.4 (Jelly Bean/Key Lime Pie). - **Isolating Backports via Merge Commits**: Rather than cherry-picking backported upstream features directly into the main branch alongside vendor hacks, backports (such as Dave Howells' signed kernel modules or V4L updates) are applied to a dedicated backport branch sharing a common baseline and integrated via merge commits. This allows clean removal when transitioning to a newer kernel base. - **Preventing Contaminated Commits**: Modifying both upstream kernel files and proprietary out-of-tree drivers within the same commit leads to merge conflicts and breaks reverts when tracking Linux stable. The speaker experimented with `checkpatch.pl` enforcement to block commits that span both file categories simultaneously. - **Eliminating Build and Test Blockers**: Header file changes act as interface modifications that frequently break user-space compilation and introduce critical test blockers. Teams are required to ensure user-space builds independently of the kernel tree, and header file changes must be made intentionally costly and scrutinized. - **Out-of-Tree Driver Repositories**: To resolve the auditing problem, the speaker proposes keeping un-upstreamed, fast-moving vendor drivers in separate Git repositories outside the main kernel tree. Instead of requiring driver teams to perform Git rebases against new kernels, the kernel feature team maintains tracking branches containing kernel glue logic, rebasing the shim layer rather than the driver code itself.

### Takeaways - Commercial embedded deployments cannot rely on "Upstream First" without clear time-to-market incentives and dedicated teams. - Kernel teams must prioritize minimal foundational upstreaming (serial console boot to ramdisk, P/C power states, suspend-to-RAM, eMMC persistent storage, and USB gadget for ADB) rather than minor peripheral drivers. - Segregating changes through isolated backport branches, strict separation between core kernel and driver commits, and moving volatile drivers out-of-tree isolates kernel migration complexity and preserves validation history.