Gracian Crisan: RT troubles, lessons learned, and open questions

## Overview Gracian Crisan from National Instruments presents lessons learned from six to seven years of deployment experience with preempt RT on their real-time hardware platforms. National Instruments builds hardware and software for test, measurement, and control markets, primarily targeting ARM and 64-bit Intel CPUs with a disembodied CPU plus FPGA combo architecture using OpenEmbedded.

## Key Topics

**1. Memory-Mapped I/O Read Stalls (Hardware-Level Problem)** The most significant trouble involves CPU write buffers overwhelming slow I/O devices. CPUs can write exponentially faster than I/O devices can sync, causing buffer accumulation. When a read instruction accesses the same memory region, it blocks waiting for buffered writes to complete, resulting in latency spikes exceeding 400 microseconds. Discovered with U1000/U1008 Ethernet drivers and TPM chips, the root cause is extensive bus fabric with multiple levels of buffering and arbitration. Current workarounds include adding delays after register write stretches (Ethernet driver) or replacing I/O read calls with inline functions that force cache flushes after writes (TPM). The speaker frames this as a common hardware problem and invites community discussion on solutions.

**2. High-Resolution Timer Stacking** Multiple timer leads and timeouts from scattered threads can accumulate into large latencies, particularly with conditional variables using timeouts. The issue manifests as 15-microsecond processing sections stacking up to double the baseline latency. Sebastian submitted a patch shortly before the summit that resolved this by rewriting soft IRQ handling for light armors, demonstrating significant improvement in latency histograms. This problem is now largely solved upstream.

**3. Priority Inheritance Gaps in Pthread Library** Survey of glibc's pthread library reveals only pthread_mutex supports priority inheritance when explicitly enabled. Reader/writer locks, semaphores, and conditional variables lack this support. Conditional variables use non-PI internal futex mechanisms, creating priority inversion risks. A bug report filed in 2010 remains open; previous patch attempts by Darren and others were not accepted upstream. The POSIX standard itself has unresolved questions about signal ordering and conditional variable semantics.

**4. Interrupt Thread Priorities and Priority Inversions** Managing interrupt thread priorities and preventing priority inversions presents ongoing challenges. Example: high-priority watchdog interrupt attempting I2C acknowledgment gets blocked by lower-priority I2C traffic. Proposed solutions include exposing interrupt thread PIDs through procfs or using userspace daemons (such as Red Hat's RTC-TLD equivalent) to monitor and assign priorities to dynamically created interrupt threads after boot.

## Technical Details Analysis uses cyclic test, stress tests, kernel shark traces, and histogram latency distributions. The speaker discusses specific hardware topology effects (PCIe switch counts, bus frequencies) and demonstrates using external oscilloscope references to validate kernel timer accuracy after upgrades. Discussion covers clock source selection (ACPI PM vs. TC), configuration options added for security that impact performance, and instrumentation techniques for detecting runtime access patterns.

## Takeaways 1. Check configuration options and security features after kernel upgrades—they can significantly impact latency 2. Verify clock sources after upgrades; some may be disabled unexpectedly 3. Run reboot tests as a simple yet effective method to catch boot-time and shutdown issues 4. Compare timer traces against external references (oscilloscopes) to distinguish tracing overhead from real performance changes 5. Memory-mapped I/O buffer stalls appear to be a common hardware issue warranting community attention and potential tooling for detection