Packet, Qualcomm, & Mesos: A Window into the Development of the Armv8 Ecosystem

### Overview In this presentation, Ed Vielmetti, Special Projects Director at Packet, discusses the ongoing evolution of the Armv8 (64-bit Arm) server ecosystem and the collaborative efforts between Packet, Qualcomm, ARM Holdings, and various open-source communities to bring enterprise datacenter workloads to bare-metal Arm hardware. Highlighting the 'Works on Arm' project, Vielmetti explains why specialized hardware matters beyond traditional virtual machine abstractions and outlines the comprehensive stack-level engineering required to make 64-bit Arm servers routine and reliable in production environments.

### Key Topics - **Specialized Datacenter Hardware**: Moving beyond the traditional perception of Arm as solely an embedded or single-board computer (such as Raspberry Pi) architecture, modern data centers require high-density compute for workloads like artificial intelligence, machine learning, autonomous vehicles, gaming, IoT, and network function virtualization (NFV). - **Qualcomm Centriq 2400**: Packet demonstrated and discussed Qualcomm's 48-core Arm server running the Centriq 2400 processor. Built on Armv8 (64-bit only) architecture, it features custom core pairs arranged on a system bus ring interconnect, providing high memory bandwidth, dense core counts, and out-of-order execution optimizations suited for threaded, I/O-bound parallel workloads. - **The 'Layer Cake' of Platform Bring-up**: Transitioning a new architecture to production requires methodical bring-up across multiple layers: 1. *Boot Firmware & Baseboard Management*: Configuring low-level firmware, BMC chips, and IPMI for automated provisioning. 2. *Operating System Kernels*: Mainstreaming patches into Linux distributions (such as Ubuntu, Debian, CentOS, and Red Hat Linux) with cooperation from silicon vendors like Qualcomm and Cavium. 3. *Compilers, Runtimes, and Libraries*: Ensuring toolchains properly optimize hardware instructions and algorithms. 4. *Containers and Orchestration*: Developing multi-architecture container support in Docker and porting cluster orchestration tools such as DC/OS and Mesos, along with dependencies like Kafka, Spark, Jenkins, and InfluxDB. - **Upstream-First Community Strategy**: Successfully supporting open-source software requires contributing all fixes directly upstream rather than maintaining downstream forks. Vielmetti details filing bug reports, discovering legacy issues (such as an outdated Google `glog` dependency blocking Mesos compilation on Arm), and tracking fixes continuously across communities.

### Technical Details - **Bare-Metal Provisioning**: Packet provides hourly and monthly bare-metal cloud deployments managed through APIs and tools like Terraform and Ansible, eliminating intervening hypervisors to expose native hardware capabilities (such as ARM TrustZone and TPM support). - **DC/OS Bring-up Demonstration**: Vielmetti demonstrated bringing up a four-node DC/OS cluster on Intel using Container Linux via Terraform in 20 minutes to illustrate the baseline target for porting the identical stack smoothly onto Arm. - **Verification and Continuous Integration**: Transitioning beyond one-off ports requires establishing continuous integration (CI) pipelines that run regression suites on bare-metal Arm servers, avoiding mathematical inconsistencies, catching regressions early, and ensuring performance parity without degrading existing x86 implementations.

### Takeaways - Hardware diversity and dense core architectures provide major price-performance advantages for heavily parallelized workloads. - Full-stack engineers spanning chip design, firmware, kernel drivers, and high-level applications are rare but critical for bringing up new architectures. - Developers can access bare-metal Armv8 systems and track ongoing porting efforts via the Works on Arm initiative (worksonarm.com) to validate open-source software and integrate native Arm CI testing.