Atmosphere 2018: Andy Davies - FEATURE TOGGLES: THE GOOD, THE BAD, AND THE UGLY

### Overview Andy Davies explores the operational realities, benefits, and severe pitfalls of using feature toggles in software engineering. Drawing from catastrophic real-world failures, successful enterprise turnarounds, and personal development experiences, the presentation illustrates how improper toggle management can destroy a business, whereas disciplined implementation accelerates continuous delivery and reduces technical debt.

### Key Topics - **Catastrophic Failures and Anti-Patterns**: The presentation begins with the collapse of Knight Capital, an American high-frequency trading firm. During a deployment across eight servers, one server failed to receive the update. Knight Capital reused an existing feature toggle named for an obsolete feature ("Power Peg") to activate new functionality ("SMASH"). When enabled, the un-updated server triggered defective eight-year-old dead code that could no longer count executed orders, firing orders indefinitely. A panicked rollback to the old code exacerbated the problem across all eight servers, ultimately costing $450 million in 45 minutes and bankrupting the firm. - **Toggle Lifecycles and Types**: Davies categorizes toggles into distinct operational roles: - *Release Toggles*: Facilitate continuous delivery and phased rollouts. - *Experiment Toggles*: Short-lived toggles tied to metrics and A/B testing (e.g., button colors). - *Operations Toggles*: The only category that should live long-term, used for system degradation or load shedding (such as disabling heavy background tasks during database spikes). - *Permission Toggles*: Strongly discouraged as an anti-pattern; user permissions belong in dedicated authentication and authorization systems (such as IdentityServer or Keycloak). - **Binding Phases**: Toggle states can be bound at compile time, startup, periodic polling intervals, or per-activity checking. Davies warns against compile-time flags due to deployment delays when emergency changes are needed. Periodic background polling with an in-memory store is recommended for dynamic flags to avoid per-request network latency.

### Technical Details & Architecture - **Hewlett-Packard Transformation**: HP’s printer firmware team (FutureSmart) struggled with a monolithic codebase of 10 million lines across 800 developers, releasing only twice per year with an eight-week build/test feedback cycle due to excessive branching and compile-time `#ifdef` toggles. Merging consumed 25% of developer time. By migrating to trunk-based development and startup feature flags loaded via model-specific configuration files, HP reduced feedback cycles to two days on actual printer hardware and achieved continuous deployment readiness. - **Architectural Isolation via Branch by Abstraction**: Rather than scattering conditional toggle checks throughout the codebase, teams should use design patterns like branch by abstraction, dependency injection, factories, or decorators. For front-end code (such as React), components can be wrapped to isolate conditional rendering to a single location. - **Case Study (Phased Email Service Rollout)**: At a previous company, an unstable SOAP-to-Exchange service was rewritten using RabbitMQ and decorator-pattern workers behind an `IEmailConnector` interface. A feature toggle routed through Active Directory groups enabled a gradual, team-by-team rollout over 6–8 weeks. Early beta testers uncovered order sequencing bottlenecks, leading to the creation of separate direct and bulk queues before full adoption. Once 100% traffic was stabilized, the toggle and old code were deleted.

### Takeaways Davies outlines five core rules for managing feature toggles: 1. **Never reuse a feature toggle**: Always introduce a distinct toggle for new functionality. 2. **Enforce short lifespans**: Treat toggles as temporary scaffolding and delete them alongside old code immediately after rollout (operations toggles being the rare exception). 3. **Name toggles explicitly**: Avoid obscure ticket numbers or vague labels; names must communicate exact intent. 4. **Maintain sound architecture**: Feature toggles are not an excuse for messy code; encapsulate toggle checks in single integration points. 5. **Monitor toggle usage**: Track how frequently toggles are queried and evaluate rollout metrics to know precisely when code is safe to clean up.