GeeCON 2024: Wojtek Ptak, Andrzej Grzesik - Tools for simplified large scale systems

## Overview Wojtek Ptak and Andrzej Grzesik present a framework for building scalable, high-performing engineering organizations, grounded in organizational theory and practical experience at Revolute (a fintech platform with 40 million retail customers). The talk moves beyond technology to focus on culture, structure, and tooling that enable rapid, reliable software delivery at scale.

## Key Topics

**Organizational Models** The speakers introduce Westrom's organizational framework, categorizing companies into three types: pathological (power-oriented, low cooperation, messengers discouraged), bureaucratic (rule-driven, formal, processes dominate), and generative/performance-oriented (high cooperation, encouraged collaboration, shared responsibility). High-performing organizations align with the generative model.

**The Knowledge Discovery Problem** Large organizations struggle with "knowledge discovery"—knowing what services exist, who owns them, how to reach the right team, and understanding dependencies. This hidden complexity becomes a major source of friction as companies scale. The solution is implementing service discoverability platforms.

**Service Discoverability Tooling** Tools like Spotify Backstage and Revolute's internal "Tower" provide a centralized catalog of services with critical metadata: ownership, SLOs/SLAs, materiality (importance to the business), and upstream/downstream dependencies. Automation via CI/CD ensures this information stays synchronized and prevents it from becoming stale.

**Systems Thinking and Complexity Management** Using Weinberg's model, the speakers distinguish between randomness (introduced by multiple technology choices, processes, and skills gaps) and inherent complexity (driven by business reality). Success requires managing randomness through standardization while accepting necessary complexity. Organizations should limit technology choices, unify CI/CD pipelines, and ensure consistent skill levels across teams.

**Organizational Anti-Patterns** The talk identifies recurring problems that signal poor organizational health: - Blurred ownership ("why do they commit to our code base") - Blame-shifting between teams ("it's not our problem") - Hero engineers creating knowledge silos - Centralized platform teams making top-down decisions without accountability - Ticket-based bureaucracy replacing direct collaboration - Technology constraints driving business decisions

**Post-Mortems and Incident Management** Blameless post-mortems are critical for organizational learning. Effective post-mortems follow a template covering what happened, impact, detection, response timeline, root cause (via five whys), and actionable lessons. They should be rapid, accurate, open to the organization, and tracked to ensure action items are completed. Near-miss incidents should be treated similarly.

**Decision-Making and Communication** The RACI model (Responsible, Accountable, Consulted, Informed) clarifies roles in decisions and reduces unnecessary meetings. Teams benefit from having clear rules about who must be involved versus merely informed. RFCs and design reviews should be open, with participation encouraged for teams dependent on the service being designed.

**Dependency Hygiene** Teams must actively track and understand their dependencies, including SLOs/SLAs of upstream services and awareness of planned deprecations or migrations. Visibility into these dependencies, enabled by discoverability tooling, allows teams to anticipate and respond to changes.

**Architectural Refactoring at Scale** When systems become painfully slow to build/release or expensive to maintain, refactoring is necessary. Using the Theory of Constraints, identify the single biggest bottleneck, focus on removing it, then repeat. Avoid large rewrites; instead, drive decisions around a single metric (e.g., mean time to production). Revolute's example: a 3,200-endpoint monolith with 20 contributing teams and >1-hour build time was modularized by focusing on reducing deployment time to 10–20 minutes.

## Technical Details

**Revolute's Architecture Approach** The simplified service topology includes: front-end, front-end API (endpoint definitions), flow services (domain orchestration), state services (domain logic and persistence), and gateway services (infrastructure separation). All Java services run on Kubernetes with PostgreSQL as the operational database and event bus. CI/CD automation configures monitoring, governance, and observability out-of-the-box, reducing manual setup.

**Monolith Transformation Case Study** - Initial state: 3,200 endpoints, 500 consumers, nearly 20 teams, build time >1 hour - Goal: mean time to production ~1 hour - Approach: focused on modularization to shorten deployment cycles - Result: deployment time reduced to 10–20 minutes per team

**Practical Recommendations** - Make boundaries explicit (ownership, dependencies, SLOs) - Centralize knowledge in wikis or dedicated platforms - Practice continuous ownership refactoring whenever confusion arises - Automate service discoverability integration into CI/CD - Shorten cycle times to reduce fear and enable rapid iteration - Apply stress to systems to identify breaking points and drive improvement - Challenge assumptions regularly; post-mortems and peer reviews are learning opportunities

## Takeaways

1. **Bad systems beat good people.** Success depends on designing organizational and technical systems that enable fast, correct decisions, not on individual heroics. 2. **Generative culture drives performance.** Shared responsibility, clear boundaries, and psychological safety (blameless learning) outperform rigid or blame-driven organizations. 3. **Reduce randomness through standardization.** Unified tech stacks, processes, and skill-building limits unnecessary complexity and enables scaling. 4. **Shorten feedback loops.** Fast builds, deployments, and post-mortems create conditions for learning and rapid iteration. 5. **Make knowledge explicit.** Hidden assumptions and tribal knowledge are a scaling bottleneck; centralize and automate wherever possible. 6. **Ruthless focus drives results.** High-performing teams apply stress, identify constraints, and systematically remove them rather than redesigning everything at once.