Beyond the Perimeter: Retrofitting VPC-SC at Enterprise Scale

### Overview In this talk, Akshay (Lead, Production Security at Wayfair) and Priya (Senior Cloud Security Engineer at Wayfair) discuss their experience retrofitting Google Cloud's Virtual Private Cloud Service Controls (VPC-SC) across an enterprise scale brownfield environment consisting of more than 5,000 GCP projects. The initiative was motivated by common challenges surrounding compromised service account keys and subsequent data exfiltration. While traditional network firewalls and Identity and Access Management (IAM) govern *who* can access data, IAM does not dictate *where* data is permitted to go. VPC-SC serves as a data exfiltration control to enforce data perimeters and specify boundary conditions for data movement.

### Key Topics - **VPC vs. VPC-SC**: Clarifying that VPC handles private compute network connectivity, whereas VPC-SC establishes data exfiltration boundaries. - **Core VPC-SC Constructs**: Understanding service perimeters, restricted services, access context policies, and ingress/egress controls. - **Design Trade-offs**: Comparing monolithic perimeters (simpler operational management, broad external defense) against multi-perimeter architectures (granular isolation, cross-perimeter bridges). - **Phased Implementation**: Transitioning from design and service prioritization to dry-run discovery, log pattern analysis, exception handling, and final enforcement.

### Technical Details - **Implementation & Infrastructure as Code**: Wayfair opted for Terraform rather than the Google Cloud Console or gcloud CLI, citing the need for auditable, scalable, and version-controlled Infrastructure-as-Code (IaC) governance. - **Service Selection**: The speakers advised tackling high-volume, high-complexity services first (using BigQuery and Google Cloud Storage [GCS] as primary examples) rather than lower-complexity services, ensuring sensitive data assets are protected early. - **Discovery & Centralized Dry-Run Logging**: VPC-SC was initially enabled in dry-run mode, which logs perimeter violation errors without actively blocking traffic. Because per-project Cloud Logging is difficult to audit across thousands of projects, logs were routed via log sinks to a centralized BigQuery dataset or storage bucket for analysis, aided by Google's VPC-SC visual dashboard. - **Categorizing Traffic Patterns**: Access patterns were grouped into four distinct categories: 1. *User Compliance*: Enforcing context-aware access and verifying device management through Google Workspace so that only managed devices receive access. 2. *Trusted Applications*: Preferring service account impersonation or Workload Identity instead of allow-listing static service account keys. 3. *Vendor/External Applications*: Validating legitimate business need and integrating with Third-Party Risk Management (TPRM) so access is tied to contract lifecycles. 4. *Google Service Dependencies*: Accounting for inter-service behaviors and platform dependencies. - **Edge Cases and Platform Limitations**: - *Storage (GCS)*: Violation error logs report bucket/resource names, but VPC-SC allow-lists must be configured at the project level, requiring separate project lookups. - *BigQuery*: Exporting VPC-SC error logs into a BigQuery dataset can trigger VPC-SC violations that itself requires allow-listing. Additionally, queries spanning multiple resources outside the perimeter might not enumerate all referenced resources in error logs. - *Cloud Build*: Calls from Cloud Build into GCS buckets within the perimeter are blocked because Cloud Build runs from Google-owned GCP projects. - **Scalable Exception Management**: To avoid constantly modifying Terraform policies across 5,000+ projects, Wayfair utilized Google Groups. An automated workflow linked with SSO adds identities into Google Groups, which then sync to the VPC-SC access policies (taking roughly 5 to 10 minutes to propagate).

### Takeaways - **Enable Early**: Retrofitting VPC-SC onto established brownfield architectures requires sifting through millions of logs; establishing perimeter rules early in cloud adoption is significantly easier. - **Automate Hierarchy Enrollment**: Platform automations must natively enroll new GCP resources into the perimeter rather than treating security as an add-on. - **Clear Communication & Advocate**: Maintain alignment between platform and security teams, communicate timelines before and after enforcement, set up clear incident reporting workflows, and file feature requests with GCP for product edge cases.