BSidesSF 2019

### Overview In this BSidesSF 2019 session titled "Strangeways Here We Come: A Journey from On-Prem to Cloud First with AWS," Victor Clark, Cloud Security Engineer at Insight Engines, discusses the tactical and strategic process of migrating an enterprise product from an on-premises deployment to a cloud-first architecture on Amazon Web Services (AWS). Using Insight Engines' flagship natural language Splunk querying tool ('Investigator') as a case study, Clark shares architecture patterns, infrastructure-as-code practices, and operational lessons learned ('turbo buttons') to build secure, scalable cloud environments.

### Key Topics - **Strategy and Team Composition:** Successfully going cloud-first requires key roles including a lead architect, a hands-on infrastructure engineer/individual contributor, and a dedicated QA testing lead to manage testing pipelines. Clark highlights leveraging cloud provider credits programs, aligning with stakeholder and board preferences, navigating undocumented requirements, and establishing GDPR-compliant data retention policies. - **On-Premises Limitations vs. Cloud Architecture:** On-prem installations struggled with scaling and short customer change windows that hindered feature releases and bug fixes. The application refactored its natural language processing engine from an on-prem component into a multi-tenant cloud microservice, replacing local heavy installs with an API client communicating with AWS-hosted services. - **Automation via Infrastructure as Code:** The architecture leverages roughly 1,000 lines of AWS CloudFormation combined with configuration management tools (such as Ansible, Salt, Chef, or Puppet) and internal CloudFormation references to create modular, reproducible environments for deployments and disaster recovery.

### Technical Details - **Request Flow and Edge Ingress:** End-user queries are packaged with authentication/authorization headers by the API client and sent over an encrypted channel to AWS API Gateway. User authentication and signup/login flows are managed using AWS Cognito with OAuth authorization code grant flows. API Gateway handles request routing, rate limiting, deactivating abusive API keys, and enforcing integration timeouts up to 29 seconds. - **Private Ingress with VPC Link and NLB:** To avoid exposing backend systems to the public internet, API Gateway connects via AWS VPC Link (noting AWS limits accounts to 5 VPC Links) into a private VPC. A Network Load Balancer (NLB) operating at Layer 4 routes traffic across availability zones (`cross-zone` enabled) to the compute tier. - **Containerized Logic on AWS Fargate:** The core natural language engine is containerized and hosted in an AWS Fargate cluster. Public IP access is explicitly disabled in the CloudFormation task definition, using `awsvpc` network mode, IAM task/execution roles, and configured AWS CloudWatch log stream prefixes. Clark advises running container-level health checks rather than NLB-driven health checks to avoid self-inflicted denial-of-service conditions. - **Data and Secrets Isolation:** Relational databases (AWS RDS PostgreSQL) reside in private subnets guarded by strictly hardened security groups (e.g., ingress restricted to TCP port 5432 and egress constrained to the VPC CIDR block). Database credentials and secrets are managed via AWS Secrets Manager with granular IAM policies, eliminating hardcoded repository secrets. - **Monitoring and Alarms:** AWS CloudWatch is used for centralized logging, filter pattern matching (such as filtering for non-200 HTTP response codes to catch outages before AWS notifications), and metric alarms integrated with incident management platforms like PagerDuty.

### Takeaways - Emphasize defense-in-depth: keep backend compute (Fargate) and databases (RDS) entirely in private VPC subnets with zero direct public IP exposure, tightly scoped security groups, and least-privilege IAM execution roles. - Break large CloudFormation templates into smaller, modular stacks to streamline incident response and isolate single-component redeployments. - When building cloud teams, focus on general distributed systems and site reliability expertise, as specific cloud vendor syntax can be acquired quickly across platforms.