Immutable Data Infrastructure Behind The Firewall • Quinton Anderson • YOW! 2016

The speaker discusses the challenges of managing a large, complex Hadoop and Teradata data warehouse infrastructure within a bank's firewall, including increasing costs, poor developer experience, and upgrade risks. To address these issues, they adopted a strategy of immutable infrastructure built upon Open Source components like OpenStack, Docker, and Mesos, achieving significant cost reductions and improving the developer experience. However, they acknowledge that long-term cost-of-change benefits are still to be realized, pending further observation and workload growth.

Infrastructure Challenges

• 00:00:36 The bank's existing data warehouse infrastructure suffered from high unit costs, particularly for platform upgrades, coupled with a poor developer experience, slow provisioning, and inadequate infrastructure contracts. These challenges hindered scalability and agility, impacting both cost and risk profiles.

Immutable Infrastructure

• 00:14:54 The speaker advocates for an immutable infrastructure approach, where every component, including firmware, operating systems, and configurations, is treated as code under version control. This codification allows for automation, continuous delivery, and enhanced reproducibility, ultimately improving the efficiency and reliability of the infrastructure.

Open Source Stack

• 00:14:59 The adopted technology stack primarily relies on Open Source components, including OpenStack, Docker, Calico for networking, and Mesos for resource scheduling. This choice enables cost savings through commodity hardware and eliminates vendor lock-in, while promoting flexibility and community support.

Continuous Delivery

• 00:17:19 The system emphasizes continuous delivery, where any change to the infrastructure or application is treated as a pull request in GitHub. This workflow promotes collaboration, reproducibility, and simplifies automation, providing a robust deployment pipeline for all components.

Migration & Testing

• 00:25:15 The migration process from the existing environment to the new immutable infrastructure is carefully managed through network separation, VLAN tagging, and a staged rollout. Testing involves multiple stages, including Docker image verification, functional tests in a virtualized environment, and deployment through development and staging environments before production.