From Zip-and-Pray to Infrastructure as Code: Modernizing a Platform With No CI/CD

The starting point: everything was manual

The platform worked, and it served real customers. But under the hood, operating it looked like this: to ship a change to a Lambda, you ran a shell script that zipped a folder, then you uploaded that zip to the AWS console by hand — the script even had the S3 upload line commented out, so there was no artifact history at all. Backend services had a little more structure, but the theme was the same everywhere: changes were applied by a human clicking through a console.

• No CI/CD. A push to the repository did nothing; deploys were a manual ritual, and only the people who knew the ritual could ship.
• No infrastructure as code. Every database, load balancer, NAT gateway, stream, and IAM role had been born by hand in the console and was remembered — or forgotten — in someone's head.
• Secrets bundled into deploy artifacts. Production database passwords and API keys were baked into the zip files that got uploaded, on runtimes as old as Python 2.7.
• No cost attribution. Nobody could answer "what does this service cost?" or "what is this client's footprint?" because nothing was tagged and nothing was mapped.

Step 1 — Put everything into Terraform

The core of the transformation was moving the entire infrastructure into Terraform — not a token module or two, but the whole estate: compute, networking, storage, streaming, security, and monitoring. The end state is the entire estate in Terraform across 75 modules, organized into clear domains, with two workspaces that keep development and production in lockstep.

The payoff of this is hard to overstate. Once infrastructure is code, a change is a pull request: reviewable, diff-able, and reversible. A "plan" shows you exactly what will change before it happens. Dev and prod stop drifting apart because they are generated from the same modules with different variables. And — critically for the cost work that followed — you finally have a complete inventory. The map exists. Anything not in the map is, by definition, a candidate for deletion.

To keep it maintainable at that scale I standardized on a single configuration-object pattern per service: all of a service's knobs — CPU, memory, task counts, autoscaling targets, environment wiring — live in one typed block instead of scattered across the codebase. That is what later made a change like "portal API from 31 tasks to 1, then to 3 across availability zones" a two-line edit with a reviewable plan, rather than a risky console session.

Step 2 — Automate the deploys

With the infrastructure declarative, the application deploys needed to become automatic too. Every service repository got a GitHub Actions pipeline wired to the same simple contract: push to the dev branch and it builds, tests, and deploys to the development environment; merge to the production branch and it ships to production. No zips, no console uploads, no tribal knowledge.

This is more than convenience. Automated pipelines mean every deploy is identical, logged, and attributable to a commit. Rollbacks become "revert the merge." Anyone on the team can ship safely, not just the person who remembered the ritual. And because the pipeline is the only path to production, the old habit of hand-patching a running service — the thing that silently creates drift and mystery outages — simply goes away.

Step 3 — Make cost and configuration visible

The last piece was turning the infrastructure from a black box into something you can inspect — both technically and financially. On the config side, that came for free with the Terraform migration: every setting is now readable in one place, in version control, with the history of why it changed. On the cost side, I activated cost allocation tags across the estate, so spend can finally be sliced by service, environment, and client.

That visibility is what made the entire cost-reduction effort possible and, more importantly, defensible. Instead of guessing, every optimization could be measured against Cost Explorer broken down by tag, verified in CloudWatch, and recorded in a living audit trail with the dollar impact of each change. "We think this is cheaper" became "this line item went from $4,210 to $1,580, here is the query that proves it."

You cannot optimize what you cannot see, and you cannot safely delete what you cannot reproduce. Infrastructure as code gives you both — the visibility to find the waste, and the confidence to remove it.

Why the order mattered

It would have been tempting to jump straight to cutting costs. But every safe deletion, every rightsize, every "this has been idle for years" call depended on this foundation being in place first. The Terraform inventory is what surfaced the orphans. The CI/CD is what made changes reversible enough to be worth attempting. The cost tags are what turned hunches into evidence. The modernization was not a detour from the cost work — it was the thing that made the cost work responsible instead of reckless.

What I would tell anyone doing this

1. Codify before you optimize. The inventory you get from moving everything into Terraform is worth more than any single change you could make by hand.
2. Make the pipeline the only path to production. The moment hand-patching is possible, drift and mystery return.
3. One config object per service. At scale, scattered settings are how you end up with 31 tasks nobody remembers sizing.
4. Tag everything for cost from day one. "What does this cost?" should be a query, not an investigation.
5. Treat forgotten resources as the default failure mode. Anything created by hand will eventually be abandoned; anything in code stays accountable.