DevOps Transformation
SSH deploys + rollback terror → <10 min pipelines, <30s rollback, deploy-related incidents near zero.
- Role
- Lead Platform Engineer
- Context
- SaaS / Platform
- Duration
- 4 months
- Team
- 2 platform engineers (me + 1) supporting 25 developers
Why they called
Twenty-five developers were deploying by SSH into a bastion, pulling code and restarting services by hand. A release took two to three hours, one in five had to be rolled back, and a rollback itself took 30 to 60 minutes. Infrastructure existed only in the console. The team shipped once every two weeks and spent much of the time in between recovering from the last deploy.
Tech stack
- GitHub Actions
- Terraform
- AWS ECS
- CodeDeploy
- Docker
- ECR
- S3
- DynamoDB
- CloudWatch
- Slack Webhooks
Business problem and production context.
Business problem
Engineering team of 25+ developers was deploying manually via SSH. Release cycles took 2-3 hours with frequent rollbacks. No automated testing. Bugs were caught in production. Infrastructure was provisioned via ClickOps in the AWS console with no version control or reproducibility.
Architecture decision
Designed and implemented a complete CI/CD pipeline using GitHub Actions with reusable composite workflows. Migrated all infrastructure to Terraform with remote state in S3 + DynamoDB locking. Implemented blue-green deployments on ECS with automatic rollback on health check failures. Set up multi-stage environments (dev/staging/prod) with promotion gates. Added automated testing (unit, integration, E2E) with 95%+ coverage requirements enforced in CI.
The useful proof is the decision surface, not only the result number.
These are the parts of the work that show production judgment: ownership, constraints, rollback, cost, and observability.
Owned scope
- CI/CD, deployment safety, and rollback improvements for teams moving beyond manual releases
- Release health signals and ownership boundaries for production changes
- IaC and automation paths that reduced risky human coordination
Evidence artifacts
What would fail first?
Every project has constraints. The useful work is naming them early enough that rollback, cost, and ownership are designed before an incident.
Manual releases
Release confidence was limited by handoffs, manual checks, and unclear rollback.
Service ownership
Deployment automation needed owners who understood how to respond when a change failed.
Environment drift
CI/CD reliability depended on consistent environments and predictable configuration.
Trade-offs accepted on purpose.
Production risks and how they were controlled.
Small steps, visible changes, fewer surprises.
- 01
Documented existing deployment process: SSH into bastion → pull latest code → restart services → manually verify. Average deploy: 2-3 hours with 1 in 5 requiring rollback.
- 02
Migrated all 12 repositories to GitHub Actions with reusable composite workflows: test → build → push to ECR → deploy to ECS with CodeDeploy blue-green.
- 03
Converted all infrastructure from ClickOps to Terraform with remote state in S3 + DynamoDB locking. Ran `terraform import` on 180+ existing resources over 3 weeks.
- 04
Built multi-stage promotion pipeline: feature branch → dev (auto-deploy) → staging (auto-deploy + E2E tests) → prod (manual approval + blue-green + Slack notification).
- 05
Added test coverage enforcement in CI. PRs below 95% coverage automatically blocked. Wrote 400+ integration tests as bootstrap to get the team past the initial hurdle.
Before and after.
What the team runs now.
The point of the work is what the client's own engineers can change and operate without me.
- The GitHub Actions pipelines and the shared composite workflows across all 12 repositories. Each team owns its own pipeline configuration.
- Terraform for every environment, with remote state and locking, so no change goes through the console any more.
- Blue-green deploys with a one-click rollback from Slack, run by the developers themselves with no platform engineer in the loop.
- The 95% coverage gate in CI, kept green by the team after the two weeks of paired test writing that got them past it.
Rollback path
Rollback was treated as part of architecture: deployment strategy, state changes, owners, and verification.
Cost considerations
The main cost benefit was lower incident/release drag, plus cleaner non-production environment usage.
Observability notes
Signals included deploy frequency, failure rate, rollback time, environment health, and customer-impact alarms.
Related services
Have a similar production problem?
Send the current architecture, bill concern, deployment pain, or GenAI reliability issue. I will help you find the first real bottleneck.
Request the reliability audit