Stop the firefighting and ship again

The short answer

A team that spends its weeks firefighting is not a team with bad engineers. It is a team running a system that produces the same incidents on a loop, and the loop has a small number of causes. Find them with two weeks of an honest incident ledger, then fix them in the order that removes the most repeat pages: usually the deploy path first, rollback second, observability on the flows that matter third, and the shared state nobody owns fourth. Protect a fixed slice of roadmap time while you do it, so the fix does not become the new firefight.

The rest of this page is how I run that, drawn from a CI/CD and IaC transformation for a 25-developer team that was deploying by SSH, taking two to three hours per release, and rolling back one release in five.

Recurring incidents are a system property

When the same class of failure happens every couple of weeks, the cause is structural. Nobody chooses to break production the same way twice; the system makes it the path of least resistance. A deploy that requires a human to SSH in and restart services will, sooner or later, be done in the wrong order. Infrastructure that exists only in the console will drift until something depends on the drift. A rollback that takes an hour means every incident lasts at least an hour. None of that is a people problem, and treating it as one, with more process, more checklists, more careful engineers, does not change the loop.

This matters for how you talk to the team. The first step is to take the blame out of the room so the ledger is honest. If people are worried about how an incident reflects on them, the ledger will be incomplete, and you will fix the wrong thing.

The incident ledger: two weeks, no blame

For two weeks, log every interruption to planned work, not just the ones that got a ticket. What broke, how it was noticed (alert, customer, someone happened to look), how long it took to find the cause, how long to recover, and what the recovery actually was. Include the small ones: the manual restart, the config tweak, the "just re-run the deploy". They are the loop.

At the end of two weeks, group the entries by cause, not by symptom. On the team this page draws from, the documented starting point was deploys that took two to three hours, one in five needing a rollback, rollbacks taking 30 to 60 minutes, and four to five deploy-related production incidents a month. Written down like that, most of the ledger pointed at one thing: the way code reached production. The ledger turns "everything is on fire" into a ranked list of three or four causes, and that list is the plan.

The four usual roots

1. The deploy path. Manual steps, one engineer's laptop, an image tagged latest, a migration applied by hand. Every manual step is a place the same mistake can be made again.
2. No rollback. If undoing a release takes longer than making it, every incident is at least that long, and every deploy is a bet. Fear of deploying is a symptom of this, not of the team.
3. No observability on the flows that matter.Not a dashboard wall; an alert that fires when checkout, or sign-up, or whatever earns the revenue, stops working, before a customer says so.
4. Shared state nobody owns. The database three services write to, the config bucket, the console-built VPC. Drift accumulates there because no change to it is reviewed, and it fails in ways no single team can see.

Most ledgers point at two of these. Almost none point at "the engineers are careless".

Fix in the order that removes the most repeat pages

The order is set by the ledger, but the usual shape is: make the deploy path repeatable, make rollback fast, then observability, then bring the unowned state under version control. Deploy path first because it is the cause of the most entries and because every later fix ships through it. Rollback second because it caps the cost of every incident that is still to come. On the CI/CD transformation, the work was pipelines with reusable workflows across all 12 repositories, blue-green deploys with automatic rollback on failed health checks, and then the slow part: importing 180-plus console-built resources into Terraform over three weeks so drift had nowhere left to live.

The results were deploys under ten minutes instead of two to three hours, rollback in under 30 seconds instead of up to an hour, five to eight deploys a day instead of one a fortnight, and deploy-related incidents down from four or five a month to zero or one. The number that mattered to the team, though, was the rollback time. Knowing you can undo a release in 30 seconds is what made people willing to deploy often, and deploying often is what made each release small enough to be safe.

Protecting roadmap time while you do it

The trap is that the reliability work becomes the new thing that eats the roadmap. Avoid it by fixing the slice, not the calendar: a small platform group, or one or two people rotating, owns the fix work, and the rest of the team keeps shipping through whatever deploy path exists today. As the new path comes online, teams move onto it one at a time. Do not freeze features. A freeze tells the business the fix costs the whole roadmap, which is exactly the perception you are trying to reverse.

Expect friction where the fix changes habits. Test coverage enforcement in CI caused pushback on that team at first; what resolved it was two weeks of pair-programming the tests, plus 400-plus integration tests written as a bootstrap so the gate was reachable, and test utilities that turned common cases into one-liners. The gate stayed at 95% afterwards because the team could meet it, not because it was imposed.

What the team should own afterwards

The test of whether the firefighting has actually stopped is what your team runs without outside help once the work is done. It should be all of it: the pipelines and their configuration, the infrastructure code and its state, the rollback button, the alerts and who they page, and the ledger itself as a habit rather than an exercise. On the team this page draws from, each product team owns its own pipeline configuration on top of shared workflows, developers run their own blue-green deploys and rollbacks from Slack, and no change goes through the console any more.

That is what "regain control" means in practice: not that the incidents are gone forever, but that when one happens, your team finds it first, fixes it in minutes, and goes back to the roadmap. If you are still in the loop and want the ledger done with someone who has run this before,start with the project rescue review.