When There’s no Route Forward (Or So You Fear)

Note: This is a reflection on project work done at one of my previous employers, covering a period from late 2000s to early 2010s.

Agile, automation, DevOps and “continuous everything” are on everyone’s lips these days, yet very few people stop to ask why the industry moved this way. For many years, software projects followed a then- familiar routine:

• Thorough upfront planning, repeated before each major milestone.
• Long stretches of development behind closed doors.
• Mostly manual QA, with some automation likely built on a custom testing framework that one of the developers wrote from scratch.
• Rare releases, perhaps once a year or even every two years, preceded by careful testing and code freezes that sometimes lasted months.

We now know this was far from ideal, but it worked for a long time because it matched what the world expected. The constraints of that era explain a lot:

• Hardware limits: multi-core CPUs only reached consumers about 10 years ago (mid 2000s).
• Unreliable connectivity: I was an early broadband adopter in my country 14 years ago (2002), when “broadband” meant roughly three times dial-up speed or about 16 KB/s.
• Slow provisioning: setting up test environments meant manual installs (yes, from CDs or DVDs), and only after management approved the hardware expense.
• Immature virtualization: VMs had existed since early 2000s, but until recently VMs were barely easier to configure than physical machines, and their performance was underwhelming.

This was how things were done across our project and most others in the company. Nobody questioned the process until after we actually went live with the product and that’s when things began to fall apart.

For roughly two years, things went from bad to worse. The tension between the development and operations teams escalated into something close to open war, and management’s attempted fixes alienated people on both sides. In the end, most members of both teams left for other companies.

With a few years of distance now, it’s clear that with the way things were going the collapse was inevitable, but then it could also have been avoided. So what went wrong?

• Feature planning: Before every major milestone, we spent 3-4 consecutive days discussing features, and we developers built most of what was agreed. About a year later, when the updated product shipped, the problems started to show. Some features weren’t needed at all, because Ops had requested them without knowing how the product would really work. Others were “buggy”: not from careless coding, but because the product behaved the way the developer imagined, not the way the operations person expected.
• Test automation: We blamed the lack of QA time for the buggy releases and tried to introduce automation. We built a full high-level testing engine that ran command-line calls and checked the responses. It could never cover the product’s complexity, and by the time it was mostly finished, the product had moved on. Eventually QA spent most of its time maintaining the engine instead of testing new features.
• Performance problems: We initially took the blame for several performance issues. Later root-cause analysis pointed elsewhere: reporting scripts that dumped entire databases at once, and a badly configured monitoring setup that effectively amounted to a denial-of-service attack. Different members of the operations team had installed these without telling anyone.
• Open conflict: At the peak, emails flew daily claiming that our software was buggy and unreliable. Meanwhile we were helping to fight fires and growing more frustrated, convinced we were doing our best to no avail. Looking back, I think everyone had a point, but at the time there was no quick or easy path to peace.

Eventually management stepped in and made things worse. They decided the root cause was our project management process, so we were told to move from “Waterfall” to Agile, with Sprints and regular Scrum meetings.

This was a step in the right direction in principle, but the execution was poor. We were already in firefighting mode, constantly reviewing, fixing and working around problems, and now we spent an hour or more every day in stand-ups that added little value. As the development team shrank with departures, this became the new normal.

What did Agile actually change for us?

• We had Sprints, but the workload stayed the same. Priorities may have become a little clearer, but tasks were still handled whenever a developer was free, and the backlog kept growing.
• We released no more features than before, because Operations “could not handle them”. The goal of a deliverable at the end of each Sprint was never met. We did ship frequent emergency fixes, but they had nothing to do with Sprint timing.
• We still held week-long planning sessions about the next year’s tasks, which undermined the whole point of Agile.

In short, Agile did little for us, given how we adopted it and how we kept interacting with Operations. The development team took the change as an undeserved punishment, and most people resigned within the following year.

What should have been done? I’m convinced now that we should have adopted DevOps. With the right mindset, many things were within reach:

• Merge Development and Operations so that they shared ownership of the production environment, in the literal sense of being in the same office and under the same manager.
• Set up a Continuous Integration pipeline to free up QA time.
• Build a fast, frequent, automated deployment process.
• Replace manually configured servers and patchy scripting with proper configuration management.

We could have saved the project if management had made better decisions. Instead, everyone fell into “saving face” and “shifting blame” modes, and they probably just didn’t know of better ways of working.

Until our very last day, we developers believed that writing the best possible code was the key to a good product. We also felt that even if we did our jobs well, management and Operations might sabotage the result, so the fight couldn’t be won. A few years on, I’ve learned that good code can only take you so far.

That’s it for today. Thanks for reading!