Release management

Most ICT companies or departments face some kind of release problem. You have a product, be it a software package or a data warehouse, which you maintain and improve. You need to keep the production environment(s) stable for the users, but they also require bug fixes and new features and this brings problems with it.

Usually, it's pretty difficult to apply fixes and improvements to the production environment as soon as they're produced. You need to at least test the new features first, partly manually, and you need to check that your fixes don't break anything else. Not only that but in many companies, new version deployment brings some kind of fixed administration cost with it. Basically, every deployment costs something. Which raises the question, how often should you release a new version? After leading our Semanta development team for a number of years, I've a few thoughts about this question which I believe don't only apply to core programming teams.

For most managers, the go-to response to software quality problems is to decrease version release frequency. The logic behind this is that the more time spent on a version means the more time available for testing and that the fixed costs of a release (aka excruciating pain) are borne less often. There are several problems with this reasoning.

Firstly, your users and clients are likely to make you produce intermediate versions if the interval between set versions is too long:

"Project X simply cannot work with version 2 which misses features α and β, that are supposed to appear in version 3 in half a year's time. So please, give me version 2 with α and β in a week, otherwise we'll miss that huge business opportunity!"

 

 

And intermediate versions will increase your release frequency in an uncontrolled way. You're likely to find yourself quasi-releasing under pressure and relying on a fervent prayer for quality assurance. Interestingly, in my experience no amount of the business users' ex-ante agreements on the low frequency of versions will change this. Faced with a business opportunity that needs α and β soon, they will make you do it no matter what they said a week ago. Business people are not paid to be consistent.

Secondly, the release process is a process and for most processes, practice makes perfect. People work well if there is a recognisable heart-beat to their work. After twelve monthly releases, you’ll find your team going like a Swiss watch on the thirteenth one. Some of the release costs may really be fixed costs, but most aren't.

A shorter release cycle also makes the goal visible, smaller, easier to grasp. In other words, small releases tend to be better defined. There are usually only a few features added plus a handful of bug fixes. The change is easier to analyse, one can make a better guess about the side-effects of changes one made a week ago than about stuff that was done half a year ago.

If you believe that there are strong reasons why your organisation needs long release cycles, you may want to check if this is not a symptom of a deeper problem. On the technical side, a lack of automation often pushes up the cost. Releases (and the related upgrades) should be as automatic as possible. Script languages are your friends. This applies to both testing and deployment. We have a set of automatic scripts that click through Semanta’s Encyclopadia before every release. I wish the test coverage were higher, but nevertheless the scripts run for 40 minutes and take a lot of pressure off our testers. A deployment process should never be a document describing 50 error prone manual steps. Without automation, it's easier to deploy a new application than upgrading an old version. So invest time into taking the pain out of that by automating as much as possible.

Very often organisations believe they're unfit for short releases because they produce ‘large updates’. "You know, I really need to work on this for 5 months before I can even start thinking about passing it to the public." This mentality needs to be challenged. It's often very healthy for projects to try to generate value-added earlier rather than later. A lot of software can be used productively, long before it's feature-complete. Not only that, the final version of the “large update” will profit immensely from the feedback given by users of the feature-incomplete versions. Early exposure to real users is better than two tons of analytical documents.

 

Conclusion

It may sound counter-intuitive and even a bit masochistic, but if your releases are causing you pain, do them more often. Being able to release quickly is an indicator of the overall health of your testing, upgrading and deployment process. Trying to release faster will make you fix the underlying causes of the pain.

Moreover, new features and bug fixes will typically save your customers time and/or money (which is why they wanted them in the first place). Bringing new versions out earlier, therefore has a direct economic value.