14 April 2008

Deployment Gone Wrong

Here’s a situation:

Your development team has begun implementing a shared enterprise infrastructure. Nobody on the team has ever done this before and they do not take the time to think about how updates will be handled for the software as more applications begin to use it.

The first version of the shared infrastructure goes into production but it is not yet complete. There are enhancements and bug fixes that need to be done before it is fully implemented. Meanwhile, there are multiple applications being written to use the shared infrastructure. All of these projects are running concurrently with the enhancements and bug fixes for the infrastructure.

There are many things that need to happen in order for this scenario to work with minimal problems and no outages for the applications using the shared infrastructure. Off the top of my head I suggest that you need configuration management, a well-defined deployment process that is controlled and repeatable, regression testing before anything is ever put into user-acceptance testing or production, and a well-defined and controlled build process to start.

What happens if your environment doesn’t have all let alone any of these controls in place? The answer is that the infrastructure will fail in a spectacular way at the least opportune moment. Let’s look at what could happen if just one of these controls, such as a well-defined deployment process, is not in place.

If you do not have repeatable or automated deployment processes, you could end up missing steps and having the deployment fail anywhere from subtle to spectacular ways. If your processes are not automated and also not well documented, you could end up with just one person who knows how to deploy the product. This will put you in an at-risk position if that person is out sick or decides to leave. This could also lead to critical deadlines being missed.

All of this will make the entire department look bad to the customer which can potentially lead to cut funding or lost contracts. The problem for me, as a business analyst, is that I am not managing these risks on any one project; I am managing them from a departmental process standpoint.

Therefore, I must document the potential problems from this lack of process in my risk plan for every project. Consequently, I will do things such as make sure I schedule the user acceptance testing at least several days after the deployment to the testing environment. Another mitigation plan I might use is to make sure that I have an escalation plan for deployment problems and that I implement it immediately to avoid last minute issues. Risk management and mitigation can avoid some problems, but it will never offer a substitute for good processes.

Clearly this is not a preferred solution. Rather, a good solution would be to have a well-defined and controlled deployment process to limit the risk involved. The process must be developed from within to get as much buy-in as possible. That said, it must also be enforced from above to ensure compliance by all.

In some cases, all efforts to get the department to institute good build practices will fail. Unfortunately, this means the business analyst or project manager must be prepared to deal with the risk as best they can.

Photo by Nictalopen at Flickr

Search This Blog