Search This Blog


13 April 2008

Best Practice

How often have you sat in a meeting and heard the term Best Practice used to support an argument, or read an article using the term Best Practice as a justification for a particular assertion? It is increasingly becoming an industry buzz word and in doing so is losing its original meaning as it gets used out of context.

Best Practice describes processes and techniques that measurably deliver the best outcome against any other known process or technique for a given criteria. In most disciplines, this is evidence based and when the term is used, it is invariably also done in conjunction with a citation to a peer reviewed study that supports that claim of being Best Practice.

In many industries, Best Practice is somewhat compromised due to the cost of doing studies over statistically significant population sizes. So you have the associated industries funding these studies which can lead to dubious results being published. A classic example of this that most people will be aware of is the tobacco industries involvement in studies into the health impact of smoking. From the opposing conclusions currently being published almost monthly into the health impacts of mobile/cell phones, some people are beginning to accuse the telecommunications industry of attempting to influence published results in a similar manner as the tobacco industry has been shown to have done.

There are some things in IT that we can verify. As an example, using method x vs. method y to retrieve data from the database will result in a difference z in performance. While sometimes difficult and not without its flaws and prone to influence by vested vendor interests such as Sun vs. IBM vs. Microsoft, benchmarking is a measurable approach to ascertaining and confirming Best Practice.

Similarly, longitudinal studies of many projects for the causes of defect injection can identify practices that either contribute to or alleviate the injection of these defects through extrapolation and inference techniques. These studies then contribute to what we understand as Best Practice. Unfortunately, there are many such claims to Best Practice derived from anecdotal evidence, rather than derived from actual case studies.

Some aspects that are termed Best Practice are in reality, simply industry practice. If I'm writing COBOL, everything is upper case. This convention is by industry practice. Whereas if I'm writing in Java, I have different syntactical conventions, and different again with C#. Unfortunately, the term Best Practice is widely abused in the IT realm, and often what is simply a convention for a given community, is erroneously referred to as Best Practice.

In choosing what process to follow under what circumstances that meets Best Practice is fraught with difficulties. The IT industry is unfortunately in a unique position, where the term Best Practice in the traditional sense, usually does not apply for many aspects of a project. Doing a significant project usually runs into the millions of dollars and is a unique undertaking. To ascertain whether one process is better than another in the traditional sense would require the same project to be repeated a number of times using the same process and different people to accommodate for the variance that people introduce, and then repeated for each process being compared, with all other factors kept as similar as possible and for the deliverables to be similar. Financially, it is simply not feasible to cater for such a combinatorial explosion in approach. Often an IT project ends up not being financially justifiable as it is, much less being able to support the same project being done multiple times to ascertain what is Best Practice. So academics attempt to extrapolate from observations, which has limitations in what conclusions can be derived.

Many of the published figures that do exist supporting different processes being Best Practice are based on results obtained from final year students being tasked with following a certain process and compared against the results of fellow students following a different process. These results rarely allow for differences in outcomes that would arise from how the students have been taught up to the time of the comparison, and usually do not have sufficient population sizes for the results to be significant. Similarly, they invariably do not cater for the level of experience in the subjects. You would expect a junior developer to derive greater benefit from a much shorter feedback loop than a senior developer. By how much at this stage is probably academic. But if you base your conclusions that short cycle iterative development is Best Practice based on the result of studies on students, then you already have a bias built in to support that conclusion. This is not to say that short cycle iterative development is not a good practice. But you need to be wary in the interpretation of supporting studies and understand the inherent bias in such studies.

So we have the situation where everything from CMMI, Six Sigma, eXtreme Programming, etc all claiming to be Best Practice and all having figures that support them.

A similar situation exists in choosing how best to notate requirements. Which is Best Practice: UML, use cases, stories, scenarios? Or are they all equivalent? How often have you read articles saying the use of UML and use cases in a CMMI process is Best Practice or the use of story cards and XP is Best Practice, with neither citing a reference to support their claim of Best Practice or citing studies that have built in bias?

When you next read of someone claiming process x is Best Practice with no supporting citation, view it as an opinion, rather than an industry Best Practice. Often that opinion is reflective of the current industry trends, but being the flavour of the month still does not making something Best Practice. If there is a citation to a supporting study, understand the bias in the study that may influence the results and lend itself to the conclusions being made and if possible determine who is funding the study to ascertain what spin may be on the results.

Essentially I'm asking that you do not take at face value any claims of what is Best Practice, whether supported by studies or not. And in your own articles and arguments, be conscious of what Best Practice implies and use it appropriately. If we as an industry become more rigorous in our use of the term, then it may actually start meaning something again. But until then, it will increasingly become a meaningless industry buzz word, if it is not already.