28 February 2008

Is the BA an endangered species with the growth of Agile?

First, I'm a project focused software developer, team lead, designer, architect, jack of all trades, who has been on projects that have used various methodologies over the years, including of late some agile projects.

I'm not a big blog reader, or a big blogger, but like most people I have an opinion on things, and for some reason that opinion is valued by various people I have met in my life's journey, so I have been invited to provide input into this forum from a technical perspective.

Doubtless, my blog posts will be intermittent, and possibly contentious. So this marks my debut posting for the Better Projects blog. I have been reading of late various blogs around the place (usually when hunting for other information) on the imminent demise of the role of the BA, with the growth of Agile and the perception that the role of a BA is not required in an Agile project.

From a BA community perspective there seems to be a sense of impending unquantified unease, if not one of impending doom. For the IT community, there is growing conjecture on whether the BA is really an unnecessary evil that can be dispensed with in an Agile project.

To paraphrase Mark Twain, rumours of the death of the BA is greatly exaggerated.

To understand why I'm of this opinion, I want to consider four things. What is the business? What is a BA? What is the role of a BA in an Agile project? What is the future of Agile?

What is The Business?
To generalise, the business are focused on two things:
  • Operations; which equates to staffing, training, process compliance, materials, machinery, transport logistics, supply, SLA's, demand or orders, stock levels, etc.
  • Management; the analysis of cyclical trends, identifying markets, advertising campaigns, market share, competitor analysis, capacity, contracts, etc.
For the company to be successful, management has a need for accurate statistics to be derived from their operational areas, and operations has a need for efficient, low waste production, yet retaining flexibility to meet varying levels of demand.

The Business are not necessarily good at identifying where they can improve, how to go about change, or even to identify what process' they do undertake. They often however, identify the need for change but will usually struggle to articulate what this change will look like. This is when they usually turn to the BA.

What is a Business Analyst (BA)?
I will probably be telling the readership of Better Projects how to suck eggs on this one, but it may or may not be insightful to hear the perspective of someone from the outside describing what they believe a BA is. I am happy to have my view corrected though.

A Subject Matter Expert (SME) while often employed under the guise of a BA title, in my mind is not one. The software developer who has grown tired of the never ending chasing of the technological tail and has "retired" to the BA role, is also not necessarily one.

The two main attributes required of a BA are:

1. A facilitator; they act as a communications bridge between management and operations in the business, as well as between the business and technology. Sometimes even as representative of their company, in the communications with external parties. For this they require a number of skills and tools such as being able to:
  • manage meetings with people from a range of disciplines
  • identify appropriate stakeholders
  • networking across disciplines and business units
  • communicate to and alleviate concerns of the various stakeholders with tailored messages to each group
  • manage stakeholder expectations
  • create presentations, and present to various stakeholder groups, of varying size audiences
  • convey concepts, ideas, details
  • generate enthusiasm, sell a vision, keep the dream alive
This requires some knowledge of various disciplines, from the business domain through to the technical domain, as well capability in the tools of the trade.

2. An Analyst; with an understanding of the business domain, the ability to analyse inputs, outputs and stakeholders, map processes, identify deficiencies, envisage and communicate potential solutions, perform cost analysis and projections, understand and represent The Business in its needs, being able to identify and articulate the detail that they cannot, and helping to ensure that the target vision is being met with any deliverable.

But most importantly, a BA is not an island of knowledge. A BA is a member of a multi-disciplined intelligent team, where ego is often a major contributor to internal friction, but where it has no place. They need to avoid being proscriptive in their proposed solutions, and for the best chance of success within a project environment, they need to take a collaborative approach.

Both The Business and the IT team have a dependency on the BA's, but when the BA acts alone and arbitrarily as with any other member of this team, it is to the detriment of the deliverable and all who are involved.

The Business needs to be involved for the whole project life to validate what is being developed and potentially identify new needs in light of what they have seen.

The IT team need easy and ready access to The Business. For many things, the BA acts as a representative of The Business, but for the IT team members, it is often extremely beneficial to also be able to directly interact with the target audience. Ideally (though in practice it seldom if ever happens), all team members would have some exposure, if not hands on experience, within the target environment so that they can have some appreciation of the potential impact of their decisions.

It is also important for the IT team to be reviewing requirements, even if in draft form and contributing to estimates and decisions from the inception of the project. Often the IT team are brought in only after the requirements have been "finalised".

Which neatly segues into the next section.

What is the role of a BA in an Agile project?
There are a growing plethora of urban myths around what an Agile project is and what it can achieve, and this posting cannot do any justice in addressing them, so I will not attempt to. There are also quite a number of criticism's emerging around Agile, some of which are in my opinion quite valid. Again, I am not going to attempt to address them in this post. The following will likely make me sound like a strong advocate of Agile processes. In truth, while I see much benefit from the process approach, I am also somewhat critical of it. But that can be left for another day.

To understand a BA's role in an agile project, it is important to understand what an Agile project is.

I guess the best place to start is the Agile manifesto. There are a number of Agile methodologies, but the two most commonly encountered are probably eXtreme Programming (XP) and Scrum. There are a number of methodologies jumping on the bandwagon of Agile, that are mostly Agile in name only. While it seems to be growing in popularity in the business domain, Six Sigma in my mind is one such example that is not an Agile methodology, despite its coupling of late with Lean, out of which the principles of Agile have their origins.

Essentially, the main focus of an agile project is to reduce the feed back loop from the time something is requested, to the time it is validated. At the micro level, this involves practices such as continuous build environments, Test Driven Development, daily stand up meetings, and Pair Programming up to the macro level where iterations are kept to short durations and feedback elicited from the target user as often as possible.

There is also greater emphasis on interdisciplinary cooperation within an Agile project. The traditional process is horizontally layered, where each layer is the responsibility of a dedicated team of specialists, and they have little interaction with team members of other layers. Typically it would be layered something similar to the following:

The Business (receipt of project)

Production Support and Migration Teams

Testing Team

Software Development Team/Infrastructure Team

Business Analysts (requirements gathering)

The Business (request for project)

In an Agile process, it is more vertical, with the above structure turned 90 degrees. Ideally, you would have a representative from each layer in a collaborative co-located team, and the team as a unit is responsible for seeing one "story" proceed from inception through to completion. In a full XP model, these team members would be swapped between teams by up to twice a day so that everyone has an understanding of all aspects of the project and the key man risk is mitigated.

This does not mean that requirements gathering, or documentation is no longer needed. However, the process of capturing them and the format they are captured in will not be likely to be as per a traditional process. Decisions still need to be made, recorded and prioritised, rules still need to be captured and referenced, processes still need to be mapped, communications still need to be compiled and disseminated.

These are the skills that BA's are meant to be competent in, if not their speciality, and cannot be replaced simply by the developers and testers having direct access to, and talking directly with The Business to validate things for themselves. Those communications will still require facilitation. In a pure XP implementation, there would be no BA, only The Business involved in requirements elicitation and functional and user acceptance testing. However, in reality, even SME's from The Business often lack the skills to adequately do this task, and this is where the role of the BA can help fill this space.

An Agile project is fast paced, consisting of consecutive short sharp development cycles for the members of these sub teams. One of the biggest challenges for a BA is to be able to capture the requirements to a sufficient level of detail for the developer and tester within the current iteration, inside of the required time available, without leaving the developers waiting, while still being available to field any questions with minimal response time from both developers and testers.

If anything it arguably requires a BA to be even more capable and disciplined on an Agile project than with any other methodology.

Additionally, not all solutions involve IT. It is my opinion, that IT solutions should only be seen as a last resort, not because of the human impact factor, but because of the high risk associated with IT project deliverables, not just in cost and schedule but also acceptance and fit for purpose, when often there is likely to be a cheaper, lower risk alternative in business process change.

Unfortunately, we seldom model the risk aspect into our valuations of alternatives. But even with a purely business orientated process change, there is no reason why such an initiative could not also be done using Agile principles.

I should point out though, that I am painting the picture of a BA's role in a large corporate environment. For small web development outfits, who usually do not employ BA's anyway, going Agile for them will not change their perceived lack of need for a BA.

What is the future of Agile?
It is always dangerous to Crystal Ball gaze, but here goes for the next few years.

Many large corporations have commenced using some form of Agile in some of their development. This penetration into the corporate IT industry will continue to increase.

Many small outfits risk tarnishing Agile with cowboy behaviour masked under the aegis of "Agile", when in reality they are not following anything but a chaotic methodology.

The hype will begin to dissipate as per every previous "silver bullet" to hit the industry over the last 50 years, and the Agile zealots will be marginalised and voices increasingly ignored as Agile is absorbed and morphed into something else that deals with distributed development teams of varying mediocrity, and silo corporate entities across which even the smallest of corporate projects find the need to navigate.

The focus of project management will begin to change to be more inclusive of all stakeholders, which can only be a good thing, and there will be an increasing expectation from customers for earlier deliverables of incremental releases, which will change how system testers need to approach regression testing, and production access is modified to allow more frequent migrations. Project managers will have a wider range of tools and strategies from which to choose to apply to a project, and the increasing support of the corporation in using them.

There will be increasing tension between the likes of CMMI/SPICE/V-Model processes and Agile based methodologies, including with the likes of government contractors, as Agile projects struggle for official endorsement and recognition as valid methodologies.

But at the end of the day, there will still be a need in the corporate space for a BA, regardless of the process used on the project.