Agile software development has emerged as one of the most popular software development methodologies in the recent past. Much of the rise has to do with the failings of other methodologies like the waterfall model and the bureaucracies present in implementations of prescribed methodologies like CMM. The principles of the Agile movement strive to match business needs with development realities and reduce waste as much as possible.
I have used Agile development practices in a few projects as well as observed how it is used practically in other projects. Using Agile concepts has served me well in many of these projects. Projects that involve continuous product development and enhancement benefit significantly from Agile project management, especially the Scrum model which allows time boxing of features within a specific “Sprint” cycle.
Unfortunately, the advocates and practitioners of Agile development has positioned Agile against other development methodologies, thus resulting in many cases of “throwing the baby out with the baby water”. While Agile brings many useful ideas and strategies to software development, it has rejected many practices that have significant value in practical software development. In this article, let me focus on requirements management.
What does Agile say about requirements management?
Agile Manifesto: “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”
Martin Fowler: “they (Agile methods) are less document-oriented, usually emphasizing a smaller amount of documentation for a given task. In many ways they are rather code-oriented: following a route that says that the key part of documentation is source code.”
The general view of Agile practitioners towards documentation seems to be:
- The program code is the primary source of understanding the system being built.
- Since requirements are continuously changing, requirements document will always be out of date, while the source code will be always up to date.
- So read the source code if you want to understand what the system does.
- Reduce documentation as much as possible to reduce unnecessary overhead.
While some Agile projects do maintain good documentation in the form of use cases and other artifacts, many Agile projects do not. Not because they cannot, but because they won’t as a matter of principle. The way that work generally gets done despite this seems to be:
- Focus on test-driven development: Making sure that the application is tested against expected behavior using both automated tests in the form of unit tests (NUnit, JUnit, etc.) and integration tests (FIT). Continuous integration and daily builds are the norm in such development environments.
- More frequent communication: There is more heavy interaction between the team members. The daily scrum also helps raise any blocking issues that helps prevent developers from getting stuck in some module.
- Using wikis and other documentation: Here is where it gets interesting. Almost always, I have seen that Agile development ends up creating artifacts to explain requirements in the form of emails, Word documents (for specific functionality) or pages within a Wiki.
Some of the practices such as greater test coverage are commendable, but the others raise many questions:
- First of all, using source code as documentation creates many problems:
- Source code is about the behavior of the system. Requirements is about the needs of the customer. The system behavior may not meet the customer needs. Where do we go to understand what the customer requested in the first place?
- Source code can contain faulty implementation or buggy code, especially when it comes to nuances. Should that be treated as requirements?
- 10 lines of requirements translates into more than 10 lines of code and that too, code not necessarily at one location. Should the developer hunt for such requirements throughout the source code?
- Instead of consolidating requirements into a centralized location, it is now dispersed into several other places and communicated through several different channels such as wiki pages, emails, chat sessions, bug tracking software, etc. and this happens over a period of time. As a result, no one really knows the entire requirements accurately. Even a full-time product manager cannot keep track of such changes consistently if the communication is so dispersed and undocumented.
- New project members take significant time to come up to speed. This is not as big of a problem at early stages of the project, but as the product grows bigger, this is a huge challenge. Apart from the lost time, there is greater risk of new project members misunderstanding requirements and introducing new defects.
One of the arguments given by Agile proponents is that Agile is meant for smaller, highly cohesive teams with motivated, talented individuals. Even if that is the case, you still hit limits of human biological capability when it comes to what can be stored and easily recalled in human memory. Even if you have been breathing the same module for months, you will still sometimes get stumped when you look at a piece of code and have no idea why you wrote it that way.
And then if a project is successful and needs to expand, it will need more developers to join the team. Even if they are highly talented, these developers still need enormous amounts of time to understand the system. In many cases, the existing developers need to reduce their programming time and spend it on training the new developers. Good documentation could reduce the time spent here.
Much of this heartache comes from a fundamental mistrust and fear of documentation. In a sense, some fear is justified because projects can get overwhelmed with documentation, but that is no justification to swing from one extreme to the other. Here is a possible middle way:
- Spare a few minutes to customize a documentation template that works for you — easy to write, easy to read and easy to maintain.
- There is no need to spend effort to collect 100% of the requirements before coding. However, collect enough requirements so that programmers can code without having to make design choices at the same time.
- When new requirements surface because of new business needs or because of something that came up while coding or testing, incorporate that back into the requirements first.
- Focus on making the requirements document the most accurate and up-to-date artifact of the project. If you have a dedicated product manager or requirements analyst, this task should be their responsibility.
A well-maintained requirements document does not hinder other good Agile practices. But still you will find significant opposition from Agile practitioners that maintaining requirements will take time and distract people from implementation. This is true, but only in the short-term. In the long run, the lack of coherent requirements and the resultant confusion and redundancy will eliminate and overrun any time or cost savings previously achieved.