Software Metrics — Incremental Use

by Krishna on July 2, 2007

My last post on software metrics was about how we should attempt to measure only those metrics over which people have control and feel they have control. In this post, I will discuss how metrics should be used incrementally to understand the actual processes and dynamics within a team or organization and thus be in a position to improve them.

Let us say that you want to measure the quality of a software product you are building. You are not collecting any metrics yet, but you want to get started. Let us say that you start with a simple measurement, “number of bugs discovered in every build”. It looks simple enough — the fewer bugs reported each build, the more stable the product is.

However, when you start measuring something, it affects the behavior of the people. This is the classic Hawthorne effect, namely, people change their performance when they are being observed. You may think this is a good thing — people will reduce the number of bugs because that is what is being measured. Isn’t that good? Yes, if that is what people actually do.

In reality, people behave in strange ways when you start measuring something. Instead of actually trying to reduce bugs, some of the developers may try to reduce the number of reported bugs. How can one do this? Well, the developer may decide to work on areas that are likely to yield a great count of bugs, like user interface level bugs, while ignoring more critical areas where bugs are harder to find because of complicated logic.

A more dangerous situation is where the developer tells testers to inform them about bugs, but keep them outside the bug tracking software. You will see less bugs being reported, but in reality nothing has actually changed with respect to quality. To prevent this from happening, you may decide to reward testers for reporting more bugs. But that can also backfire, since testers will now start to report trivial bugs as well as issues that are really not bugs, but different interpretations of expected functionality.

At this point, you will realize that what matters is not just the count of overall bugs, but also the quality of bugs found. A low count of severe bugs is more critical than a greater number of simple bugs. Severity could be measured in different ways. It could mean an important user function working improperly. It could mean that a particular module has been designed entirely wrong and has to be coded again from scratch.

When you start collecting this metric, you gain some insight into the actual quality of the software. But then you have more questions: Even though the severity bug count is going down, how many bugs are still outstanding? Are you winning the battle with the bugs, meaning are you fixing more than you can find? Did you spend more time fixing bugs than you could have spent redoing the work?

You realize that one metric alone will not cut it. You need to collect more information to manage the work better and also find out if any changes you are making are improving things or making them worse. The good thing about things like bugs is that if you already have an automated bug tracking system, you can get most of the information from the software itself.

For example, a typical bug tracking system should have various bug statuses from Reported to Closed. If you have access to the data repository or if the software allows you to do reporting, you can find out how much time it takes for a bug to go from being reported to closed, or how much time an average bug takes in development time. By setting this up properly, you can have up-to-the-minute statistics on your hand at all times.

Looking at one particular aspect (in this example, bugs) from different angles can allow you to understand how to improve your project. For example, if you notice a certain type of bug being reported frequently, that should ring an alarm bell in your head — maybe it is time to design something that can take care of that particular issue forever instead of fixing one bug at a time.

Although many people frown at this, looking at statistics of each individual can also be illuminating. You can find out if a particular developer is having a problem with a certain type of error checking. Sometimes, an individual tester may be accidentally missing some types of scenarios that other testers are working with. Looking at data the right way can provide insight into each person’s performance and way of working. Just be careful to do it sincerely for the project’s sake instead of using it as a weapon for manipulative purposes.

Many software departments in companies and even software companies themselves do not collect metrics. This is understandable because software metrics can require significant resources. However metrics can help one do a better job. Incrementally collecting metrics and continually tweaking the collection process and the metric itself can be a good strategy. This avoids a huge up-front investment and at the same time, helps one learn more about one’s own processes and organization.

Comments on this entry are closed.

Previous post:

Next post: