Comments (1)

By Krishna, November 27, 2009

Brilliant post on testing by Michael Bolton (emphasis mine):

Bug Investigation and Reporting (time spent on tests that find bugs)Test Design and Execution (time spent on tests that don’t find bugs)

Module	Time spent on tests that find bugs	Time spent on tests that don’t find bugs	Total Tests
A	0 minutes (no bugs found)	90 minutes (45 tests)	45
B	10 minutes (1 test, 1 bug)	80 minutes (40 tests)	41
C	80 minutes (8 tests, 8 bugs)	10 minutes (5 tests)	13

[...]If we are being measured based on the number of bugs we find (exactly the sort of measurement that will be taken by managers who don’t understand testing), Team A makes us look awful—we’re not finding any bugs in their stuff. Meanwhile, Team C makes us look great in the eyes of management. We’re finding lots of bugs! That’s good! How could that be bad?

On the other hand, if we’re being measured based on the test coverage we obtain in a day (which is exactly the sort of measurement that will be taken by managers who count test cases; that is, managers who probably have an even more damaging model of testing than the managers in the last paragraph), Team C makes us look terrible. “You’re not getting enough done! You could have performed 45 test cases today on Module C, and you’ve only done 13!”

And yet, remember that in our scenario we started with the assumption that, no matter what the module, we always find a problem if there’s one there. That is, there’s no difference between the testers or the testing for each of the three modules; it’s solely the condition of the product that makes all the difference.

The obvious larger lesson here is that if you are going to use metrics, comparing numbers is not going to do anything. You have to be willing to spend the time to understand what is going on. Too often, managers use metrics as a club to hit employees on productivity, something that is enormously counter-productive. As employees find a way to make the metrics towards what the manager wants instead of the needs of the project.

This is also a case for collecting fewer metrics so that you can spend greater time analyzing them. For example, A looks pretty good on test coverage, but there might be the possibility that we haven’t written some necessary tests. Or now that we know C is pretty buggy, we may need to look at the development process – what stage these bugs are getting introduced and why?

In several recent posts, I have talked about the quality versus productivity issue in mature software organizations. Since deadlines loom larger, they are often given higher priority by developers and testers at the expense of quality. And so, managers have to make a special effort to emphasize quality and avoiding shortcuts. Creating productivity metrics swings the pendulum in the other direction. It introduces pressure to get more done instead of getting things done right. So instead, when you evaluate these kind of metrics, stop looking at “finding lots of bugs” or “getting enough done”, but instead at the direction the product quality is taking.

Comments (0)

By Krishna, August 26, 2009

Guy Kawasaki had a series of posts where he called out various types of people (venture capitalists, engineers, marketers, etc.) on untrue things that they are used to saying. There are situations when using a tough word is appropriate, but this is one case where Kawasaki is just wrong. Most people are truthful most of the time. Good business sense demands that you trust them, because if you are always watching out for lies from people, you are in deep trouble.

Now I understand the possible Digg-bait nature of the title. Also perhaps, one may give Kawasaki the benefit of the doubt in that he was talking about “white lies” and attributing incompetence rather than malevolence to each category of people. And that seems to be the case with the set of engineer lies, where Kawasaki says that he has a soft spot for engineers, but you shouldn’t really take them at their word.

I find the list a bit disturbing because it is a clear case of management divorced from what is happening in engineering. Every project requires good project management, otherwise it will only end in problem-hiding and finger-pointing. It is not about assigning a *project manager*, but whether the team has a grasp on the what, when and how of the project at all times.

Let’s take each of the points Kawasaki mentions (Do read the entire post so that you understand his argument):

1. We’re about to go into final testing: If this is a lie, it means that there was no project plan. It also means that there were no regular project releases so that incremental testing could be done and pertinent user feedback rolled back into the product.
2. Even my mother can use it: That is why you need testing by end users, as early in the project as possible. And it can be done on the cheap too.
3. Our beta sites really love the product: If this is a lie, you need to choose a better set of beta sites and people to test the product, and find a better way of getting honest feedback. Whether users will pay for the software is directly related to how much they love the product. How you make money off it is a question for the business owner and the marketing/sales folks, not the developer of your organization.
4. We’d save time if we’d just start the whole project over from scratch: This is probably true in some cases, but the most relevant point is that you have an engineer with the wrong skills. Usually, the engineer who said this does not have an idea of the technology, language or framework used and they express this sentiment because they would be more productive in their specialized environment. Spend time and money to find someone who has worked in the same environment. And if you find such a person and they still say the same thing, then your existing code base really sucks and you probably should start the whole project over from scratch.
5. This time we got it right: I am puzzled why Kawasaki thinks this is a problem. If this is a lie, you have serious trust issues with the engineer.
6. It works on my computer: OK, I grant this. I will probably write a longer post about this point, but it is a more complicated issue than it seems at first glance. The application probably works fine on more than “one computer in the world” and the developer is tired after being through many situations where the user has failed to follow some simple explicitly-stated instructions. One solution is to have customer support people field the calls from non-computer-savvy people first and let through only complex issues to developers.
7. We have great testers and a great bug-tracking system: I suppose this lie is coming from an engineer at a startup hyping stuff to an entrepreneur, because I cannot see that happening inside a company. I would hardly think that this is a common lie even in that situation, but Kawasaki should know better with his experience in dealing with startups. In any case, once again, you need to have testing early in the game, and by testers who are independent of the developers. I believe the technical term is “due diligence”.
8. I don’t know much about sales or marketing: I don’t know what point Kawasaki is trying to make here, but why would you care whether your engineer thinks sales and marketing are easy? I see jokes all the time about programmers and managers, but that doesn’t prevent them from doing their jobs.
9. What I’m doing is scalable: If the engineer is clearly junior and incompetent to answer this question, why was it asked in the first place?
10. This will take five minutes to fix: I grant this too. But this will only be a problem if you don’t have a good testing regiment. If you have frequent builds and you have a well-defined build-test-release cycle, you won’t get this answer.

Most of these “lies” seem to stem from lack of proper testing. If you release frequently and have independent testers, things will usually not get to the stage where management loses trust in the engineers. On the contrary, it will build confidence as new features are rolled out and the system becomes more stable as bugs are fixed. A regularly updated plan would also be good.