Comments (3)

By Krishna, November 21, 2007

People often talk about the 5 day workweek, but in reality, such a division of time and work is only applicable to hourly-paid workers. Financial people divide time into quarters of the year. Marketing folks divide time into advertising campaigns and trade show events. The technical crowd divides everything into projects. The IT support crew divides everything into periods of relative calm punctuated by hectic fire-fighting.

Most companies are dominated by people who work on the business side (sales, marketing, finance, etc.). And like it or not, their milestones tend to become the milestones of other people in the company. For example, if the marketing vice-president has an exposition coming up, that event acts as a point in time around which product deliverables get done. If the finance head needs to meet certain targets before the date of filing results, newer versions have to released and sold before then.

This causes significant conflict between the business and technical sides in the company. The delivery dates to which the technical team can commit and the order in which they want to work on the deliverables have no relation to the business deadlines. Unfortunately, neither side knows the language of the other. The businesspeople don’t understand the technical difficulties involved. The technical people don’t know how to express their concerns in an understandable manner or how to negotiate deadlines well.

Often, the business people accuse the technical team of padding the estimates. In my experience, this is very ironic because

• Software developers and teams are more likely to underestimate the complexity of building an application, especially during the initial rounds. This is because they have been given “high-level” product features and have not understood the implementation details.
• Software development invariably tends to run over budget and over time. The accusation would make sense if the business folks could point to any project that did the opposite. You cannot win this argument though – someone always points to Parkinson’s rule.
• Experienced engineers with less management experience are likely to over-estimate their capability of building the application. When formal estimation techniques are not used, they will try to relate the project to some other work they have done and provide an estimate.

Business people are more persuasive and more aggressive at getting what they want. If they talk loud enough, even the most confident engineer has self-doubts and tries to figure out how much he can bend the estimate. For example, he thinks, “If this deliverable takes 2 months, I could probably bring it down to 6 weeks by using tool “X” and maybe working a few extra hours every week.”

The problem is that an under-estimated project is further under-estimated, resulting in a Death March. The tragic aspect is that everyone knows that the goals are not realistic, but are so committed to the project that they put extraordinary effort. In the end, all they get for their hard work is blame and the bitterness of failure. Sometimes, people can be burnt for life after going through such an episode.

My point is that if you are a business person with little understanding of technology or project management, do not presume to know better. You must trust your technical people when they give you estimates. If you don’t like the estimates, it does not mean that they are wrong, just that they are inconvenient to you. If you over-promised someone, go back and re-negotiate. It is better to lose your face in front of someone than to submit your team to weeks of hell.

In a future article, I will suggest ways for technical people to avoid committing to difficult or impossible deadlines.

Comments (0)

By Krishna, November 17, 2007

Simplicity is one of those goals that everyone talks about, but few achieve. When designing applications, simplicity is supposedly a paramount concern, yet many applications never achieve that state. Very often, we see simple applications that are very basic in terms of functionality. Or, we have highly functional applications that are very complex for end users.

I believe a reason for this situation is that people consider only 2 stages of simplicity – “simple” and “complex”, while in fact, there are 3 possible stages as follows:

Simplistic -> Complex -> Simple

The arrows designate the passage of time in designing and creating such applications.

When you design an application to meet a particular need, you create a “simplistic” application that caters to that particular need. It does not meet any other need, but it is easy to use. This is a good strategy for designing version 1.0 of an application. You can focus on the most important need for your customers and create an application that meets their needs, without making it cumbersome to use. In other words, do not try to make everyone happy, just your core customers.

Once your application is in production, users will clamor for more features that may or may not be directly related to your core product objectives. For example, they may ask for export functionality of certain data to another application. To stay in business, you must try to meet at least some of those demands; otherwise you will lose your users to your competition. As you add more features, the complexity of your application keeps increasing. When you add more clutter, your software becomes more difficult for new users who find it complex and hence may be lost.

The challenge, therefore, is to add new features without sacrificing ease-of-use of the application. One strategy is to let the application become cluttered, refactor the user interface, become cluttered again, repeat. Another, probably better strategy is to carefully understand user interface implications before adding new features. The latter is not always an option when you are faced with competitive pressures.

In the real world, we find different companies coming out with applications in each phase, and making the older products obsolete. For example, first generation simplistic cell phone just made phone calls. Then complex phones came out with several functions and accompanying complex user interfaces. And now you have the iPhone and other newer generation phones.

Many people confuse “simplistic” and “simple”. If your software doesn’t do much, it can be very easy for end users. It can also be very easy to develop. But your end users will get frustrated when the application does not grow to meet their needs. And, your application will be easy pickings for your competition to copy.

Thus you have a paradox: You have to continuously add complex behavior into the software while reducing (or least not increasing) the complexity it presents to the user. That is true simplicity.

Comments (0)

By Krishna, November 4, 2007

In response to a recent article by Jeff Atwood about BitTorrent, Charles Petzold wrote an amusing post on how a failure of the commercial market could be used to justify illegal behavior. The debate was interesting because it brought out many aspects of the copyright situation. Petzold is a content producer and is rightfully worried about piracy. Atwood is a consumer and is rightfully agitated about not finding something he wants to purchase.

Downloading content has become simpler over the years, with better software and faster Internet connections. The costs are virtually zero. In fact, with the advent of YouTube and other streaming media services, a content user does not have to download anything. The whole experience has become seamless as the only task required from users is to click a link.

Content owners have tried to apply various types of Digital Rights Management to prevent copying of digital content. This has proved problematic because lack of DRM compatibility between devices means that users get frustrated when they cannot perform legitimate transfers or backup their content. Older, paid content runs the risk of being lost because of hardware issues. Methods to bypass DRM, although illegal in various degrees, are available for those motivated to do so.

The economic effects of unauthorized distribution of creative content (music, movies and books) are not very clearly understood. For example, would users involved in such activities have ever paid for the content, if they could obtain it legally? If the answer is no, then the economic loss to the producers is low. But the answer could very well be different. A significant portion of such users may have paid for the content if they had no other avenue. This is particularly true if the content was very popular.

Nobody is tracking this, but my guess is that the overwhelming majority of unauthorized distribution comprises of popular content which the users presumably value. Let us assume that this content was not available. For zero economic loss to the producers, they would have decided to forgo it or borrow it from someone. The producers would have experienced some loss if downloaders bought the content or rented it. We don’t know this information.

If content were freely available or easily downloadable, paying users may decide to go the free route themselves. After all, they stand little to gain from subsidizing the free riders. If my friend downloads 1000 songs for free, why should I pay for them? Even if I paid one cent for each song, I would be paying $10, $10 too much.

The equation is muddied by the fact that the more content is downloaded and used, the more popular it (or the creator) could become. For example, software product makers often overlook piracy of their products when they want greater public exposure. The popularity of a movie or song can translate into greater sales of the content itself or associated merchandise. This is clearly a good option for struggling artists, but perhaps not so much for established ones.

However, there is no guarantee that greater marketing results in greater success. The work of the artist or creator may appeal to a limited audience for various reasons such as cultural trends. Giving content for free is a marketing tactic, not an assurance of success. For example, Radiohead’s “Pay what you want” program is a tactic that became successful. It may or may not work for others imitating it.

A key aspect of the entire debate is the difference between content creators and content owners or publishers. The former are the actual persons like authors, writers, singers, etc. who have created the content. But the content is usually sold by big publishing houses, movie distributions and music labels. Unauthorized distribution is frequently characterized by sentiment against the large profits (by extension, undeserved) of such content owners.

Such feelings are reinforced by the heavily criticized lawsuits filed by the RIAA against ordinary people for copyright infringement. The frequent changes to copyright law, extending copyright term extension and criminalizing copyright-circumventing technologies at the behest of media lobbies, have turned many people against them.

Resorting to legal measures and government intervention usually reflects the incapability of an industry to adequately respond to economic conditions. The “failure of the commercial market” is real and the gap is being filled by unauthorized distribution and downloading. It is possible to invent a business model that could take advantage of these gaps. For example, in the movie and TV industry, the gaps are:

• Unavailability of the movie (or TV series) (regardless of their popularity) for months after their initial release. I think this has something to do with the popularity of the movie or syndication rights with other networks or just “perfect” timing (like releasing the last season’s DVDs a month before the new season), but it is still a nuisance.
• Unavailability in many geographic regions across the world. This may not be true of recent Hollywood blockbusters, but it is certainly true of movies in other languages, older movies, etc.
• Impossibility and/or inconvenience of viewing on different devices or computers. DVD region codes are a prime example of ridiculous thinking in an age of heavy international travel and migration.
• Difficulty (illegality?) of fair use of video clips for legitimate purposes like news reporting, discussion groups, etc.
• Difficulty of random access. For example, in any long-running TV series, there will be a few episodes that stand way over the rest. In most cases, it is impossible to buy or rent them individually.

A last problem is the one that Atwood faced: simply, lack of information about availability. Will the DVD ever come out? If the producers do not wish to sell the content, do they wish to suppress it? Do they mind if someone else makes money out of it or just distributes it for free? We simply don’t know. The current copyright law (in the United States) does not address the situation at all, as it allows copyright to extend to the life of the author plus 70 years.

The key problem with media companies is that they do not understand how disruptive technology can be to their existing sales channels. They perhaps do not realize how draconian laws and intimidating lawsuits do little to reduce common public behavior. They do not see their own failures in addressing market shortcomings. And in the long run, they will be replaced by nimbler companies that understand how to take better advantage of the market.

Comments (0)

By Krishna, November 3, 2007

Even if you don’t own Google stock, you still benefit when Google’s stock price keeps climbing upwards:

1. Ad-free services: If it were any other company, services like Orkut and Analytics, among many others, would be chock-full of advertisements. Google could be making more direct money from services like Blogger, but at this moment, that does not seem a priority and it may stay that way for sometime.
2. Lower premium penalties: Compare the standard and premium versions of Google Apps. Unless you are an organization, you are not missing all that much. Compare that with many important features, like forwarding and POP access, that are missing in the free Yahoo! Mail version.
3. Better applications: Google can continue to hire the best employees and buy out the best applications in the world. Many of Google’s acquisitions are among the best in their market. And Google continues to be innovative in different areas.
4. Better products from competitors: Google continues to put pressure on competitors to keep improving their offerings. Yahoo! has dramatically improved their email product. Microsoft Live is a much better application than it used to be, and Live Images and Maps compare favorably with the corresponding Google services.

It will be interesting to see what happens in the next few years. If Google continues to grow and establish their complete supremacy over web applications, then you and I will be managing our entire online life through Google. Google could also move strongly into other areas such as hardware, communications, entertainment, etc., thus becoming a much bigger part of our offline lives.

That would be a different monopoly than Microsoft, which only owned the “functionality” piece. Google owns the “data” as well. The debate over how this data should be used and how well protected it is will continue to grow larger. This may not seem a huge issue for you now, but consider the information you are providing to Google products when you access them:

1. Your private and public conversations with other people: Gmail, Groups, Talk, Blogger, Orkut.
2. Who you know: Orkut, Picasa (photo sharing), YouTube.
3. How you spend your money: Checkout, Finance, Products, Web History.
4. How you like to be entertained: Web History, Reader, Analytics.
5. What political/religious/cultural views you subscribe to: Web History, Reader.
6. Which places you visit – Maps, Weather, Web History.
7. What you do every day – Calendar.
8. What ideas you have – Gmail, Documents, Notebook, Blogger (including private blogs), Page Creator.
9. What you read – Reader, News, Books, Web History, Bookmarks.

Should I go on? The question is: how much of this information would you like other people to know? And this is not just about Google. Any company, including Yahoo! and Microsoft today, that has web applications collecting vast amounts of user data as part of its operations is in the same situation.

Comments (0)

By Krishna, November 1, 2007

Of the many differences that separate a simple software project and a complex one, this one is the most critical: A complex project has requirements that do not fit into one human brain. Many people understand this concept vaguely, but never understand it deeply or consciously. So let us analyze what this means.

A complex project has data and functional relationships that are many magnitudes numerous than a simple project. This is not necessarily related to the number of features that an application has. Nor is it necessarily related to the number of screens, reports, tables or lines of code. For example, a Garbage-In, Garbage-Out application with 5 input screens and 1000 canned reports may be less complex than one with 20 input screens and 30 dynamic reports, even though the former is a larger application.

The complexity comes from how interdependent the various modules the application are. Most of us are familiar with the concepts of coupling and cohesion in software design. But those concepts are also relevant to requirements management. Any requirement has the potential to affect the behavior of another requirement in the system. The more they actually affect each other, the higher the coupling between requirements in your proposed system.

Let us take a billing system. Every transaction affects AR, AP and associated financial statements. Now, let us add the requirement of managing a clerical error. Can the transaction be canceled? How long after the transaction can it be canceled? What happens if the error is discovered after a quarterly reporting period? What do you do about past reports? Introducing one seemingly simple requirement has affected many modules within the application.

Let us take an opposite example. When Google added a new Presentation module to their Google Docs, it hardly affected the rest of the system, except a name here and a link there. Although the Presentation module is very complex in itself, it did not affect the functionality of the rest of the system in any significant way. Now, Google can continue adding new modules and it won’t have a problem until it attempts to do what Microsoft Office does: Embed one document type in another document.

In a complex project, it is very difficult to understand all the complex rules involved. Consider 3 dependent events, the order of which is unknown. We have to consider 6 different permutations of the events. 4 events result in 24 permutations. 5 events result in 60 permutations. In a real system, the different transactions affecting the system result in an incredibly high number of possible permutations.

With a simple level of complexity, it is easy to keep all the system requirements in your head. Many small projects operate that way, successfully. As the complexity increases, you have to rely on external methods like some form of documenting requirements. Take your pick – software requirements documents, wiki pages, user manuals, test cases, etc. The primary symptom of non-documented requirements is the increasing number of regression bugs. If you ever have to tell someone, “This was working before. Now you broke it”, then it usually means that the other person does not know what the requirement is.

With complex requirements, you cannot rely on documented requirements alone. The reason is that since requirements affect one another, the person documenting them has to continuously cross-link and/or duplicate requirements so that it is evident to the developer what is happening. For example, in our billing system example, when we document “canceling a transaction”, we have to link to many other sections in the requirements. This is practically difficult and usually error-prone.

One strategy is to reduce permutations by enforcing some constraints in real life. For example, many billing systems do not allow cancellations. Instead, you have to put another entry – a negative entry. This means that functionality used to handle regular entries can be re-used, reducing the complexity. However, this may be problematic, because it creates greater hurdles for end users. Also, when your rules are defined by government regulations, you don’t have much leeway in changing them.

When you are building a system from scratch, another strategy is to build incrementally. Build a core that is less complex. Make it stable. Add a complex requirement. Stabilize the system. Repeat until you are done. Sometimes, you realize that a requirement is so incredibly complex that you simply cannot get your arms around it and it is consuming too much of your time. The best thing you can do is abandon that requirement after negotiating with your customers or doing more market research.

If your complex system already exists and you are adding new requirements, my heart goes out to you. It is a thankless job. 99% of the time, proper documentation is missing. Customers do not appreciate how much time you have to spend understanding the application. The benefit of any change is heavily outweighed by the outrage and anger that happens if you break something, purely because you didn’t know about it.

That brings me to the final word. Don’t be gung-ho about complex requirements. They are the meanest of all – they have no pity for people mouthing buzzwords or the latest technological fad or management double-speak. Tread with caution. Accept your human limitations. Try to reduce complexity at every step. Analyze every requirement carefully. Maybe, just maybe you may succeed.