Overdesign is also very common with databases. People get hung up on rules like “third normal form”. Sometimes when you over-normalize a database is [sic] makes it impossible to query the data later on. If your application is going to require reports, sometimes its better to design the database with the reports in mind. In other words, let the reports dictate how the data is stored.

This kind of advice is insane, ridiculous stuff. Take a minute and read what the third normal form means. Not following the third normal form means essentially having duplicate data in different places in your database creating a serious nightmare in maintaining data integrity. What you gain in reporting, you will lose when you start writing code to ensure that data is consistent everywhere.

Normalizing a database does not make it “impossible” to query data. If you have many tables because of normalization, work is harder because you need multiple joins to do the trick. But if you know elementary SQL, you can get it done easily. On the other hand, if you haven’t normalized properly, you have very silly problems. For example, in the Wikipedia example, to find out how many players were born on the 1st of January, you have to write code that does not count the birthdays of multiple players twice. In a normalized database, it is an extremely simple query.

It is ironic that the author talks about “overdesign” while falling victim to premature optimization. First, understand what normalization is supposed to do and how it helps you. Learn SQL or use tools that can help you get your data out easily. Profile your application and find ways to improve performance. And only if everything fails, then go for denormalization. And that too, only after understanding its downsides and how to prevent them.

Read my previous article on the same subject.

I have noticed something similar on the Amazon wish lists. It avoids an unnecessary click, but at the same time, allows you to quickly undo the action if you had accidentally clicked the first time.

This reminds me of one of my pet peeves: people who complain about the Delete confirmation dialog box for files. Generally, the person who makes this complaint “cannot imagine” how stupid the operating system architects and developers are there. The user wants to delete something. You have a Recycle Bin. Why doesn’t the operating system just get out of the way? And who reads the dialog box anyway? One complainer suggested that this was just a way for the developer to shift responsibility onto the user. Since the user clicked Yes, the programmer is no longer responsible if the user accidentally made a mistake. And the implication is that the programmer is lazier and needs to own the responsibility. Or something of the sort.

Obviously, I have no idea what the intention of the programmers were. But let’s assume that there was no confirmation dialog box and the second you clicked Delete, the file is deleted. That is great if you meant to hit the Delete key. But what if you had meant to hit the “End” key (to navigate to the end of a list of files) and your stubby finger pressed down both the Delete and End key. What would probably happen is that the currently selected file gets deleted, you navigate to the end of the file list and you don’t realize that your file has just been zapped.

But what about the Trash can, you say? The problem is that if you don’t know the file has been deleted, you may not bother to check its contents before emptying it, especially if you had been deleting other files. You may not even realize the mistake until days or weeks have passed and then even a disk recovery service may not be able to help you.

I elaborate on this seemingly minor issue because although user interface issues are more complex than they seem at first glance, because they don’t seem so complex, everyone has an opinion. Even those who have no clue about user interaction or the design choices that made a choice necessary, even if it was not the most optimal. This was best illustrated by a recent comic “How a Web Design Goes Straight to Hell” by Oatmeal Comics. As the author notes,

I actually had a client include their mother in the design process so that she could provide feedback and criticism. [...] You are no longer a web designer. You are now a mouse cursor inside a graphics program which the client can control by speaking, emailing and instant messaging.

It is a tough world out there for user interface and user interaction designers.

[A] great deal of effort has gone into the design of networks, databases, and other technical dimension of software. Books have been written about how to solve these problems. Developers have cultivated their skills.

Yet the most significant complexity of many applications is not technical. It is in the domain itself, the activity or business of the user. When this domain complexity is not dealt with in the design, it won’t matter that the infrastructural technology is well-conceived. A successful design must systematically deal with this central aspect of the software.

Those of us with a software background have a tendency to get mired in the technical details of implementation. All of those discussions around the duct-tape programmers and unit tests and inversion of control. In some ways, it is a good debate to have. But they are just tools to do the actual work, which is to provide solutions to real-world problems.

And in the real world, there is enormous complexity. If you take a business application, there are so many rules and constraints demanded by various departments in the organization. Ignore software for a moment and think of any organization you have worked in. Remember the employee manual? That is just the start, isn’t it? There are so many other de-facto rules for inter-personal or inter-group coordination because of organizational history, etiquette, bureaucracy, etc.

Then there are outside rules by government and standards bodies, to name a couple. The sheer volume of these rules makes it very difficult for any one person to truly understand them all, let alone the inconsistencies and relationships between them. To make matters more complicated, these rules are changing always. Not everything all at once. But one rule here, one standard there – this month, that month. And your requirements keep shifting ground. There is never a fixed target.

That is the real reason why building software is complex. And it is not going to be solved simply by having a better tool or process which only help if you have somehow managed the complexity of the problem. Sometimes, the real-world complexity is, in fact, too difficult to manage. And the best you can do is take a small piece of the problem and try to solve it. Or maybe you can influence the real world to get rid of a business need like, say, end some silly bureaucratic process, and avoid the need to build a complex piece of software.

I was reminded about this when I read about the Chandler project in Scott Rosenberg’s “Dreaming in Code“. Much of the book is about the innumerable discussions that the team members had about requirements and design which lead to the project being at standstill for significant periods of time. Also how they kept second-guessing themselves and changing decisions in mid-stream.

This is not unique to Project Chandler. Projects go through this phase where the programmers need to understand their end goals and how they plan to get there. The question is how much time do you plan to spend on it and how soon you can make decisions to get on with the construction phase? Where is the point at which you have diminishing returns from further analysis?

I wrote a post sometime back about the pitfalls of soft coding, which is writing an application in such a way that the user can configure its features ad infinitum. Soft coding exists, in part, because the team could not make a decision about what the user wants. Instead, the project team kicks the decision to the user’s court.

You can also see this in design, where there is a lot of hand-wringing about writing loosely coupled code. The idea is that you should be able to remove any class or library at any time, plug in a replacement and just have to change one line in a configuration file without compiling. So much engineering effort goes into writing such maintainable code, when in reality, most of the code will never be changed again in the lifetime of the product. And most of such decoupling is quite unnecessary because it could easily be substituted by someone taking two minutes to change the code.

Now, you may think my previous paragraph is stupid. How dare I promote tightly coupled software? But there lies the problem. Many programmers have been bitten by bad code written by bad developers that they have gone into over-engineering mode. Instead of having to change code, let’s make it so that the users can dynamically change the behavior of the software. Instead of having to recompile code, let’s make it so that some administrator can change an XML file.

Sometimes, I feel that programmers are stuck creating compilers when they should be creating applications. Compiler, as in, give users the ability to specify in minute detail how the application should behave at run-time and allow them to change their minds whenever they wish. Whatever else an application may do, you can guarantee that the “Settings” module of the application will grow and grow, as programmers delegate decisions to users.

This tendency to favor configuration over convention is perhaps due to a fear of being wrong. It is less risky to allow users to make the choice than find out what the more intuitive way of doing something is and implement that. So keep on building indirection upon indirection.

We need to break out of this cycle. Keep the application as simple as possible. Follow best practices, but don’t beat them to death. Start coding sooner. Build prototypes to validate your requirements. Build pilots to validate your design approach. Channel user feedback into refactoring the application. But just don’t sit there re-examining your decisions and trying to come up with the perfect design.