Programming is Easy, Software Development is Hard

by Krishna on January 23, 2011

David Brooks has a great essay on which skills are difficult to master:

She’s protecting them from the most intellectually demanding activities because she doesn’t understand what’s cognitively difficult and what isn’t.

Practicing a piece of music for four hours requires focused attention, but it is nowhere near as cognitively demanding as a sleepover with 14-year-old girls. Managing status rivalries, negotiating group dynamics, understanding social norms, navigating the distinction between self and group — these and other social tests impose cognitive demands that blow away any intense tutoring session or a class at Yale. […]

Participating in a well-functioning group is really hard. It requires the ability to trust people outside your kinship circle, read intonations and moods, understand how the psychological pieces each person brings to the room can and cannot fit together.

In the same vein, I have always been puzzled by people who say that math is hard. In my opinion, your understanding of most math that you need is directly proportional to the effort you put in, and less to do with your intelligence. It is like a ladder. Once you master each step, the next step is easy as long as you put in the effort. In fact, most of school is just that: put greater effort, reap better rewards. Through enough investment of time and effort, you can increase your scores at even exams like the GMAT that purport to measure intelligence and capability.

But there are many other areas in life where this dynamic doesn’t work because they are not one-dimensional. Dealing with other human beings is one such scenario. You have to use different tactics (kindness / toughness / incentives / self-abasement, etc.) to get other people to do what you want. Getting a bunch of people to work towards the same goals means learning not only to deal with each person individually, but also how those people interact with each other. For example, how giving one person more responsibility in the team will affect the ego and motivations of other persons in the same team.

Programming is a learnable skill. The more you work on it, the better you become. You can master the intricacies of a language, understand different patterns of building software, acquire the knowledge of useful algorithms. Practice keeps your skills sharp. There will always be edge cases which are beyond you, but for most mundane programming tasks, if you are willing to improve yourself, you can do the job. In this regard, programming is just like math. In math, 1 + 2 = 3 always (unless you are in some weird alternate universe). Your program is either right or wrong. And you have metrics about performance, program size, etc.

But let us take software development. I am making perhaps an arbitrary distinction between “programming” and “software development”. So let me be clear. Programming is solving a well-defined problem. Software development is defining the problem in the first place. Once you have a question, it is usually easy to solve it once you have the tools and the knowledge to use them. If you don’t know the question, things become a lot harder.

Software development is about that tricky part. It is about understanding your customer, sometimes talking to them and sometimes not knowing them and only understanding them through observation. It is also about working with multiple people to build a coherent quality system. It happens in the real world where things can go wrong and frequently do. I don’t need to elaborate because you know what I am talking about. It is far easier for a single programmer to pick a problem and solve it than for a huge team to define the problem and set about to find a solution to it.

As a related example, I was recently intrigued by all those YouTube videos of world records by Rubik's Cube solvers (< 10 seconds!). Wow, those guys must be so intelligent! Anyway, so I bought a Rubik’s Cube and found that there are well-defined steps to solving any Rubik's Cube. And if you are sufficiently advanced, you can actually solve any Rubik's Cube in 20 moves. So practice, practice until you get there. Maybe you cannot do it in 10 seconds, but even 30 seconds will impress friends who don’t know this.

Similarly, programming looks like some supernatural act to those who don’t know programming. And some nerds like to encourage that. But in reality, if you have the programming skills, most of what you do is incredibly banal. Good programmers recognize this, seek out greater challenges beyond what they know and push themselves. But doing what you already know is not such a big deal.

{ 83 comments }

Llewellyn falco January 24, 2011 at 1:16 am

I don’t understand post like this. If programming was easy, it’s just the people problems that are hard, why not always have 1 programmer? If adding people just makes it harder? ( or hard period )

Btw: math isn’t easy ( and I love math ) try proving something new if you think it is. Fermats last theorem took centuries? Were people just being lazy?

Yes people can make even simple things impossible, but programming isn’t easy.

Krishna January 24, 2011 at 2:28 pm

@Llewellyn, many things are easy, but you need more people to do it in less time. Example: construction workers, clerks, janitors, etc.

As for advanced math, I did qualify my statement by saying “most math that you need”. Many people find even high school math (algebra) difficult, let alone Fermat’s last theorem.

Rick January 25, 2011 at 10:41 am

Having written software (and programmed!) for almost 40 years in all different industries, I can state two truths: Most math I need is easy, most customers are hard (at least hard to please). I never got further than high-school algebra and written some very complex systems, none of which, fortunately, are rocket guidance systems!

k_money January 25, 2011 at 4:57 pm

I believe math is just an anlogy. He tries to convey that answering a question is easy. Defining a question is hard.

Rob Mills January 24, 2011 at 10:11 am

Completely agree. In fact you have to be able to interpret what the customer is saying. Many times I find they don’t actually know what they want or how to explain it. You have to be able to lead them through that part.

Stewart Sims January 24, 2011 at 10:22 am

Very interesting observations, I wrote a similar post on my blog recently about the relative ease of learning to program (and that it largely is down to practice and perseverance rather than intelligence):

http://blog.ssims.co.uk/index.php/2010/12/programming-its-really-not-that-scary/

Having said all this, its worth noting that there is still a barrier in many people’s minds with regard to learning to program, and perhaps the sheer fact that at first it seems very daunting makes it difficult initially.

Alleagra January 24, 2011 at 11:20 am

While I agree with the general thrust of your piece, I am not so impressed by Brooks’ essay. I rather take the Philip Greenspun view on this.
http://blogs.law.harvard.edu/philg/2011/01/19/stupid-white-man-criticizes-smart-chinese-woman/

Anecdotally I have to say that that second paragraph you quote has raised considerable hilarity among folk I know. As to his suggestion, much the same could be said about any animal interacting in its social environment. They tend not to play piano though.

Krishna January 24, 2011 at 2:36 pm

@Alleagra, I don’t disagree with Greenspun, but he is arguing against a straw man. Brooks’ point is simply that social skills are much more difficult to master than academic skills, because the former is not well defined when compared to the latter. The general point is that moderation in everything is good – extremes in either way are not good.

Paul W. Homer January 24, 2011 at 12:25 pm

Great post.

In some sense, it’s the difference between being able to do large scale arithmetic in your head, and being able to prove a new theorem. One involves understanding and applying the rules correctly, while the other involves finding a working set of new rules.

Paul.

Mike January 24, 2011 at 1:11 pm

Nice post…

I can definitely relate to this as a developer and see the differences in building a software system and programming. I have a friend that always tell me, “make sure the problem and solution is clear before you start typing any code, then the programming will just feel like typing and the code will be elegant”… Programming in a sense is like building a room in a mansion, well defined and narrowly focused. Software development on the otherhand is like architecting the mansion, specing it out, creating the blueprints, etc..

I have been able to learn new languages and write some pretty cool code but the part I struggle in, which is improving all the time, is getting the problem space down in my head and designing the system with the team. As you have mentioned, Software development is a multi-dimensional thought process and not a linear flow. Programming
algorithms can still be and will be challenging at times depending on what you are developing, but having a clear view of what all key parts
are and coming up with a clear design and solution will never be a 1-2-3 process.

Ryan January 24, 2011 at 1:25 pm

Couldn’t agree more. I’m currently in the position of pseudo project manager and I have to define problem space, fit customer requirements into our current roadmap, figure out how to keep the back end guys and the UI guys on the same page, etc….. my “procrastination” has become teaching myself scala with project Euler because even the seemingly hard tasks are so much easier when you already know what the end result is supposed to look like.

Wirawan Winarto January 24, 2011 at 2:11 pm

to your definition, programming and software development are in different level of abstraction. I don’t think any of them is easier than the another.
in business-related area, mostly yes. programming is easier because it consists mainly of writing code to store and load data. software development is harder because most of them are consist of planning complex business processes in a team. but “solving well-defined problem” is not always easier. for example what if you are trying to build an innovative faster-than-ever massive search engine? software development (“dealing with teams”, “defining problem”, etc) is usually the easier part. while designing and programming a smart and optimal algorithm is the real punchline.

I don’t think one is easier than another. it totally depends on the context.

Krishna January 24, 2011 at 2:37 pm

Good point, Wirawan. There are definitely cases where programming is very complex even when the scope is simple.

mtcoder January 25, 2011 at 12:07 pm

but its not the code that is the complex part, its the generation of the solution. What method are you going to attempt at first? Are you going to define a new set of classes which handle a special algorithm? What is that algorithm? What metadata are you going to try and crawl and how are you going to put it together to equal a faster search result? Not a stick of that is programming, does having a bit of programming knowledge help? Yes knowing what the language can do helps define the tools you have for the solution, but at no point should you create a single line of code till you have a detailed blue print to go with. The more complex the code the more complex the software development is.
The two are linear, in some aspects. The better and percise the software development is the better and easier the code will be.
Code doesn’t think for itself, well unless your writing AI but even then your just having it pretend to think. You need someone to define what the code is to do, then you just write the code to do it.
That is why programming is easy, and software development is difficult. Heck with Intellesense and things like MS’s Razor language programming is becoming something everyone in high school does during exploratory class period. The difference is those people won’t and typically don’t do well with software development. Software Development is an art, “painting is easy” you pick a brush up draw some lines and your finished. Making a master piece takes planning and knowledge of where to place what and how to get it to incorporate into other pieces of the picture.

Tian Xin January 24, 2011 at 2:17 pm

um… if ‘programming is a learnable skill. The more you work on it, the better you become.’ then so is the same with software development. I guess what you mean with ‘getting a bunch of people to work towards the same goals’ is also can be learned from experience.

Krishna January 24, 2011 at 2:43 pm

Not really, Tian. Programming is, more or less, an exact science. If you learn something, it stays learnt and you can apply it in similar situations.

Managing people is not an exact science. You may need different techniques for almost similar situations because of the people and the context.

erase January 24, 2011 at 2:32 pm

“I have always been puzzled by people who say that math is hard.”

that’s because you understood how to learn it easily. what they really mean is “math is hard to learn”, which is true for many people. if they never grasped a good way to go about learning new math concepts on their own and the teaching methods were not conducive to their own learning style, then they will always find it difficult to understand the concepts quickly – which often leads them to believe that they will never be good at math and they simply give up.

Krishna January 24, 2011 at 2:44 pm

@erase, fair enough. Teaching styles could definitely affect learning.

Tomasz Kowalczyk January 24, 2011 at 2:40 pm

I completely agree with this post. Programming meant as “writing code” is not that hard – you only need to know what language keywords mean, and write them in proper order, as in mathematical equation. Programming meant as “creating a program”, whatever the “program” is very hard, because there are many possible ways to achieve some effect – but only few ones with good performance / usability / visual interface can attract end users.

Jens January 24, 2011 at 5:44 pm

Though I tend to agree with you I don’t think this is the big deal anymore
after all building new stuff in sw is not that hard.

I am more concerned about issues like maintainablility and increaing complexity.
As my observations over the years are that the size of newly developed applications has increased, driving the number of bugs up, two factors that together has coursed increasing prices on both to continue existing project and starting over. In addition we have probably yet to see the bigger part of law makers concern about the quality of the output of our craft.

tomo January 24, 2011 at 10:21 pm

If programming is only to generate a code that satisfies a given constraint, the post is right. We, however, expect programs to be sustainable (readable, modifiable and so on) and reusable (understandable, adaptable, generic and so on), which involve hard people issues.

Michael January 25, 2011 at 2:37 am

Why do we need to compare programming and software development? Actually software development consists of programming. Why do we need to compare them if it is part of the other? There are situation where programming is hard if the common native functions are not there. Programming is hard when innovation is needed.
For example is this situation:

The user wants a report that multiple headers should be printed per page automatically (your website should automatically detect it) even if you have a customized report.

Have you seen the statistics why there are a lot of people having blogs on ‘how to’ in codes?

Krishna January 25, 2011 at 6:45 pm

Michael, the reason why there are so many blogs with “how to code” posts is that they are straightforward. People issues are a little more complicated, though you can find quite a lot of snake oil “management” salesmen peddling their pet theories.

Ahmed January 25, 2011 at 7:18 am

It depends on type of problem and mind set.
Also, as people are unpredictable creatures, technology is always changing and you need time to learn, research, experiement and integrate.
The time and difficulty level of dealing with tough people in your team may equal to time and difficulty level of dealing with tough technology challenges.
You may see that the first challenge is tougher than the later because your mind set prefere the later i.e. you are originally a programmer not a manager.
Ahmed.

Krishna January 25, 2011 at 6:51 pm

Programming is a hard skill and someone without any formal training (such as a non-technical manager) will find it hard. I would like them to make the effort to understand programming so that they understand the work that their subordinates are doing.

But that is neither here, nor there. The point is that managing people and projects is a difficult skill to master, even in non-software realms. And it is not something you can learn from a book, or by rote learning. You have to learn from experience, learn different techniques, learn to exercise good judgment – a whole host of things.

James J. R. Aiello January 25, 2011 at 9:04 am

An excellent article. As someone that’s worked in IT for 25+ years, it pinpoints the root cause of the poor software quality delivered today. IT, and IT projects, are often managed by persons with finance or business degrees that don’t understand the differences outlined in this article.

After rescuing a $500K project that was plagued with problems and 2 years behind schedule, I had to do a “knowledge transfer” to a resource that was just out of college. Management cited their view that we were interchangable and the fact this resources worked cheaper. Within 6 weeks, the project feel into it’s prior state.

Christian Sciberras January 25, 2011 at 9:09 am

On the other hand, shouting out that programming is actually easy will only increase the number of idiots who can’t tell between Java and JavaScript…

Krishna January 25, 2011 at 12:03 pm

How would that happen, Christian? I don’t get it.

Bob January 25, 2011 at 9:20 am

I can’t think of evidence for this, however I would think that the problem is not that social skills are more difficult to master than academic skills, but that social skills receive less attention during studies and training. This is especially true while studying/training to become a programmer. If you spent all those hours working in groups, making speeches, performing in plays, attending parties and praticing sales techniques instead, you’d have excellent social skills… and you’d be technically useless. Like others have said, it’s about balance.

Pedro January 25, 2011 at 9:58 am

Hi,
Weird thing, I was thinking something very similar this morning, but instead of “easy” I would said that programming is fun, if you enjoy it, then it should be easy for you, its like solving a puzzle and like creating a sculpture at the same time, I agree, is something that you can master but I don’t think that its something like math.
I agree that the hard part of software development is everything but programming.

Kas January 25, 2011 at 10:32 am

Great article.

Ricardo Santos January 25, 2011 at 10:48 am

Whatever skills you practice during your childhood, will be the ones you see as “easy”.

Everything can be done trough practice, but you are more receptive during childhood. Even social interaction. Ever heard the term “social engineering”?

Louis January 25, 2011 at 10:52 am

I mostly agree with the article, but to say that something is easy, isn’t right. If you know something – it’s easy for you, if you don’t – it’s hard.
Programming isn’t ’1,2,3′. All programmers know that most of the programming tasks could be done in defferent ways and here is the defference between good and bad programmer.

Roberto Rodriguez January 25, 2011 at 11:37 am

1+2 = 10 (in base 3)…
Like this (change the point of view), you talk about software development like a business consultant…

Paul W. Homer January 25, 2011 at 11:38 am

Great comments. It’s nice to see so many people taking an interest in this post.

I could be wrong, but it seems to me (by guessing and intuition) that most of the commenters who disagree with this post are younger and/or less experienced, while most of the commenters who agree are old and crusty like myself. If that is the case, I’m not surprised, since as is the case with any profession, it takes significant time beyond just basic schooling before people achieve a mastery (some say 10,000 hours). A huge point in favor of Krishna’s post is that this split in the comments implies that programming is easy enough to allow many of the newer programmers to believe they have mastered their profession long before they really do. Looking back, I realize that this was true of myself as well. I was sure I knew how to write good code years before I actually could.

Paul.

Ivan January 25, 2011 at 12:51 pm

Well, let’s just say, programming is easy for some people and difficult for others. But it is the same with playing the piano. Or pickpocketing… But I programming is easy.. Software Engineering is not. Unfortunately there is always confusion between the two…

roundy January 25, 2011 at 1:09 pm

As a lifelong programmer, I have a favorite analogy I tell my friends about my work: Where programming is like knowing how to move each chess piece, software development is like actually playing the game of chess. It’s especially like chess because all day you are thinking in the abstract, many moves ahead of the task at hand.

CodeBozo January 25, 2011 at 1:14 pm

Restating what others have said, this discussion is all about the definition of programming. What are the boundaries of the definition? Are we including program design? Understanding and applying design patterns and principles like OCP, DIP, LSP and ISP is not easy at all. Kudos to those of you who are not challenged by patterns and advanced design principles.

Additionally, what if writing the program requires knowledge of a very difficult to grasp domain? Understanding the domain might have nothing at all to do with social interaction or proficiency in a particular programming language.

While there is some truth to the premise of this article, the boundaries need to be better defined to make an assertion that “programming is easy.”

Marc January 25, 2011 at 1:57 pm

I’m a software developer for 25 years. 5 years ago. I followed an NLP (neuro linguistic programming) course. One of the the basics of NLP is the world and the mental model we have on that world. In my humble opinion, software development is translation ‘the model of the world’ of the user(s) to the model of the world the machine has on that world. So I think, software development is in the first place a communication task. As we all know, good communication is difficult. In my university classes, the emphasis was on the technical aspects of ICT, not on communication.

joey January 25, 2011 at 2:07 pm

This is my first time here, I found you in my email from “The Code Project” and want to say thanks. I am middle aged man (50) and after just finishing two years of IT classes at community coll. I am amazed at how little I know after a bunch of A’s! I am trying to learn C++ on my own, and am approaching it like math, as you say, a step at a time. When I first went to community in my 20′s, I started with remedial algebra, and went up to calc.3, and its funny how you look back and say to yourself “what was hard about that” after you know something. I never thought I would get that far, but I couldn’t stop, it was all night school.

This is just a long thank you Krishna- it was very inspiring to me as a newcomer that hard work can pay off. If you don’t mind, I also would like to contribute an idea from business guru Stephen Covey about people that may help in dealing with folks- “Seek to understand before seeking to be understood”- it works! Thanks!

Krishna January 25, 2011 at 6:54 pm

Thanks Joey. That is a good quote.

tom January 25, 2011 at 3:16 pm

In my opinion, your understanding of most math that you need is directly proportional to the effort you put in, and less to do with your intelligence. It is like a ladder. Once you master each step, the next step is easy as long as you put in the effort.

Math you learn in high school is like a ladder. You get introduced to algebra, which leads to geometry then more advanced(multivariate) algebra and sometimes calculus. These are the foundations for a lot of more advanced math, and this foundation is what test boards have come up with for mathematical aptitude; there is no difference between the mathematics on the ACT/SAT and the General GRE, the GMAT or the MCAT.

However in college, if you pursue computer science, you take logic and maybe statistics. If you pursue engineering, you take vector calculus, linear algebra, complex analysis, differential equations, etc. The algebra/trigonometry/calculus foundation is assumed, but the path you take through advanced mathematics is anything but linear or ladder-like, and there is no A->B->C path through to understanding most advanced mathematical concepts.

In fact, this is where I took offense to the fact you believe that the more effort you put in, the more reward you get out. IMHO, this is definitely not the case for more advanced mathematics. There comes a point where you need to be intelligent, you need to be cognitive to understand the concepts and what is going on. And there also comes a point where an engineer cannot understand all of the mathematics they get exposed to, since the breadth of mathematical knowledge is not just the simplistic foundation.

However, if you think that math is the thousand year old concepts of basic algebra/geometry/trig then any layperson can master those fields with study, but that’s not what math is.

Krishna January 25, 2011 at 4:17 pm

Mike, I agree with you. Obviously, there is a line somewhere where more effort stops paying off and then you need other qualities. This is as true of math as it is true of other subjects or even sports.

But when many people say they find it hard to understand math, they are not talking about the advanced stuff. They are talking about basic algebra and trigonometry, and that astounds me.

Daryl Lucas January 25, 2011 at 4:38 pm

It looks to me as if you’ve done a good job of showing that software development is more difficult, relatively speaking, than programming. I agree.

But so what? What are we supposed to do with this information? What do you see as the implications?

I would like to see you take this further. As is, it strikes me as much like saying, “The ocean is bluer than the sky.” Doesn’t affect me.

Krishna January 25, 2011 at 7:02 pm

Good question, Daryl. One take-away is to work better at understanding the problem that has to be solved. Good programmers know how to solve problems easily. That is what they do all day. But take a break from the code sometimes, look at the whole picture – the customers, the end users, project members, etc. Sometimes it can be humbling.

AnnMaria January 25, 2011 at 5:24 pm

I disagree that programming is easy – and I love programming and have been doing it for nearly 30 years. Solving well-defined problems, e.g., give me the average sales for every store, is pretty easy. Most of my programming problems don’t come in well-defined buckets. They come from clients in terms like, “We’d like to better understand our customers’ view of our services and provide useful information to both our management and to the customer based on what we find.”

I completely disagree that social skills/ people skills are harder. They are different, but not necessarily harder.

In every organization where I have worked there have been more people wanting to run around and have hours of useless meetings about specifications that could be figured out in 15 minutes and many fewer people who could actually write the code. When there are cutbacks, the middle managers with “people skills” are the first to go, which makes me question that this is so much of a harder skill to develop.

It’s also a heck of a lot harder to define. You can tell who is a good programmer by what the programs they write do and how well. If your project team is doing great, it’s not so evident. Maybe they are doing a great job DESPITE you.

Krishna January 25, 2011 at 5:47 pm

Ann, you are confusing “programming” (as I defined it) with “requirements analysis” (which I defined as part of “software development”

In the example you provided, if those middle managers had good social and management skills, you would not have “hours of useless meetings”.

Being able to get to the right specification is a tough skill and not many people know how to do it.

LucianoQ January 25, 2011 at 6:34 pm

Wow! What a deceptively simple post! On first reading I was agreeing – yes, yes, that’s right – it matches well what I learnt in 40 years in IT. But on thinking more deeply about it, I changed my mind.
There are many dimensions that need to be considered to determine whether one is able to do an activity. Here are a few:
1 Body of knowledge.In many fields we stand on the shoulders of giants who have created a huge body of knowledge – we need to acquire as much as we can of that knowledge to give us the skill to do something. For instance, now you might say it is easy to be a doctor and cure people of most problems – however in the times of alchemy it was not quite so easy to cure people.
It becomes much more difficult when the knowledge does not exist and one has to create it by themselves. In my case I noticed that I can do many sudoku puzzles up to a certain level of difficulty, then I cannot anymore. I have deliberately avoided looking up articles on techniques to solve them – so I have developed all my techniques on my own. But some puzzles I just cannot do – it does not matter how much effort I put in, or how many other puzzles I do. Perhaps given an infinite amount of time … but I’ll come back to this later.
2 Courage.Some situations require more courage than others. Depending on the group dynamics in a team it might be difficult for someone to go against the grain, or against the boss, and say to do things differently.
In a similar vein are other attributes like strength and stamina. It is easy to run, but I cannot do a marathon – most people never will regardless of how much they try.
Watching a painter (a house painter, not an artist), one easily forms the impression that it is easy to paint. I have seen many friends end up in tears in doing renovation projects because to save money they thought they would do the easy bit, the painting. Until one studies a skill in depth, one cannot judge how difficult it is – the more one learns, the more one finds nuances. I can “sort of” play the piano. Anybody can hit the keys. But I can never be good enough to be on a stage.
Fact is that many aspects of life involve competition. It is not just a matter of writing a program, but doing it better and more quickly than others. The young people learning social skills are really in competition with each other. Who will be a leader. Who will be heard. It is easy to play tennis. It is hard to be champion.

There are many more dimensions. But all this brings me to the most exasperating habit I have seen in just about all the programmers I met (hundreds). Just like in your article, they believe that given enough time and resources they can program anything. So, I go to a programmer and ask – can you do this – sure, if you give me enough time and all the things I need. Unbelievable. There is no other industry or trade in which people respond this way. If I say to someone – can you weld me a gate – they immediately say yes or no – no talk about I have to go look up how to do it, nor I have to go find the equipment. Can you play the piano. Yes or no. If they say yes: can you play Moonlight Sonata. Again yes or no – never anything like – “give me one month”. Can you fly an airplane. Can you write a program to play chess. Ha – try that one. The programmer might write you the program, and I bet it barely will play as well as a novice. Would you regard that as being successful? Someone who has only learned the piano for 6 months – would you regard them as being able to play Moonlight Sonata? The programmer will quickly respond “but you didn’t tell me you wanted the program to play well, you didn’t give me all the specifications”. No other professional would say this. No tiler would ever say “you didn’t tell me you wanted the tiles laid in straight lines”.

It actually is damn hard to be a good programmer, or, for that matter, good at anything.

LucianoQ January 25, 2011 at 6:39 pm

Oops – a middle part of my post went missing – proving it is hard to write and proof read well – I mentioned 4 dimensions – I’ll try to recover/rewrite them shortly. Sorry.

LucianoQ January 25, 2011 at 6:53 pm

Sorry about that – here are two paragraphs that go before “Watching the painter”. (I think I know what happened – I used a bit of html to make some words bold, and probably lost and end tag! In my old age my programming skills and attention to detail are going down very rapidly :)

3 Competition. Fact of life is that competition is everywhere. The you people learning social skills are in fact competing – and learning to compete – who will be leader, who will be heard, who will be respected. Usually it is not just a matter of writing a program – but to do it better and more quickly than others. I can play tennis, but will never be champion. Many people apply for a job – only one gets it. So for the people who don’t have a job it is hard.

4 Depth.Each skill has a level, ie “depth.” Just like knowledge, the person with more knowledge has the advantage, so does the person with more skill. Watching a painter … sorry, the next is on my previous post. Sorry again.

Omit January 25, 2011 at 6:59 pm

It would all be so easy without the users.

Daniel Paull January 25, 2011 at 11:07 pm

In the real world the problem being solved is a moving target, be it because the client/user does not know what they want or because their needs change (often dramatically) over time. This is why software exists – it is intended to be malleable, adaptable and cater for change at low cost, say, compared to changing hardware. This is what makes “programming” hard – how do you solve a poorly specified problem, let alone one that changes over time?

If you think that programming is simple, then I suggest that you’ve never programmed well. The rest of the software development life cycle – eliciting requirements from customers, forming a functional specification, high-level design, maintenance, support, etc – all pale in comparison to the complexities of programming in the face of changing requirements.

Of course, if you assume that requirements can be well defined and are static for the lifetime of the product, then programming is trivial. I’ve never seen this assumption to be true.

Krishna January 26, 2011 at 12:39 pm

Daniel, managing changing requirements is the responsibility of whoever is in charge of project management. Yes, in the absence of effective project management, a programmer has to navigate changing requirements. But it is not difficult programming, it is just the tediousness of tearing down things you have written to meet a new requirement. It is not difficult as in tough to do, it is difficult as in “frustrating”.

Daniel Paull January 26, 2011 at 7:57 pm

Krishna – I have assumed that the requirements are managed just fine. It’s the “tearing down things you have written to meet a new requirement” as you put it that is the problem. I am sure you agree that the amount of affected code should be minimised – how is this achieved?

It is awfully flippant to claim that you can just throw away the affected code and write against new requirements; in the extreme case this means that you may have to throw away all the code and start over. This is expensive and time consuming – an absolute killer for software projects.

So, at the level of programming, what do you have to do to minimise the amount of code that is affected by arbitrary requirements changes? I am sure you do not have an answer to this question?

Dealing with people is nondeterministic since people are unpredictable. This appears to be the basis for why you think all the non-programming parts of software development are hard; you seem to think that determinism makes the world simple. What a load of rubbish! Not many people would think that “walking to the corner shop” is hard, yet we are dealing with people (eg, other pedestrians and drivers) along the way. There is a chance that I will get mugged or run over, but I still would not call “walking to the corner shop” hard.

Maybe this is what you mean – are you really talking about risk? Do you mean that software development carries more risk than programming?

You say “Programming is a learnable skill”. Rubbish. Programming is about problem solving (you state this yourself). I, as do many other programmers, rely a lot on intuition to solve programming problems. It is this intuition that is the risky part of programming. This makes programming nondeterministic. This makes programming hard.

Do you also claim that dealing with people is not a learnable skill?

Krishna January 27, 2011 at 12:02 am

Obviously, any software project has to be written with some malleability in mind. Reduce coupling. Single responsibility. Inversion of control. Yada, yada. Point is good software design and architecture. However, there are limits to how flexible you can make any software. Better programming can delay hitting those limits, but sooner or later you will. A side-point is that this flexibility does not come free. It comes at the cost (in money and effort) of better design and architecture. Whether that cost is justified is dependent on how many actual changes in requirements happen.

In any case, I don’t disagree with you that we should program (within reason) to minimize code that can be affected by arbitrary requirement changes. But what is your point? Is it hard to do? Aren’t there well-defined software design principles that can be used?

Programmers rely on intuition? Maybe true for some types of programming problems. But categorically for all programming? Do you have anything to back that up?

Daniel Paull January 27, 2011 at 12:53 am

“A side-point is that this flexibility does not come free. It comes at the cost (in money and effort) of better design and architecture.”

You assume that “better design and architecture” is proportional to the time an effort put in. Rubbish! Better design is generally inspired. Where’s my muse?

“In any case, I don’t disagree with you that we should program (within reason) to minimize code that can be affected by arbitrary requirement changes. But what is your point? Is it hard to do?”

Yes. Extremely hard.

“Aren’t there well-defined software design principles that can be used?”

Well defined? No. They tend to be abstract and have limited applicability. Theory on software development is still emerging – an active area of research with very few, if any, definitive answers.

“Programmers rely on intuition? Maybe true for some types of programming problems. But categorically for all programming? Do you have anything to back that up?”

Developing new algorithms is often done in part through trial and error. Form a hypothesis, implement it, test it, rinse and repeat to converge on a satisfactory outcome. To converge quickly, intuition is generally used over a brute force approach. Without intuition, software development would not just be hard, it would be NP-Hard.

Since you seem to think that both math and programming are similar in that they are learned skills. Perhaps you should go read some of what Alan Kay has to say about “math appreciation”. Basically, real math is non-deterministic, based on intuition and hard, what you learn at school is math appreciation. Maybe you will accept his opinion?

Daniel Paull January 31, 2011 at 8:38 pm

Your reply to the last post? Do you mean where you said, “Is it seriously so difficult to know how to do a build, use a VCS, write a unit test? I assume you know all the items you listed. And that is because you learnt it. Others didn’t. That is the difference between you and them.”

I chose to ignore that display of ignorance.

I could write an article in the style of what you have written here but from the other point of view. It would go something like the following:


Gathering requirements is easy. After all, all you need is people/social skills – skills that you’ve been building all your life. Gathering requirements can be described as a two step process:

1) Ask the client what they want.
2) Write it down.

All the stuff that follows on from this is extremely difficult. You have to use analytical skills that you have only been developing for a few years in contrast to the social skills that you’ve mastered over the past decades.

Blah, blah, blah.

I would then go on to list some of the difficulties of “programming” and flippantly dismiss the “software development” as trivial.

My honest opinion of all this is pretty conservative. We all understand the software development life cycle – I would not try to claim that any single part is harder than any other part. However, I am sure we all agree with the text books that errors made early in the cycle become increasingly expensive to fix as the cycle goes on. As such, it is imperative that errors in requirements are discovered and fixed very early otherwise they may cost a fortune to fix.

Maybe this makes “software development” important, but not necessarily hard.

Daniel Paull February 1, 2011 at 8:21 pm

Well, if you’re interested, I work in a variety of industries, but predominantly in Exploration and Mining. My expertise is in spatial data management and visualisation. I’m currently working on the application of high performance computing to visualisation of mining data. All the projects I work on are RnD, leaning more to the R side.

You might read that and think that it’s no wonder I find programming hard – they sound like tricky, complicated areas, and in part you might be right. However, the hardest parts of programming are cross-cutting and affect all programmers. Things like:

* Memory management
* Multi-threading
* Ensuring sufficient test coverage
* Optimisation
* Designing for scalability
* Data synchronisation (eg, keeping the GUI udpdated with respect tot he underlying data)
* Data migration and application upgrade paths
* Catering for changing requirements

Perhaps the difference between me and the developers you rub shoulders with has to do with the approach to building systems. You speak of programming as if you can just “crank the handle”; like you’re walking a well beaten path. This is not what programming should be about. When I feel like I am “cranking the handle”, then I investigate the possibility of building a “handle cranker” so I never have to crank the handle manually ever again. Now that’s programming! That’s innovation! That’s progress! And that’s hard.

What do I mean by “hard”? Hard is synonymous with “difficulty”, which in turn is a measure of likelihood of success. What is the chance that I will succeed with the building of my “handle cranker”? Every new software component you try to write carries a risk of failure, but what are the rewards if you succeed? This is the balance we need to strike. You seem to think that programming should take a conservative approach, while I am willing to take on more risk (and then reap the rewards that follow, of course).

Paul W. Homer February 2, 2011 at 11:53 am

With the exception of data migration, isn’t the list of problems you’ve described extremely well explored and heavily documented? Between all of the text-books, papers, web sites, conferences, etc. there is a phenomenal body of existing knowledge available out there that you can use as best practices or at very least a starting point. Why re-think, from scratch, problems that have already been deeply thought about?

A huge problem I see with our industry is that many programmers are going off in seclusion to working very hard on re-inventing solved problems. Problem solving is fun, but if someone’s already written a book on how to solve the Rubik’s cube, it seems unreasonable to me for someone to go off and spend years redoing that effort. Why make things harder than they need to be?

Paul.

Daniel Paull February 2, 2011 at 8:27 pm

“With the exception of data migration, isn’t the list of problems you’ve described extremely well explored and heavily documented?”

(Note: I should put exception handling on that list also.)

Each aspect that I mentioned may be easy to describe and understand in isolation, but when they are combined it becomes extremely difficult to reason about the execution of your program. Getting memory management right in the face of exceptions is extremely difficult let alone if you have a multi-threaded application.

There is lots of good techniques, RAII for example, that help a lot and should be used liberally. But these “best practices” don’t ensure that your program is correct is a holistic sense.

With modern programming techniques (use of DLLs, dynamic types, code generation, etc) we can’t even rely on static analysis tools like Lint to help us prove that our programs do what we intended. If you rely on the vigilance, reasoning and intuition of the programmer, then you’re bound to get it wrong. Unfortunately, this is the plight of all programming more complex than “Hello World” examples.

I wanted to find an article published in Dr Dobbs some years back about exception safety in generic containers, but it hasn’t jumped out at me. However, I found this one:

http://www.drdobbs.com/184403758

Which opens with “Let’s face it: writing exception-safe code is hard.” This article is written by Andrei Alexandrescu and Petru Marginean who are both well respected authors and programming experts. If they find exception safety hard, what chance does the average programmer have of getting it right?

You say that there is a body “of the text-books, papers, web sites, conferences, etc. there is a phenomenal body of existing knowledge available out there.” Yes there is. Paradoxically, the more I read the less I feel I know.

Paul W. Homer February 3, 2011 at 11:41 am

@Daniel & Luciano,

You both make an excellent point. At the architectural level, a big system is a series of very complicated trade-offs. Encapsulation helps, as do standards, best practices, theories and sticking with consistent paradigms, but unfortunately, while a lot of the ‘micro’ coding issues are discussed at length, these higher issues often get ignored.

But it is for exactly this reason that I think splitting the work in constructing software between the categories of ‘software developer’ and ‘programmer’ is extremely important. For exception handling for instance, someone well studied in the issues, and aware of the trade-offs should make the decisions on how to handle it in a particular way. As the lead software developer, they should set this convention for all of the other programmers to follow as they build up the system. And they should do it based on experience and knowledge, not just intuition and well-written blogs.

I don’t want to take too much freedom away from the programmers, but if they all choose to handle the same system-wide problem differently, chaos will erupt and the project stands a good chance of dying because of it.

And I think this is Krishna’s point with this post (and it certainly is one that I’ve tried to make in the past). Programming is about writing code from specifications. The specifications don’t have to be rigorous and fully detailed (it takes some of the fun out of it), but it can be seen as a deterministic activity. We need X lines of code, to do the following, and it should do that with standard algorithms, conventions and best practices. Software development on the other hand is all of the ‘soft’ skills that go into making programs useful. It includes creating the concept, identifying the real problem, analysis and a whole series of complex architectural and technological trade-offs. It also includes packaging, support and patching. It can’t be taught in high-school, and only really comes from decades of experience and a massive knowledge-base (and it gets harder every year). A big system needs a team of programmers, but it also needs a software developer to lead it to a fruitful conclusion. After five years, most programmers are considered senior, but I won’t put my faith in a developer with anything less than a decade or two. Still, we do, and this likely accounts for the ridiculously large massive failure rate that the software industry continues to live with, decade after decade. But it’s about time that we distinguished between these two related, yet very different roles, so that less people make the mistake in the future of hiring a programmer to lead their project, when what they really needed was a software developer.

Paul.

Daniel Paull February 3, 2011 at 8:20 pm

“Programming is about writing code from specifications.”

Yes, I agree that that is the intention. The hard part is writing code that is resilient against changes to the specification. Would you do agree that the specification will not specify how to handle changes to the specification itself?

“The specifications don’t have to be rigorous and fully detailed (it takes some of the fun out of it), but it can be seen as a deterministic activity”

If specifications are not “rigorous and fully detailed” then every developer will offer a different interpretation of the specification and design/implement something completely different – all solutions will satisfy the specification. Maybe now you starting to understand where non-determinism creeps in?

My personal view, and the basis for the methodology that my company (www.thinkbottomup.com.au) follows, is to encompass requirements rather than merely meeting them. This allows us to build up software that solves classes of problems rather than individual, specific problems. The result is the ability to reuse of much of our software across seemingly unrelated projects. This would most likely be lost if I wrote directly to your specification.

“We need X lines of code, to do the following, and it should do that with standard algorithms, conventions and best practices.”

That’s easy to say, but the devil is in the details. Abstract concepts like “best practices” do not tell you how to handle any specific situation. I’d rather focus on utilities and frameworks that do things “once and for all” to remove the human element as much as possible to reduce error rates. See my comments earlier about building a handle cranker.

If you can draw up a “Guide to Deterministic Programming” that is not full of holes, contradictions and controversy then I will eat my own hat.

“It can’t be taught in high-school, and only really comes from decades of experience and a massive knowledge-base (and it gets harder every year).”

I thought you were talking about programming when I read that as it rings so true.

“But it’s about time that we distinguished between these two related, yet very different roles.”

Sure, why not. However, the distinction should not be “Programming is Easy, Software Development is Hard” for two reasons:

1) It’s not true
2) Given (1), you alienate the programmers on who you will later come to rely.

If you want to clearly differentiate the roles (which I though was done back in the 70′s with the waterfall model) then do so in a factual, unbiased, informative, non-judgmental, non-arrogant manner.

Paul W. Homer February 3, 2011 at 11:08 pm

Hi Daniel,

Interesting points, but I think a detailed response would easily exceed the bandwidth potential of these comment boxes. I suggest that if you’re interested in continuing this discussion, we move it to somewhere that allows wider text. I have a non-used Google Group that might be useful, but anywhere were we can type in longer points would be fine.

http://groups.google.com/group/the-programmers-paradox

Paul.

Daniel Paull February 3, 2011 at 11:13 pm

Nah, I think this forum is fine. I’m going to tune out of this conversation soon anyhow.

Paul W. Homer February 4, 2011 at 12:25 am

OK, I understand.

Paul.

merc January 26, 2011 at 3:44 am

actually in both programming and software development, we often apply what we learn, and when we meet a new scenario, we start to think of new ways to handle the situation. Often people would say that programming is a hard science which once learn can be applied iteratively, however when a programmer meet a new problem, he or she also have to spend time overcoming and finding the solution.

For software development, we often also apply management skills which we have learnt or encounter before and further adapt from there. Think of it as, a manager never handles an egoistic person before, after going through once, he/she would be better equip to meet the next egoistic person. Think of it as human profiling and adaptation.

In fact, I think both programming and software development are skill sets that would require a person to practice over time to get better at it. No one is born with either skill. For programming, there are various skills which allow a person to start of more prepare, same for software development. Because of the constant need to interact with people, human and behavioral psychology profiling are possibly the necessary skill set.

Krishna January 26, 2011 at 12:42 pm

Merc, what I would suggest is that the universe of programming problems that a typical programmer deals with is considerably smaller than the universe of different personality and human resource issues that a person has to deal with. As an example, take any project. You will have one problem and many people to interact with. Defining the problem becomes difficult simply because you have so many people with their viewpoints, ideas and suggestions.

Daniel Paull January 26, 2011 at 8:03 pm

You should also note that we need to arrive at a consensus on many issues with in the team of programmers. All the same problems of dealing with people arise.

Have I just proven “software development” to be a subset of “programming”?

LucianoQ January 26, 2011 at 9:55 pm

You are well on the way. The topic has fractal qualities.

Rob Kraft January 26, 2011 at 11:07 am

Great post. Nice delineation between programmer and software developer. We all know that some people writing code are better at it than others, and more valuable to a company. It is difficult to quantify and measure the differences. This is a step in helping us identify the differences.

LucianoQ January 26, 2011 at 11:38 pm

If programming is easy, why is there a huge difference in productivity between good and bad programmers? A study in 1966 established: “When a programmer is good he is very very good, but when he is bad he is horrid” (written before “political correctness” for gender). http://canonical.org/~kragen/sackman-erikson-grant-1966.pdf

Kragen goes further and says that some programmers are so bad that they have negative productivity – ie they are incapable of finishing a project. Something I have been aware of for decades.
http://www.mail-archive.com/kragen-tol@canonical.org/msg00250.html

IT is still partly in the “alchemy” days. The IT industry is still undecided about programming languages, web app architecture (SOA vs REST), design (UML vs DSL vs nothing), database structures (RDBMS vs OODBMS), data warehouse structures (Inmon vs Kimball). Many programmers do not even recognize, when debating some point, that they are suffering from the “blub-paradox”
http://paulgraham.com/avg.html
- ie, to paraphrase “if your approach uses a concept that I don’t understand then it cannot be better than my approach”.

To be a good programmer one must be skilled in many things – Kragen mentions a few in the reference above – eg one should be aware of features from different types of programming languages (there is a huge list of featrues at C2
http://c2.com/cgi/wiki?KeyLanguageFeature ).
How many programmers here have heard of the languages he mentions, and understand what is special about them?

To be a good programmer is far from easy. Particularly when the IT industry itself does not even know what’s good.
So, how can programming be easy?

Usually a large proportion of any task in any domain, say 90%, is mundane and repetitive. But the remaining bit is crucial. It is how people deal with the remaining bit that separates the good from the bad, the hard from the easy, “the men from the boys”.

Programming clearly includes debugging. Is debugging easy? Well, usually with good tools it is. But occasionally it can be very very hard. So, sometimes programming is very hard.

Be careful too not to restrict the definition of programming too much, ie just to the mundane bit:

Should a programmer know how to test? Unit test? Integration test?
Should they know how to do a build? How to use a VCS? When to branch? What types of branches to set up?
Should the programmer be able to estimate how long to do a programming task? What does the task include?
Should the programmer be able to design a UI?

At this point you might say – ah, but designing a UI is not programming.
True, just like reading music is not the same as playing the piano. But, to be able to play the piano one *should* be able to read music.

How many times have you heard:
programmer – I’ve finished it – it was due today – here it is, where is my bonus?
boss – have you tested it?
programmer – no, I need another week for that
boss – then you haven’t finished
programmer – you didn’t say you wanted it tested as well!

I don’t consider this “programmer” to be a “programmer”. Programming is a lot more than coding.

Maybe the title of this should be “coding is easy, programming is hard”.

Krishna January 26, 2011 at 11:44 pm

All of what you say is valid. Question is: Are the bad programmers bad because this is hard to learn? Or they didn’t bother to learn?

Is it seriously so difficult to know how to do a build, use a VCS, write a unit test?

I assume you know all the items you listed. And that is because you learnt it. Others didn’t. That is the difference between you and them.

Joel K March 28, 2011 at 5:41 pm

The ideas in programming are fundamentally hard to grasp because the ideas are so far displaced from anything normal people are used to. Creating code to solve a problem is not a linear process, it requires experience and creativity.

Gretgor June 20, 2011 at 12:27 am

While I do agree that programming is not easy, I think your argument (e.g. that programming has more to do with debugging and manipulating technology than with mathematical logic) is not a very strong argument. If anything, it means that the IT community, to this day, has failed to develop good enough tools to turn logical ideas into code.

LucianoQ February 2, 2011 at 12:28 am

Productivity rates of programmers varies enormously – much more than in other industries.

Defect rates in software is much higher than elsewhere.

Programmers are notoriously bad at estimating.

Programmers, and those that teach programming, are nowhere near resolving the issues about good programming, not even “trivial” stuff like naming standards.

Would it be like this if programming is easy?

LucianoQ February 2, 2011 at 10:30 pm

@ Paul Homer – you say: “Why re-think, from scratch, problems that have already been deeply thought about?”

Ok – there are many articles about Java checked exceptions. Unfortunately they fall into two different, incompatible highly conflicting approaches. What should a programmer do? How to choose without rethinking? Do you know what approach to use? If you do I bet I can find well thought out articles by people much smarter than me which would contradict you.

Gerwin April 21, 2011 at 6:49 pm

I own a software company in the Netherlands. It is easy to find people that can code. I find it more difficult to find people that can aim that code towards the business problem…

Gretgor June 20, 2011 at 12:22 am

Wanna see you try to formulate a subexponential algorithm for an NP-hard problem, THEN we can say programming is easy.

Krishna June 20, 2011 at 8:15 am

Is writing algorithms for NP-hard problems a common activity for most programmers? You should look at what the average programmer does and try to discount what I have written.

Paul W. Homer June 20, 2011 at 10:11 am

Writing the code for a subexponential algorithm for something like SAT isn’t all that bad. If you can do it, it will be fairly short and contain a lot of ‘if’ and ‘for’ loops that apply some nifty abstract algorithm to a data-structure. More complex that a weight-balanced N-ary tree, but if you have the algorithm in front of you, you just need to follow it.

Finding a subexponential algorithm however, is a very daunting theoretical problem that many people have been working on for fifty years. It involves a deep understanding of both mathematics and computational theory. It is a research problem, not an applied programming one.

Paul.

hashtech2012@gmail.com September 20, 2011 at 6:08 am

Making software is almost like making a war. To keep an eye on the goal, a developer must often filter out unimportant noise. This noise can be more detail by fellow programmers or a dose of the management of anxiety, but the real success is to decide what is important and what is not.

Krishna January 31, 2011 at 5:30 pm

Daniel, didn’t know if you saw my reply in the latest post.

Krishna February 1, 2011 at 7:09 pm

See, maybe I am wrong. But if I thought what you were saying were true, I would believe it. I think the disagreement perhaps stems from different experiences by you and me, or perhaps my lack of experience.

So maybe, if you are willing, talk about the kind of projects that you do and why you feel that the programming (or coding) there is much more difficult than the other aspects of the development process.

Here are the assumptions that I am working with
* I don’t think that programming is easy as in any person off the street can pick it up. But a person with academic competence should be able to pick it up with sufficient effort.
* There are some types of programming that are very hard (especially those with deep roots in advanced math). But I don’t believe that to be true for the majority of programmers who work in business apps.
* There are frameworks and good industry practices for making applications maintainable. You can do a lot without reinventing the wheel. For the typical application, how much extra work are we talking about?

So, in all earnest, it would help the discussion if you can explain your experience. Perhaps that would even help me change my viewpoint.

Comments on this entry are closed.

{ 5 trackbacks }

Previous post:

Next post: