Why Writing Software Is Hard

Scott Rosenberg just published a new book called Dreaming in Code about a project to create a new personal information manager called Chandler.  As many software projects, this one is late.  Rosenberg was recently interviewed about the subject of software projects falling behind. He has some interesting thoughts on the issue. 
Are projects like Vista just poorly run projects?  Were they late because they were undisciplined?  Or was there something else at work?  Rosenberg doesn't give a clear answer but he does seem to indicate that it's impossible to estimate software times because it has never been done before.  He says,
"[A]s long as you are not trying to do something new -- if you are doing something that has already been done -- then you have a frame of reference to estimate how long it is going to take, and to guess how many people are going to be required, and so on. And of course in other fields of engineering where we do the same thing over and over, the same is true. If we are going to build a house, there are always imponderables, and every site is different, but you have a rough idea: two-story house, and such and such a set of features, it is going to take a certain amount of time."
This is true.  If you've done it before, you can probably estimate it.  However, if you haven't done it before--and with software you probably wouldn't be writing it if you had--there is really no way to accurately know how long it will take.  I think this is largely true, as I have argued elsewhere.  Software is unforgiving (the reason for most security holes).  Writing a program is more like research than simple engineering.  No one blames Monsanto for being late developing a cure for cancer (that I know of anyway) but Microsoft gets killed for being late with Vista.  Part of that we, as programmers, bring on ourselves.  To be a good programmer means you are smart.  Mere mortals often have a difficult time understanding how to write even simple programs.  Because of that, we often times overestimate our abilities and underestimate the size of the problem.  We don't have the information to make an accurate calculation but we refuse to admit that. Instead, we insist that everyone else trying to write software was just undisciplined and that we can do it better.  Unfortunately, we can't.
My takeaway from all this:  don't try to estimate the whole project up front.  Stay agile.  Come up with a list of work and churn through it at whatever rate is possible.  Have lots of checkpoints and don't be afraid to change your plans.

http://blogs.msdn.com/b/steverowe/archive/2007/02/03/why-writing-software-is-hard.aspx