Why Estimates are Worthless

Software estimates are most often completely worthless and a waste of time for everyone involved. Its a waste of time for the manager because the estimates that they get are usually shallow and only reflect what the manager wants to happen. Its a waste of time for the developer because of countless hours pondering what something will cost and why it would cost that, only to find out after completing the task how many things he/she missed.

 

The Ego

Estimation is in the realm of the what ifs and and maybe ifs and is usually grounded more in guesswork than in science. Egos often get in the way of good estimates when the programmers are involved. As for managers the pressure to deliver features in a certain amount of time instead of getting an accurate view of how much time a task will take.

The Unknowns

One of the big differences between Software Construction and physical construction is the amount of unknown involved. Out of all the different software projects I have been involved in, not one of them was exactly the same or even close to being the same. In fact most all of them involved a drastic change in technologies and in business domain. This sort of shift makes it difficult to predict what is going to happen in the project. Inventory applications have different kinds of problems than shrink wrap CAD programs for instance.
However, when I had my house built I asked how long it would take. I was given a reply that it always takes between 2 and 4 months and they are pretty sure it was going to take 3 months. It took exactly 3 months to complete. The reason they could be so accurate and so fast at putting my house up was not that it was easy to build a house but because there were really very few variables and unknowns in building a house. They had built several of these houses and they were the same house with minor changes that made my house look a little different from another house down the road that they had built. Experience building a house almost identical to mine is what made them so accurate at predicting how long it would take to complete mine.

Experience

I would submit to you that this type of experience for building software is almost never going to happen. However, professionals who have been working in the industry find it difficult to admit to upper managers that they do not have enough experience in the areas that they are estimating to give an accurate estimate. Often times developers understand the problem is lack of experience but are too afraid or think it would be foolish to inform their managers.
One of the biggest reasons for lack of experience is the size of the knowledge base needed to develop software. The size of the tools we use is enormous and they are adding to them every day. What you may have picked up last week can be eliminated by new features and fixes to the tools. Good developers learn how to be good at figuring out 3rd party and built in tools rather than memorizing them and mastering them.

The Dynamic Nature of Programming Software

I have never been a plumber but I imagine that installing plumbing in a new house remains the same for some time maybe even years before advances in technology makes them learn new things. I doubt that in the middle of fixing a sink they realize the pipes that were top of the line 6 months ago are obsolete now and need to be completely removed and replaced.
The dynamic nature of the business makes mastering programming virtually impossible for us mere mortals and therefore also impossible to predict.Estimates without padding to account for unknown api and technology shifts are naive to say the least and foolish and stupid to say the most. For this reason I have seen frustration on the side of both project managers and programmers alike. The programmers know why its a waste of time but the reasoning makes them sound like they don't know the profession.

Managerial Tactics

For a-lot of managers estimates are just a way of cracking a whip and motivating team members with a deadline. It is said that "If you give someone a week to complete a task it will take a week if you give them a day it will take a day." Unfortunately this is a traditional management technique that fails to realize one thing. We are not building cars or cookie cutter houses. No amount of motivation is going to get a developer to write a program in 2 days that takes a week. One of the biggest problems is no one actually knows a realistic time for the task to be completed. So in an attempt to motivate employees an laughable estimate is put into law. Because there is no possibility to hit the deadline the developers get demotivated and start ignoring all deadlines.

Psychics and Crystal Balls

Think twice before pushing for estimates on all of your projects. There are alternatives to controlling budget other than trying to predict the future like you are a psychic with a crystal ball. Instead try to reduce the scope of your project and reduce the risk. In other words do less and deliver on time. This has been a corner stone of the Agile process and I would say its one of the main reasons that Agile has been successful. Stop wasting your time going to software fortune tellers and just manage the project right. Don't get out out of control with features and make sure you are releasing features on a regular short cycle.

Good Reason for Estimates

I understand that some projects are different. Some projects involve a lot of upfront analysis and careful planning. Sometimes we are trying to determine if a project is worth it or if we should try something else. Fixed bid projects for contractors/consultants cause a hurdle for the project and the powers at be and large corporate policies can keep projects from working on an incremental approach. Just remember that most developers and managers just can't estimate. Most of the time it is just a waste of time even if you think there is a good reasons to estimate. It just gives you a false sense of security. And the way software projects work, you probably wont realize the project was never going to be done anywhere close to the estimate until the deadline is almost done and everyone looks up and says "How the hell are we going to be done next week?". In the end even if you need it to work it just doesn't.