The EgoEstimation is in the realm of the what ifs and and maybe ifs and is usually grounded more in guesswork than in science. Egos often get in the way of a good estimates when the programmers are involved. As for managers the pressure to deliver features in a certain amount of time instead of getting an accurate view of how much time a task will take.
The UnknownsOne of the big differences between Software Construction and physical construction is the amount of unknown involved. Out of all the different software projects I have been involved in, not one of them was exactly the same or even close to being the same. In fact most all of them involved a drastic change in technologies and in business domain. This sort of shift makes it difficult to predict what is going to happen in the project. Inventory applications have different kinds of problems than shrink wrap CAD programs for instance.
However, when I had my house built I asked how long it would take. I was given a reply that it always takes between 2 and 4 months and they are pretty sure it was going to take 3 months. It took exactly 3 months to complete. The reason they could be so accurate and so fast at putting my house up was not that it was easy to build a house but because there were really very few variables and unknowns in building a house. They had built several of these houses and they were the same house with minor changes that made my house look a little different from another house down the road that they had built. Experience building a house almost identical to mine is what made them so accurate at predicting how long it would take to complete mine.
ExperienceI would submit to you that this type of experience for building software is almost never going to happen. However, professionals who have been working in the industry find it difficult to admit to upper managers that they do not have enough experience in the areas that they are estimating to give an accurate estimate. Often times developers understand the problem is lack of experience but are too afraid or think it would be foolish to inform their managers.
One of the biggest reasons for lack of experience is the size of the knowledge base needed to develop software. The size of the tools we use is enormous and they are adding to them every day. What you may have picked up last week can be eliminated by new features and fixes to the tools. Good developers learn how to be good at figuring out 3rd party and built in tools rather than memorizing them and mastering them.
The Dynamic Nature of Programming SoftwareI have never been a plumber but I imagine that installing plumbing in a new house remains the same for some time maybe even years before advances in technology makes them learn new things. I doubt that in the middle of fixing a sink they realize the pipes that were top of the line 6 months ago are obsolete now and need to be completely removed and replaced.
The dynamic nature of the business makes mastering programming virtually impossible for us mere mortals and therefore also impossible to predict.Estimates without padding to account for unknown api and technology shifts are naive to say the least and foolish and stupid to say the most. For this reason I have seen frustration on the side of both project managers and programmers alike. The programmers know why its a waste of time but the reasoning makes them sound like they don't know the profession.