When I was a young software programmer, I had to develop features with estimates given by more senior programmers. If more time was required for the task, I had to explain the reasons - and I’d better be convincing about that. After some years, I became the one who had to provide feature estimates, but this did no mean it was easier: if the development team took more time to develop, I had to justify it to my management. Now, after even more years, I have to provide estimates for entire projects, not just fine-grained features.
But in essence, what do we need estimates for? For big enough companies, those are required by internal processes. Whatever the process, it goes somewhat along these lines: in order to start a project, one needs estimate in order to request a budget, then approved (or not) by management.
Guess what? It doesn’t work, it never did and I’m pretty sure it never will.
Note: some organizations are smart enough to realize this and couple estimates with a confidence factor. Too bad this factor has no place in an Excel sheets and that it is lost at some point during data aggregation :-(
Unfortunately, my experience is the following: estimates are nearly always undervalued! Most of the time, this has the following consequences, (that might not be exclusive):
- In general, the first option is to cancel all planned vacations of team members. The second step is to make members work longer hours, soon followed by cutting on week-ends so they work 6/7. While effective in the very short-term, it brings down the team productivity very soon afterwards. People need rest and spirit - and developers are people too!
- After pressure on the development team, it's time to negotiate. In this phase, project management goes to the customer and try to strike a deal to remove parts of the project scope (if it was ever defined...). However, even if the scope is reduced, it generally is not enough to finish on budget and on time.
- The final and last step is the most painful: go back to the powers that be, and ask for more budget. It is painful for the management, because it's acknowledging failure. At this point, forget the initial schedule.
- Meanwhile and afterwards, management will communicate everything is fine and goes according to the plan. Human nature...
In some cases (e.g. a bid), that’s even worse, as the project manager will frequently (always?) complain that those estimates are too high and pressuring you to get lower ones, despite the fact that lowering estimates never lowers workload.
You could question why estimates are always wrong. Well, this is not the point of this post but my favorite answer is that Civil Engineering is around 5,000 years old and civil engineers also rarely get their estimates right. Our profession is barely 50 years old and technology and methodologies keep changing all the time.
I’m not a methodologist, not a Project Manager, not a Scrum Master… only a Software Architect. I don’t know if Agile will save the world; however, I witnessed first-hand every upfront estimate attempt as a failure. I can only play this game of fools for so long, because we’re all doomed to loose by participating in it.