In a perfect programmer utopia, programmers would be no deadlines. We’d spend all day toiling away in front of our computers, and happily hacking away at our applications, without bosses or scrum masters or product owners breathing down our necks about when the final project will be complete.
Sadly, we don’t get to live in that world. We work in a world with aggressive deadlines, requests for new features we aren’t quite sure we developers can figure out how to implement, and a constantly changing technical landscape where we developers need to constantly keep up with new technology stacks, development methodologies and software developer tools.
You would think after decades of modern software development (we’re talking at least fifty years, probably more), that the science of predicting how long it takes to finish developing a software application from start to finish, has been fine tuned and honed down to the exact hour and second when the development is done, right?
I’m not convinced.
The fact that software project development methodologies have evolved over time show we’re still trying to get better about software project estimation.
The traditional waterfall software methodology, was the first stab at figuring out how to break down project requirements into milestone and a timeline schedule which would show when each major piece of an application would be completed.
I’m not sure if the waterfall methodology was actually invented by anyone specifically. To me, it’s seems like a logical way to break down a set of requirements.
Waterfall software development is broken down in several stages. It’s only when the first stage is complete, that the next stage can begin, and only when the second stage is complete, the third stage can begin, much like a real waterfall can flow down a set of steps, hence the term.
Usually a business analyst would start the first stage of waterfall development by gathering the customer requirements. As was said before, each stage of the waterfall methodology must be complete before the next stage can begin. So while the business analyst is busy gathering requirements, the software developers and quality assurance testers are fiddling their thumbs, waiting for the business analyst to complete the “project spec”.
And only when the software developers completely finish coding the application based off the business analyst’s project spec, can the quality assurance testers begin their testing. Then there’s a ping pong match between the QA testers and software developers, as the qa testers find and report software bugs and defects in the code and the developers go back to the code and put the effort into squashing the bugs.
And finally the documentation writers (assuming there are documentation writers attached to the project) can begin documenting the official functionality of the application.
As you can probably already see, lots of potential problems can happen when you tackle a software project using the waterfall methodology. What if the business analyst misunderstood something regarding the project requirements? What if the misunderstanding is so great, that the final product, even if completed, doesn’t provide the actual functionality the customer wants?
So the business analyst must rewrite or come up with a new project spec, which must then get relayed to the developers to translate into code, which then must be passed back on to the quality assurance testers, and finally to the documentation writers.
The core weakness behind waterfall methodology is it’s extremely DIFFICULT to make changes in the application. When change is necessary, it creates ripple effects throughout all the stages of waterfall development.
It also makes trying to figure out exactly WHEN a waterfall project will be complete, extremely challenging. Waterfall timeline schedules can stretch for months or even years.
The Agile development methodology was born out of a desire to improve the way project estimation and development is handled.
Agile development acknowledges that CHANGING REQUIREMENTS should be embraced. No longer is software development broken down into separate waterfall steps. It also acknowledges that there is no such thing as a “final product”.
Many companies now are following a particular subset discipline of agile called scrum. A scrum team consists of a product owner which relays her product requirement desires to the developers, database administrators, and any other IT personnel needed for the project.
Instead of planning out the entire project ahead of time, a scrum team breaks down a project into smaller slices of requirements.
A scrum team schedules work tasks in a something called a sprint, which is usually 2 or 3 weeks. During a sprint, the product owner and the development team work on a set of “stories”, that is some sort of component of the overall project. During that sprint, “feature creep” is not allowed. That is, the sprint team will only work on the agreed upon set of sprint stories.
When the sprint is complete, the development team and product owner go back and plan the next chunk of project requirements. There is no defined “end” to the project. It’s a continual cycle of delivering smaller “chunks” of the overall product.
My own company has embraced the agile/scrum development methodology for going on four years now.
Has the agile/scrum methodology improved software estimation and project scheduling?
Having worked on both sides of the fence, I see pros and cons to both methodologies.
We’ve gone over some of the weaknesses behind waterfall development. But that’s not to say there aren’t advantages to waterfall. The role of the business analyst in waterfall development, is quite valuable.
The business analyst is out there, spending the majority of their day interacting with the customer and hammering out a nicely detailed project spec. They know the right questions to ask the customer and will translate that information into documents that will contain user screen layout (assuming the project is a full blown visual application for end users).
This project spec makes the life of the software developer a breeze. Relatively speaking at least.
The project spec guarantees that the business requirements for the project have been formalized.
In agile/scrum development, we will NEVER have a complete idea of the complete project requirements in one shot. The very nature of agile development is we only concentrate and work on a small “slice” of an application, even if we don’t know the complete picture, which is more often the case than not.
I personally miss the business analyst. It’s sometimes worth it to spend the extra time and effort to go over the business and project requirements of the application. But with agile development, that’s really not possible, at least to the extent that a business analyst exerted, gathering project requirements.
Agile/scrum does make it easier to estimate timelines, simply for the fact that you never work on more than a smaller subset of the application requirements at any given time. And if something CAN’T be done by a certain deadline, then renegotiation between the product owner and dev team happens, until the set of agreed work IS doable.
But whether a dev team uses waterfall or agile/scrum, the fact of the matter remains that estimating software deadlines will ALWAYS remain difficult.
A dev team may run into some difficult technical obstacles that grind development down to a halt. An all hands critical production emergency that affects company revenue, requires that all software teams pitch in until the emergency is resolved. There’s simply lots of uncontrollable factors that often affect software development.
I’m not sure there’s ever going to be some easy “silver bullet” solution that will resolve the difficulty of software estimation. Agile/scrum seems to be the current popular development methodology, but I have no doubt it will continue to evolve.
I think whatever becomes the new methodology will learn from the flaws AND pros of waterfall and agile, and we’ll see some new hybrid methodology which will get us closer to better and more accurate software and project estimation.
