Where Agile Fails

If you're a software developer looking to work in a modern IT organization, it would behoove you to get at least familiarized with Agile development. More and more companies and organizations are embracing the agile development methodology, and like it or not, it is here to stay for the duration. Not to say something else won't replace it, or that Agile will evolve into something new (I'm betting it will), but it is certainly gaining key importance in many IT organizations (including my own). A brief definition of Agile. It's a movement that came into being as a soft of "revolution" against waterfall development. To understand Agile, it's necessary to understand what Agile was trying to change. Waterfall development, at its core, is basically breaking up an entire software project into distinct "phases". Each phase is sequential and the next phase can only being once the previous phase has been completed. The major phases include the following:

1. The design
2. The implementation
3. The testing
4. The deployment
5. The documentation

Typically, a business analyst or project manager would handle the the first phase of design. There would be lots of meetings and discussions between the business analyst and the project customer until a final project "spec" document was created, that would include all the project requirements. At no point in the design phase, could any other phases proceed. Once the design phase was finished, it would "spill over" to the phase below it, (much like a waterfall, hence, the name). The implementation phase is where the software developers (and any other relevant members of an IT team) would take the project requirements and begin translating those project requirements into actual software. It's important to note that the first phase has been COMPLETED. The business analyst is no longer needed at that point to make any changes or additions, and will usually be assigned off to other projects at that point. Note also, that the QA testers, the deployment team, and the documentation writers, are still waiting until the development phase has been completed, and the project can then "spill down" to the testing phase. And so on and so on, until the final phase has been completed. For many many years, this has been the traditional way software development has been accomplished (and still does at many places, regardless of the rise of the Agile movement). Some of you out there can probably see the cracks of flaws behind this waterfall approach to software development. Such as what happens when the project requirements have to change? In that case, you pretty much have to go all the way back to step one and start over! Waterfall development is painful when software project requirements have to frequently change over the course of a project. Thus was born the whole agile and scrum movement. Instead of attempting to plan the entire project ahead of time, agile/scrum methodology breaks up a project into smaller iterative steps (called "sprints"), that allow a software development team to deliver smaller pieces of the overall project to the customer. During each sprint, the customer (called the Product Owner) can voice their own feedback and determine if a "course correction" is needed or not. That is the essence behind agile ... the ability to deliver smaller portions of the overall project and make course corrections when needed, if the customer is not satisfied with the way the project is currently headed. Many companies are completely embracing this newer methodology. And there are no doubt, many success stories and supporters behind this movement. But having been in the software development profession for the better part of two decades, you get a sense of "forest above the trees" perspective, whenever new ideas, methodologies and movements arise in the software profession. Every new movement always professes itself to be the "silver bullet" solution that promises to make software development perfect. From a technological perspective, it happens a LOT. The OOP (object oriented programming movement) of the 1990s, promised to make software development a Utopian programming paradise. Programming languages like C++, Java and C# came into being to welcome OOP type programming. The advent of XML promised to make universal data exchange a breeze. The advent of XML/WSDL SOAP services promised to make different applications communicate to each other, child's play. Nowadays, it's all about Javascript and the push to use Javascript on all three major tiers of application development, the presentation UI layer, the middle tier business logic layer, and the backend data layer. NodeJS, AngularJS, ReactJS, and the other bazillion javascript frameworks that are all shouting to be heard to promise a Utopian programmer's paradise (sound familiar)? If I sound a little jaded, it's probably because the longer you stay in this field, the more you see one thing, sooner or later, get replaced by something new. And I am just as confident that agile/scrum will morph or get replaced by something new, just as classic waterfall development. I don't think agile is a "one size fits all" type solution. There are scenarios and situations where waterfall actually works better (certainly not a perfect solution) than agile. In many Professional Services organizations, the software development team is there to service the customer and come up with a final project deliverable at a certain delivery date. That delivery date can often be NON-NEGOTIABLE. In the case where the delivery date cannot or will not change, the customer often doesn't CARE about partial deliverables that an Agile team "sprints" on. The project requirements are essentially "set in stone" ... the customer has dictated a deadline with the understanding that it's on them, if that "drop dead date" changes. In this scenario, it's actually counter productive to tackle these kinds of projects using agile. Waterfall development is much more suited to project situations where the project requirements and deadlines don't change. Of course, it doesn't guarantee success, but in my opinion, it seems like wasted effort delivering portions of a project that the customer really won't care about until the final deliverable is complete. I know that other professional services organizations outside my own company, continue to embrace waterfall development for this very reason. I am sure there are other scenarios that don't fit well within the agile methodology of software development. There will always be new methodologies, new technology, new ways of doing things that are right around the corner. But I think there's value in looking back at where you've been ... you never know if you're leaving behind something that still holds value.