Software development changes quickly. It seems like every day, there is a new programming language or framework that pops into existence. Or a shiny new development IDE.
But it isn’t just the technology that changes. The way developers go about using that technology to make software changes as well.
These days, it’s hard to find a company or organization that doesn’t proclaim they’re an Agile or Scrum organization. As a software developer, you will most likely be expected to practice at least SOME degree of agile or scrum methodology in the way you developer and deliver software.
However, not everybody believes Agile and Scrum are beneficial. In fact, there are many vocal opponents who believe they are some of the most non productive development methodologies for the field of software engineering.
I’ve got my own thoughts and retrospective analysis regarding agile and scrum methodology, but I think it helps to understand some historical context and get a better understanding on how the overall agile movement came into being and what exactly did it replace.
You may have heard of the term “waterfall development”. Before the big agile and scrum methodology became popular, waterfall development was the traditional way software development teams tackled big software projects.
The reason why it was coined “waterfall” was because of the way that each phase of the project was structured.
- Project requirements/analysis
In a traditional waterfall project, a project usually kicks off with a set of initial requirements by the customer. The customer is anyone who desires some end result software deliverable to come out of the final phase of the project.
The customer usually interacts with an assigned “Business Analyst” who works with the customer to write down all the features and requirements into a cohesive and comprehensive document, often referred to as a “project spec document” or “scope document” or a myriad of other terms to indicate it contains the complete project requirements of the customer.
The business analyst often possesses business expertise in the same general area as the customer(s). For instance, say someone from the sales department of a company wants new software to help track potential new customers. The business analyst will most likely possess subject matter and business expertise in the world of sales.
This kind of business knowledge is valuable to a business analyst … speaking the “language” of a customer, often leads to better written project specs, needed for a project.
The reason why waterfall development is called “waterfall”, is that none of the other major phases of the project can begin until the initial requirements/analysis phase is absolutely complete … like water flowing over successive “steps” of a real waterfall.
The intention of the requirements/analysis phase of a project, is that the business analyst will continue working on the project spec until every single requirement of the project is written into the spec.
This can weeks. Or months. Or even into years! All the while, the rest of the project development team, the designers, the programmers, the QA testers, the tech writers, their services won’t be needed until the project reaches whatever waterfall phase is relevant for them.
The design phase is where all the actual TECHNICAL requirements of a software project come into play. This can involve discussions about what user interfaces will look like. What kind of overall tech stack will be used. And a myriad of other technical details need to be nailed down before a single line of code can be written. Usually a tech lead or software architect will be responsible for this phase of the project.
The coding phase is ready to begin when all the technical requirements are completed. This is where the software coders take the technical spec written by the designers, and translate that into actual code.
Once the coding phase is complete, the QA testers can begin to test the code and ferret out bugs and flaws in the code produced by the programmers.
The coding/testing phases are closely tied together. It’s only when the QA team finally signs off that the project is of a good enough quality for customer use, can the deployment phase begin.
Finally, once the application is successfully deployed into a real production environment, the overall project goes into a maintenance mode.
The key take away behind understanding waterfall software development, is that at any given moment, there can be only ONE phase of the project that can be actively worked on.
The other major take away is that once the initial project requirements have been nailed down and codified into an official project spec, the spec absolutely CANNOT change, not unless approved by either the customer and or upper management.
The implications behind this can be pretty severe.
First, what happens if the business analyst wrote down some project requirements that were incorrect? After all, we’re all human. We make mistakes and we often misinterpret what others try to communicate to us.
Tough luck. The project will steamroll its way down to the very end of the waterfall phases, without the ability to make “course corrections”.
Secondly, and even worse, what happens if the final deliverable doesn’t satisfy the original customer’s requirements?
Obviously this is the WORST kind of disaster that can happen to a software project. Ultimately, the software project is a failure if it doesn’t meet the core requirements of the customer.
The rise of Agile and Scrum
New software development methodologies eventually came into existence to address behind major issues associated with waterfall development.
- Project requirements can frequently change
- The project spec may be inaccurate
As strange as it may sound, a customer often doesn’t know or understand exactly what she wants in a final software deliverable.
Sometimes the customer can change her mind. Or events can happen at a company that can force policy changes in a software project. For example, the CTO or CIO may suddenly lay down a new directive to all software development teams to use a new piece of technology. Or incorporate enhanced security into a project.
The major problem with waterfall development is that it can’t incorporate major changes in requirements without redoing a lot of effort and/or changing the original timeline of the project.
The other problem, is related to the first issue. If project requirements can frequently change, how can you possibly write a complete and accurate spec of the entire project?
That’s why the agile and scrum movements have come into play.
The essence of agile is that it’s IMPOSSIBLE to know the exact requirements of a project. Agile also acknowledges that project changes are inevitable.
What agile and scrum development does differently from waterfall development is break down those steps or phases of waterfall development, into iterative and smaller portions of development.
Instead of the customer waiting until the absolute END of a project, to find out if the final deliverable is satisfactory or not, the customer is intimately involved through the ENTIRE phase of agile development.
The customer is actually shown incremental pieces of the overall project.
For instance, if an application requires a log in process, an agile team would work on that functionality, perhaps involving a login screen and all the functionality involved, behind that process.
Once the login screen is complete, the agile team would showcase that deliverable to the customer to see if it meets the customer’s requirements. Keep in mind that the login screen may actually be the ONLY thing that’s been implemented in the project so far.
But by getting the customer to review the implementation early on in the project, the agile team is forcing the customer to give early feedback into the project. If the login functionality needs to change, the agile team will immediately schedule another “slice” of time (typically referred to as a “sprint”) to work on getting the functionality right for the customer.
The advantage of delivering smaller portions of functionality, is the ability to make “course corrections” to a project.
My Agile/Scrum Retrospective
Make no mistake that the agile and scrum methodologies are quickly becoming the dominant standard for software development.
Waterfall development won’t completely go away. A lot of government agencies still practice waterfall and there are no doubt holdout companies that are delaying adoption of agile.
But look at any IT job openings these days, and you’ll be hard pressed to find organizations that don’t practice some form of agile/scrum for their software development teams.
Having worked in both traditional waterfall and agile/scrum teams, I think I’ve seen enough of both kinds of methodologies to see pros and cons in each discipline.
If I’ve learned one thing, having worked in the software development field for enough years, I’ve learned there is no “holy grail” of software development.
There is always a new programming language around the corner. A new tech stack that promises programmers a utopia for developing software. A new methodology that promises to deliver 100% high quality software while meeting timelines and staying under budget.
I’ve become intimately familiar with scrum methodology, which is a certain kind of agile methodology.
It primarily consists of a team of the following:
- Product owner
- Scrum master
- Software developers/database admins/dev ops/system admins, etc..
The product owner DEFINES what the development team will work on.
The scrum master is a designated person responsible for making sure a “sprint”, typically 2-3 weeks of time dedicated to working on a particular piece of an overall project, goes smoothly.
The rest of the development team will consist of the technical folks responsible for actually delivering the implementation of a product owner’s desired deliverable (called a “story”).
There are certain scrum ceremonies that happen during scrum development. There are daily standups, typically no longer than 15-30 minutes, where each member of the development team reports three things. What they did the day before, what they’re doing today, and any roadblocks/obstacles getting in the way of their progress.
The scrummaster is responsible for keeping track of everybody’s progress during a sprint and coordinating the daily standup meetings. The scrummaster works closely with the product owner and makes sure the product owner has an accurate picture of the progress of the sprint.
At the end of the sprint, there is a showcasing of whatever deliverable the development team was working on. There is often a separate meeting called a “sprint retrospective” where the team meets and talks about what things went well and what things could use improvement during a sprint.
There are separate meetings referred to as “backlog grooming” and “sprint planning”.
Backlog grooming meetings are where the development team, the scrummaster and the product owner enter future “stories” into a backlog of work that the team will work on in the future.
A sprint planning meeting is a meeting that happens just before a new sprint begins, where the team figures out what exactly will be worked on for a new sprint. This also involves fleshing out the details of each story and the amount of time and effort involved to complete those stories.
The scrummaster is responsible for making sure that the team has committed to the right amount of stories that are feasible to work on and complete, within the timebox of a single sprint period.
The essence of scrum development is a team is always “sprinting” to finish implementing a 2-3 week period of project requirements.
As a designated scrummaster for the past year and a half, I feel I’ve gained enough experience and perspective to come up with my own personal retrospective on what I think of the agile/scrum methodology.
What I like about agile/scrum is the open transparency of our software development. Anyone at the company is encourage to attend our daily standups and sprint showcases where we demo our software deliverables. They are also welcome to see the current progress of a sprint and see who exactly on the team is working on what. There shouldn’t be any surprises to upper management about the progress of development.
Agile and scrum methodology also seems to encourage closer collaboration and teamwork than traditional waterfall development. Lone wolf developers may not appreciate agile methodology, but for developers who prefer working in collaborative environments, agile and scrum certainly encourage that kind of environment.
There are certain aspects of scrum which I think are detrimental to development. There are no “rest periods” between sprints.
When I mean “rest period”, I don’t mean going down to the local pub for a spot of beer. I’m referring to things that developers need to do to keep their skills sharp. Things like researching new technology. New programming languages. New ways of developing software.
It’s just a continuous cycle of sprinting.
Sprinting and doing scrum also requires a large amount of effort by the team trying to estimate how much work goes into each sprint.
The problem I see with this process is software development in general, is ALWAYS full of unknowns. Not enough knowledge about the technology used to implement something. Or inaccurate understanding of a customer’s requirements.
Just because agile/scrum breaks down the phases of an overall project into smaller chunks of work, doesn’t necessarily mean it makes software development any easier.
A software developer is in a constant struggle, fighting with technology and many other factors to get her work done.
Yet the product owner lives and dies by the statistics and reports that the scrum process produces.
In my opinion, software development is just full of too many unknowns to accurately predict how long things will take. No matter how much time and effort you put in, to analyze and make these predictions, you will never attain 100% accuracy.
The other aspect of waterfall methodology which I think could benefit agile/scrum development, is the valuableness of the business analyst or SME (Subject Matter Expert).
Software coders CODE. They are experts in technology.
Yet in agile/scrum, they are responsible for wearing the hat of what the business analyst used to do. They need to make sure they understand the business requirements of the project.
A product owner isn’t necessarily the project customer. A product owner’s job is to ultimately decide whether to sign off on a scrum team’s deliverable. But they not necessarily know the business domain requirements of the project.
In a scrum team, the software engineers are responsible for ensuring they completely understand the business requirements of a project.
This is what the business analyst’s job used to be. Pound for pound, they know more about their business domain than any software developer. Yet the scrum methodology doesn’t seem to place any importance on the role of a business analyst. I think business analysts provided great insight into the business domain of software project requirements.
Better understanding of business requirements = higher chance of providing the right customer project deliverable.
There’s an old saying that nothing lasts forever. Agile and scrum, as popular as they may be nowadays, will surely get replaced by some new methodology.
I hope that whatever ultimately replaces it, will address some of the shortcomings I see with agile and scrum and incorporate positive aspects of waterfall development, such as taking the time and effort to focus on understanding and writing down more formal business requirements of a project.