Agile emerged in the late ’90s and took off in the early 2000s. Typical Agile projects were web-based since the DOT.COM era was booming and companies were trying to erect scaffolding for online transactions and interactions using an ASAP model.
Web-based interfaces could be set up in a few days. Databases were used to just store data, so development teams had unencumbered access to databases. Projects that took 3-6 months were broken into smaller chunks (sprints) to deliver faster capabilities (value), which was visible since these projects were mostly online.
Projects scope could easily be modified since the interface and experience were not significantly impacted and there was virtually no impact on business operations. Feature additions/changes were common and did not impact other systems (esp. Tier 1 or Tier 2). The concept of an “MVP” (minimal viable product) evolved since the first few sprints revealed sufficient functionality to provide useful value, so any further development was tabled for the future or put on hold.
Agile started out as a manifesto and evolved into a formal methodology, where project managers were elevated to scrum masters and worked with teams to capture “stories” about what the customer was trying to accomplish. Stories replaced formal requirements analysis (which relied on UML) to create an emerging set of features with successive levels of detail to guide development and implementation.
Business users jumped on the speed and productivity of Agile and investors (internal stakeholders) found the improved communications and tracking useful for managing risk and guiding project funding. If projects started tracking in the wrong direction, funds could be held up until the stoplight (red, yellow, green) returns to normal.
So, why isn’t Agile right for everything? The devil is in the details.
Project management is based on tasks that are clear and measurable to guide estimates and manage expectations for commitments. Setting up repeatable processes in operational areas is fairly consistent from an estimation standpoint. Establishing timelines and work estimates for software development and integration (which is becoming increasingly complex) are much more difficult since these are not predictable. So, IT turned to COTS to reduce the risk and leverage functionality that already exists. These projects require IT to pay higher upfront costs for licensing with the goal of reducing timelines to deliver and reduced time to extend functionality using configuration instead of customization.
But, applying Agile to COTS takes time to evolve. IT staff have to be intimate with the package so they can assess the risk and effort to configure and adjust any integration points. This works best when the integration is loosely coupled to reduce risk to other systems. Otherwise, it can get tricky. COTS support/upgrades can get complex and performance is not always linear, which can impact capacity planning for technical support. Again...Agile may not be easy to apply.
Speaking of scale, teams have to be careful trying to roll Agile out for large projects. Scaled Agile requires a different management and planning paradigm than simple sprints with a scrum master. Planning and design can consume significant resources to coordinate all the work across multiple teams working on one project. When bringing aboard other projects using Agile which have dependencies with each other, the level of planning and complexity (esp. if systems were not designed and implemented with a high degree of insulation and resiliency) is no longer linear so release management can literally bring all that productivity to a halt in order to protect (insulate) production operations.
Another constraint of Scaled Agile is the resource demands on business SMEs who need to play a key role throughout the lifecycle of Agile projects. Scrum meetings are supposed to last 15 min, but if problems occur or priorities change, these meetings can take much longer. All too often, business SMEs grow weary of the constant meetings and impact on their own productivity. This leaves IT teams to manage and scope MVP on their own, which can lead to gaps in expectations with the business community.
There are certain types of projects where waterfall should be used over Agile. Projects involving enterprise data, shared services, changes to core infrastructure or operations, complex integrations, and any projects involving compliance need to be carefully planned and executed.
More work to be done for planning and implementation around the use of Agile and how it can be scaled for concurrent development and release planning.