This paper outlines an adaptation of Agile development based on a pilot project at the Educational Credit Management Corporation (ECMC), a mid-sized non-profit corporation in the student loan guarantor industry. To support the re-engineering of mission-critical loan management applications, ECMC needed to modify Agile development methods to incorporate up-front project estimates to gain funding approval. IT also needed to redefine the traditional working relationship with its business partners to eliminate the traditional tension over project scope. What resulted is a Hybrid-Agile solution that incorporates the flexibility of iterative development with traditional project estimation techniques, while accommodating the end user's need for on-going scope changes.
Hybrid-Agile Estimating Solution
IT departments typically utilize a ‘system methodology' consisting of several phases to develop custom applications. The traditional methodologies call for a ‘requirements' phase, followed by a ‘design' phase, then a ‘build' phase, and finally an ‘implementation' phase. The traditional phased methodologies have been challenged in recent years by the emergence of the ‘Agile' system methodologies. With Agile development, a project is divided into standalone ‘releases,' each with its own requirements, design, build, and test activities.[i]
To combine iterative development with traditional estimation techniques, ECMC developed a three level estimation process for requirements.
Advertisement
Level 1 requirements represent an organization's best estimate for the system functionality up-front. This is traditionally what IT organizations have done to estimate projects for 30 or 40 years. Let's say that using traditional estimating techniques, a project is estimated to cost $4 million and take 16 months to complete. Of that $4 million, $1 million is for new hardware and $3 million is for software development, which represents the Level 1 estimate. Using the Hybrid-Agile approach, an additional 30% of the Level 1 software estimate would be added to cover ‘Level 2' requirements. The Level 2 requirements are all the changes requested by the end user following the acceptance testing of the initial monthly iterations. This is all the feedback that the IT developers typically do not count on, things like "move this column over two inches," "we forget an important field," or "I need the edits to check these additional 5 conditions." These are either refinements to the stated Level 1 estimates, or additions to the requirements based on changed business needs.
In the case of the example project, the Level 2 estimate would be sized at 30% of $3 million, or $900,000. The Level 2 estimate represents the budget that IT and the end users have for addressing all the shortcomings that result from acceptance testing of the development iterations. From a schedule standpoint, let's say that the Level 2 development process adds another four months to the total project duration.
Creating the Level 2 estimate constitutes formal recognition that specifying requirements is an iterative process, and that changes and refinements are expected during the course of the development process. By accounting for this need, IT and the end users no longer need to fight over whether a change should be accepted. Of course it should be accepted, it is part of refining the project to meet the needs of the business.
Once all of the Agile development iterations are ‘complete,' including turnaround of the Level 2 requirements, the Level 3 requirements represent those final ‘must have' changes that come out of comprehensive end-to- acceptance testing of the final integrated product. Add 10% of the Level 1 software estimate to cover Level 3 requirements. In the case of our example project, the Level 1 estimate was $3 million, so the Level 3 estimate would be $300,000. Let's also add an additional two months to the project schedule to account for Level 3 development.
The process of defining level 1, 2 and 3 requirements is consistent with the DSDM ‘MoSCoW' prioritization technique.[ii] Craddock explains that MoSCoW refers to the requirements that Must, Should, Could, and Won't be met as part of the timeboxed development iterations. The MoSCoW technique is used initially to scope out the project, and then again during each timeboxed iteration. The Level 1 estimates would fit the initial MoSCoW exercise to establish the overall scope of the project. The Level 2 estimates would be applied to the timeboxed iterations. The Level 3 estimates are applied at the completion of the final iteration before the system is moved to production.
Returning to the example project, adding the Level 1, 2, and 3 requirements would give us a total project estimate of $5.2 million, as opposed to the original $4 million estimate. The project duration is 22 months instead of 16 months. The $5.2 million estimate needs to be subjected to a rate of return analysis to determine if there is still a financial justification for the project. If there is, proceed with the project. If not, do not be tempted to advance the $4 million project, because it will invariably exceed the budget and schedule.
It is important to note that the Level 2 and 3 estimates represent hard budget constraints that both IT and the end user must manage to. They replace the open-ended, never ending additions to scope that are a part of pure Scrum development. The Level 2 and 3 estimates are intended to address the issue of defining the entire scope of a project up front, while allowing flexibility for changing requirements during a project. While changes are permitted, they are capped by the Level 2 and 3 budget. This provides an incentive for IT and the end users to manage change in a responsible way during the course of the project using MoSCoW prioritization.
To some, the Level 2 and 3 budget provisions may seems similar to the former practice of adding a contingency to the total project estimate. I would argue, however, that it serves the purpose of formally legitimizing the need to evolve business requirements over time. This is different than applying a generic contingency. For one, decision makers often challenge the need for a contingency, arguing that it is no more than padding by the IT Department. The contingency is often reduced or eliminated to make the project financials look better. By contrast, it is not an option to reduce or eliminate Level 2 and 3 estimates; they are a fundamental part of the Hybrid-Agile development methodology. The true value of the development process often comes from the interchange between developers and users when reviewing the results of iteration testing. The incremental changes from this interchange are just as important as the initial process of documenting the requirements.
Obviously, in addition to accounting for a 40% increase in development costs, the project schedule needs to reflect the additional time needed to process the Level 2 and Level 3 requirements.
Hybrid-Agile Development Process
The proposed Hybrid-Agile model is illustrated in Figure 1. The sum of known requirements
Figure 1: Hybrid Agile Development Model
are depicted at the bottom and form the basis for the Level 1 project estimate. Using the MoSCoW prioritization technique, timeboxed iterations are defined with monthly code delivery. The results of the user acceptance process in Iteration 1 has resulted in Level 2 requirement changes. Using the MoSCoW technique for Iteration 2, the Level 2 requirements from Iteration 1 are either folded into Iteration 2 or added to the requirements backlog. This iterative process continues until all of the scheduled iterations are complete. At that point, a final iteration is performed to accommodate the Level 3 requirements.
Actual estimation of work effort to code the development iterations is an evolving art at ECMC. Steindl and Krogdahl refer to the amount of work that can be accomplished in an iteration as "team velocity."[iii] Velocity can be represented by the work days of code delivered during monthly iterations, as illustrated in the ‘Iteration Burndown Chart' in Figure 2.[iv]
Figure 2: Burndown Chart
This is the actual chart used for the ECMC pilot project in March 2007. Clearly, the monthly velocity is subject of some variation. The bars through February represent completed work looking back from the current month. The current month shows a combination of completed work and work in progress. The future months represent planned iterations leading to system completion in June 2007.
Changes in Requirements and Testing
Agile methods discourage the traditional detailed requirement documents in favor of higher level descriptions of business flows, called "User Stories" or "Use Cases."[v] The idea of producing less than exacting requirement specifications can be a cultural shock to an organization that has lived by rigorous and detailed requirement documents. But Agile philosophy states that since requirements can not be fully known up front, teams should spend time instead developing the requirements as part of the iteration process. Scott Ambler sums up his experience as follows:
"My use cases need to be just good enough. Why does this work? Because in an agile environment you'll ...discover that you don't fully understand what is required, you'll work with your stakeholder to do so, and you'll build something that meets their actual needs. It's software development, not documentation development."
Steindl and Krogdahl advise that the details that flow from high level user stories should be captured not in documents, but in automated acceptance tests. ECMC has implemented automated regression tools, however, we have a long way to go in our evolution before we actually substitute detailed requirement documents with detailed acceptance tests.
Automated testing tools take on added significance in Agile development for two reasons. First, testing is continuous with each iteration, up to and including ‘final' acceptance testing. Second, since iterations build on each other, automated regression testing is essential for effective time management. Alistair Cockburn states, "With automated regression unit and acceptance tests, the developers can revise the code base and retest the entire system at the push of a button. Teams who have such tests report that they freely replace and improve awkward modules."[vi]
Challenges for the Business Users
The methodology changes inherent in Agile development impact the end users at a far greater level than the IT staff. Users are impacted in two significant ways: time commitment and fundamental role changes. Most significantly, users are asked to be a part of the project team on a daily basis for the duration of the project. This is a fundamental change from the traditional phased approach in which users define requirements and then wait for delivery of the completed system before engaging in acceptance testing. With Agile development, users are engaged in each iteration on a daily basis. ECMC has found that we need to find ways to backfill behind key end users to free up their time to become full-time members of the development team. A fundamental role change for the end users will be to invest less time in requirements documentation and more time in automated acceptance test design. This change will evolve slowly at ECMC as we gain experience with automated testing tools.
ECMC currently faces a very practical issue which severely constrains our ability to produce production-ready iterations on a monthly basis. In most organizations the business users work with business analysts to define requirements. In our case, the same business users who define requirements are also called upon to perform functional user acceptance testing. For any given Agile iteration, the same end users are expected to conduct testing while working on requirements for the next iteration. As a result, some monthly iterations have been scheduled as internal only for IT and are combined for hand-off to the business partners during periods of less intensive requirement gathering. This situation will hopefully resolve itself as ECMC gains more experience with automated acceptance testing tools.
Conclusion
ECMC has discovered that Agile development methods using iterative software releases improve the quality and timeliness of business applications. In the case of the pilot project, 20,000 hours of functional enhancements were developed in 18 ‘time-boxed' monthly iterations. All 18 iterations were delivered to the end users, keeping to the monthly release schedule, for acceptance testing. To date, over 500 software defects have been fixed by the development team as part of the on-going monthly release process. The application is currently in test against a copy of production data with over 3.5 million records. Processing speed is averaging 2 seconds or less per screen. The foundation for effective software development is indeed a trusting environment between IT and business users with daily interaction to discuss functional priorities and design considerations.
Some of the more extreme forms of Agile development advocate the use of ongoing development iterations with no defined overall project scope. ECMC, however, has adopted a Hybrid-Agile process that permits the use of monthly development iterations within the scope of overall budget and schedule constraints.
To be completely effective, ECMC must fully implement automated regression testing procedures for unit and acceptance testing. Ronald Jeffries provides sound advice on the role of "extreme testing" when he writes:
"To be sure that new features work, write Unit Tests for every feature. Save all the unit tests for the whole system.
To be sure that nothing else is broken, run all the Unit Tests in the entire system before any code is released - and ensure those tests run at 100 percent.
To get confidence from (acceptance) tests, the customers must understand them, and should provide the values tested.
You cannot have comprehensive repeatable tests if you have to manually check the results. Have a testing facility to set up and run the tests, check the results, and report them."[vii]
Finally, the end user-IT working relationship is fundamentally changed by Agile development, for the better. Agile methods require daily interaction as part of a collaborative team approach. The flexibility to accept new and changing requirements dissolves the historic tension between IT and the functional users. The focus of the project team truly becomes the production of superior code. ECMC has made considerable inroads to adopting Agile development. We still have much to learn, but we are well on our way.
About the author
John Elie is Sr. Vice President of Information Technology for the Educational Credit Management Corporation (ECMC), a student loan corporation based in St. Paul, MN. John has 25 years of experience in IT management. In his current position, he is overseeing the migration of ECMC's mission critical student loan processing applications to a web-based Java/J2EE environment. As part of this migration, the development team at ECMC adopted Agile development methods, and John has recommended modifications to the methodology to account for corporate governance and project estimating needs.
[i] Aguanno provides a good overview of traditional development and Agile development. Aguanno, Kevin, (2004). Managing Agile Projects, (1st ed.). Lakefield, Ontario: Multi-Media
Publications Inc.
[ii] Craddock, Andrew, (2007). "DSDM and Scrum: FAQ's - The similarities, differences and potential inter-operability issues." Retrieved March 14, 2007.
[iii] Steindl, Christoph, and Krogdahl, Pal, (2005). "Estimation in Agile Projects." IBM Global Services. IBM Academy of Technology Best Practices in Project Estimation Conference.
[iv] Hansen, Steve (2007, March). "Iteration Burndown Chart," EPIC Monthly Status Report, Educational Credit Management Corporation.
Eye On the Prize 
