|
The Importance of Iteration Acceptance Metrics
One of the fundamental goals of a software team is to have a stable and predictable development process. Stable and predictable means better expectation management and fewer surprises. The percentage of stories accepted within an iteration is an excellent metric to gain this stability and predictability. We define an accepted story as one that has been successfully completed and tested, is defect free, and has been approved by the product owner. When a team consistently accepts a high percentage of the stories in an iteration, it usually means that the team estimates well, the requirements are clear, the test cases are well designed, most tests are automated, and the team is working together. As part of its Agile Lifecycle Management (ALM) solution, Rally provides graphical reports that help customers track the historical acceptance percentage of stories in each iteration as well as incorporate statistical process control (e.g. Six Sigma) stability analysis. This short case study will explain why we believe that the iteration acceptance percentage should be the first high-level metric used for fostering a culture of continuous improvement.
Acceptance Percentage as a Key Agile Metric
First, let me give a couple definitions. Work is defined as the sum of a point measure assigned to each story. Stories can have points assigned to them so that the relative size of stories can be taken into account. We define acceptance percentage as the amount of work accepted within an iteration divided by the amount of work attempted in an iteration.[1] Over time, this high-level metric can be used to guide Agile teams during their quest for continuous improvement. When this percentage is high, it normally means that the team has its Agile process under control. When this percentage is low, it can be an indicator of many deficiencies within an Agile process. While acceptance percentage does not contain enough information to diagnose the problem, it at least lets the team know that a problem exists. Table 1 provides a few possible symptoms and solutions to a low acceptance percentage.
|
A Low Acceptance Percentage May Mean:
|
Possible Solution:
|
|
The Stories (requirements) are not defined with enough detail or were not thought out well enough.
|
Define the requirements with a proper mix of Stories and Test Cases.
|
|
The team did not properly estimate the amount of work outlined by the Stories and Tests.
|
Observe past performance and modify the expected velocity accordingly.
|
|
The team cannot test (and thus accept) Stories within an iteration.
|
Create more automated tests so that they can more easily be executed within the iteration.
|
|
The team cannot accept Stories within an iteration without additional participation.
|
Make certain the Product Owner is available to accept Stories.
|
|
The team has overcommitted.
|
Make certain that the team, not management, commits to what is in the iteration.
|
|
The team did not have a good understanding of what ‘done' (or accepted) meant.
|
Define the acceptance criteria before the iteration begins.
|
Table 1: Symptoms and possible solutions for low acceptance percentages.
Case Study
The following case study is from a mid-sized Agile team that has provided a subset of its data and anecdotes about its development process. The team consists of about a dozen co-located employees including development, QA, and product management staff. The company is an ISV that develops a Web-based product.
Figure 1 shows iteration acceptance percentage on the y-axis and iterations on the x-axis. For ease of reading, the background is colored for good, fair, and poor acceptance percentages. As you can see, there was an initial period of low acceptance. According to the team, there was no real focus on accepting work within an iteration although a fair percentage of the work would be code complete within an iteration but not fully tested nor accepted. The team was essentially running the test process one-half iteration later than the development process. For iteration 10, more focus was placed on accepting work within an iteration by:
- Better scope management
- More automated tests
- Defining tests early in the iteration
- More focus on story definition (thinking through the requirements)
Figure 1 Iteration Acceptance Percentage
The results show improvement, but they are still not within what we would expect to be a ‘good' acceptance percentage. Some of the reasons include:
- Loss of focus on automated testing because of the loss of an employee (and a longer than expected time to replace that employee)
- Requirements that were not well thought out and thus led the team down paths of increased scope
The team is now expecting another step change as it moves towards more test automation and test-driven development. This should help them accept an even higher percentage of work within the iteration and Rally's tools will help them track their progress.
In addition to the overall acceptance percentage charts, Rally provides integrated statistical process control metrics including percentage control charts, average and range control charts, and probability plots. These charts allow teams to make a more in-depth analysis of the development process stability. Figure 2 shows a percentage control chart (or ‘p chart') for the same timeframe as that of Figure 1. A percentage control chart is useful for evaluating the stability of a process in terms of the percentage attainment of a goal.
In Figure 2, the dotted/dashed bands represent deviations (sigmas) from the average with the upper and lower control limits representing +/- 3 sigmas, respectively. The acceptance percentage is shown in dark blue while the average is shown in light blue. A process that generally stays within +/- 2 sigmas is considered stable. Rally also provides the ability to define an inflection point where a process change occurred to help illuminate the effect of the change.
Figure 2 Iteration Acceptance Stability
You can see in the first nine iterations that the process varied wildly, often outside the +/-3 sigma bands. This process was certainly not being well controlled. During iteration 10, as explained above, the team enacted a set of process changes and the results are clear. While the acceptance percentage is not as high as hoped, the process is certainly more under control falling mostly within +/- 2 sigmas.
Conclusion
The final principle of the Agile Manifesto[2] states that "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly." In many traditional software processes, these feedback loops can take years to figure out. With an Agile process, you can measure and improve performance inside a calendar quarter. Iteration acceptance percentage is an excellent first metric for an agile team to help a team tune its behavior. Rally's Agile Lifecycle Management application can help track this metric so that teams can make certain to estimate well, elaborate clear stories, design appropriate test cases, automate testing, and work well together.
[1] Some Agile teams fill the iteration before it starts; others don't fill it much at all and only add stories as other stories are accepted. In the end, this creates a similar outcome, but slightly different metrics. The former will show less than 100% acceptance; the latter will show much closer to 100% acceptance. The Rally metrics are optimized for the former case.
[2] See http://agilemanifesto.org/
About the Author
Mark Ringer is currently Program Manager at Rally Software Development, a provider on-demand Agile Software Life Cycle Management solutions. Mark previously worked as a Practice Manager at BEA Systems. Previously, he had a career as a research scientist at the NASA Glenn Research Center and Honeywell Technology Center. Mark holds a Masters degree in Computer Science from Case Western Reserve University and an undergraduate degree in Aerospace Engineering from The Ohio State University.
| 
