At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. (Agile Manifesto principle 12)

A cornerstone of agility is the team's commitment to continued reflection and adaptation. For most teams, there is a natural cadence to this process as iterations occur frequently, typically every one to two weeks apart, and releases occur every two to four months. As such, the iteration boundary represents a frequent opportunity for immediate feedback and quick mid-course correction, primarily focused on the team's ability to simultaneously define, code and test new functionality in a time box. At the release boundary, the measures move to those things that reflect the team and the organizations ability to move that functionality from "inventory" to delivered product or system. In this article, we'll describe a set of iteration and release metrics that have been used to good effect by a number of agile teams. In our experience, teams that are effective in using these iteration and release metrics have the best chance to achieve the maximum benefits of agile.

The iteration retrospective provides the first tactical opportunity for assessment and process improvement. Indeed, for most teams new to agile, this is a key learning opportunity and most likely it will indicate the main reasons that the team has been unable to accomplish the objectives of the iteration in the time allowed. After all, team members are new to agile and much is different; it unreasonable to expect that the first few iterations be winners. This quick feedback loop is yet another reason that shorter iterations produce better results. The shorter the iteration length, the quicker the feedback and the more successful the following iterations are likely to be!

Format of the Iteration Retrospective

Iteration retrospectives are time boxed to an hour or less,

Quantitative Assessment

The quantitative assessment section proceeds via gathering and reporting the metrics for the iteration. Table 1 provides a set of metrics initially developed by a project team building wireless infrastructure software, and that we have later successfully applied in a number of diverse software project contexts. Each row in the table indicates a measure the team determined that was important to their process. The columns measure their progress over time.

Table 1: Sample iteration assessment metrics

By reviewing these metrics, teams can gain a number of real insights into of the types of measurements that are important to this agile team.

Functionality/Story Achievement

The first eight rows all measure the same thing: how many stories (which could be any of user stories, use cases, use case scenarios, jobs, defect suites, etc) did the team complete compared to plan?  It is not unusual to see this vary from 30-40% to as much as 100%, with a range of 50-80% being typical.[1] In addition, there is an interesting "root cause analysis" implied by the other rows in this section which answers the question: "for the stories that we did not complete, what happened and where did they go?"(rescheduled, deferred, deleted?)

Quality and Other Key Process Indicators

The second section of this table provides additional insights. In this section, this team appears to be tracking four important things:

• Ability to test new stories within the iteration. The team is measuring its ability to test each new story within the iteration boundary, a key aspect of agile progress.
• Defects. Yes, defects still exist with agile, the difference being that the numbers are far smaller and they are addressed on a continuous basis, rather than via an all-hands-on-deck-lumpy-triage towards the end of the release cycle.
• Progress on test automation. Clearly, this team understands that its ability to keep up with test automation is a gating factor in agility and it is tracking the rate of progress at which it is reducing the backlog of manual tests
• Unit test coverage. Another key indicator, or better a key forecaster of ultimate quality, is percentage of code covered by unit tests of one sort or the other. For most teams, 60-70% unit test coverage is goodness.

These iteration metrics, along with their trends over time will, provide an ongoing indicator of the team's real progress.

Qualitative Assessment

Once the quantitative assessment is done, the next portion of the meeting is conducted typically by answering and answering two key questions of the team.

1. What did we do well on this iteration?
2. What did not go so well on this iteration?

Table 2 below provides an example of some of the types of feedback teams provide on these two key questions.

 Table 2: Examples of qualitative team feedback on an iteration

Call for Action

The last portion of the iteration retrospective is the key. The agile team leader or ScrumMaster leads the team through an exercise:

"Given the above data, what could we do better for the next time?"

This short brainstorming process will likely produce a long list of possible corrective actions. (In one 30 minute session, a new agile team identified 25 items for improvement!) In the face of such a wealth of opportunity, it can be hard for the team lead to avoid having the team "boil the ocean." Instead, the team lead must remind the team that it has a full load of stories ahead, and it had better pick just one or two small things to improve for next time.

Using this model, at first the team may be disappointed that not all identified issues can be immediately addressed - it's the engineer in the team members, after all. But since iterations are short, the team will soon learn that, over time, all the issues will indeed be addressed (albeit to be replaced by a new set of issues at the next level of refinement).

Release Retrospective

The release boundary is the next logical time for a retrospective, and it takes a different appearance as it is likely that it will involve a larger number of significant stakeholders who live outside the team. However, a release retrospective still has both quantitative and qualitative aspects.

Quantitative Assessment

Table 3 illustrates a set of quantitative measurements one team applies to the release retrospective.

Table 3: Sample release retrospective quantitative measures.

This team measures their release progress along three axes:

1. Value delivery. Rows one and two measure the value delivered to the customer in "feature points," a relative, quantitative measure of the value each feature delivers as assessed by the product owner. In addition, rows seven and eight measure the team's progress in working off the existing backlog of "feature debt," which represents existing customers commitments.
2. Conformance to release date. This team is also measuring its ability to meet the committed release dates.
3. Refactors and architectural debt. Software systems tend to become brittle over time. Teams must constantly refactor to increase quality and provide "architectural runway" for new features and stories. While these do not contribute directly to value delivery (as they may have no near term customer impact) they are important nonetheless, and teams should measure them to encourage keeping the software current and resilient.

Quantitative Assessment

As with the iteration assessment, a qualitative assessment of the release is also warranted. This can also occur with the simple "what went well?" and "what didn't go well?" method and brainstorming for corrective actions. At the release level, however, the issues are likely to be different. Given sufficient time, the iteration retrospectives will tend to improve those aspects of team performance that they can directly control, but the release boundaries often yield a different set of issues.

Surfacing Organizational Impediments With The Release Retrospective

The nature of agile, with its constant demand for improvements in productivity, quality and time will also expose a host of issues that live outside the team's control. These organizational impediments must be raised to the next level and teams' sponsors and advocates must often be the ones to take corrective action. For example, the list of "what didn't go well below" in the table below from one team highlights a host of such impediments.

Table 4: Example of "what didn't go well in this release"

It is clear from this list that this particular enterprise has a ways to go to unleash true enterprise agility!

Taken together, frequent iteration assessment metrics, and the less frequent release assessment metrics, help the agile meet improve delivered productivity and quality at every iteration and release boundary. In so doing, the team is well on its way to achieving its real goal - that of continuously improving its ability to delivery quality software to the end users, and to be able to do so as rapidly as possible.

[1] We note that teams that operate at either end of this spectrum may need substantial improvement. The low end is self explanatory. But at the high end, which looks like excellence incarnate, additional tuning may be required. For if a team truly hits 100% of stories on every iteration, it is likely taking on too little work or deferring technical risk-laden stories until a later time. Effective agile teams typically run between 60-90% story acceptance, and even that will vary significantly from iteration to iteration. In other words, be careful what you measure.

Comments (0)

Write comment

Name

Comment

Write the displayed characters

Add Comment

Tags:

Click to add your tags...,

or Close