Are we
Agile? If we ask a leader we'll get one
perspective, but if we ask each person on the team we may be surprised by the
variety of answers. Since 2002, IBM has used
agile evaluation frameworks with dozens of teams to help them learn, improve,
and share their experiences with agile practices. The metrics in the Extreme Programming
Evaluation Framework, or (XP:EF) originally focused on XP, and similar
instances covered other methods.[i] But because the framework can be used with a
broader set of practices, it's simpler to say "the Agile Evaluation Framework"
(Agile:EF). This article shares tips on using the framework in a lightweight
manner to leverage "the simplest metrics that could possibly work."
Review of the Technique
Agile:EF contains
four parts. Context factors, indicators, outcome measures, and a survey. The metrics in the first three parts describe
what a team did. This article focuses on
the last part - the survey, which can serve many purposes:
Stimulate team discussion to
seek improvements.
Spot weak areas in the team's
process, in time to address problems before bugs result.
Find out what the quiet
people in the team think.
Describe to other teams how
much of which practices were used.
Teach and remind team members
of the practices.
Agile Pulse
Think of
this survey as an "Agile Pulse". By
taking it periodically you can monitor the health of a team's agile adoption. This can be done as part of your iteration
retrospectives, or every two to four weeks, perhaps with some variation in
questions or team members to avoid burnout. The list of questions should be short (about
15). The Agile Pulse survey should be
written so that it takes 15 minutes or less to complete. That will make it practical to take it
periodically. It should include a
spectrum of practices, not just the ones the team does. That reminds teams to consider using those
practices. If they share their
experience, it also helps other teams know which practices were not used.
An Example
Here is example
data from a team who has used the Agile Pulse survey.. Each question has a short label for the
practice, such as ‘Non solo', which means any style of code review, or
Iterative, which means ‘stakeholder feedback on working software.' The questions in the survey provide a little
explanation of each practice to introduce them or remind team members.
Figure 1
shows the average of how much each team member feels he used a practice. The scale is 1 through 10. One means a person feels their team never
uses this practice. Ten means they
always use it. Five indicates a partial implementation. An example question would be "Unit
Test: Do developers frequently run
automated unit tests?"
Figure 1:
Sample survey results
The thin
black bars illustrate the range between team members. It's the average plus or minus one standard
deviation, so 70% of our scores will fall in that range. This team is beginning its agile journey, so
we don't expect high numbers.
We use
the chart to trigger discussion. The
average for "Automated Unit Test" looks low. Is the team comfortable with that,
or is that something team members would like to address? Why is there such a wide range of responses
for "Reflections"? The team should
discuss this. Maybe some felt that
reflections were not held often enough.
Maybe some felt left out. Perhaps
some felt no actions resulted from the improvements identified. The chart can help spark discussion, and give
quiet people a louder voice.
Successfully using the Framework
Now we'll
share some observations from experience using Agile:EF that will spare you some
pain.
Advertisement
Focus time on
actions related to notable scores.
Quickly
celebrate high scores. Ignore
unremarkable scores. Focus a little time
on averages below 4, and standard deviations above 2.5 (an indicator of
divergent opinions among the team). Also,
discuss averages that have dropped since the last survey. Focus the bulk of your team's discussion time
on identifying a couple of improvements so you can make retrospective meetings
more efficient. People often love to
talk about their job. The passion for
their craft is good, but it can lead to some meandering meetings. Point them to a few numbers highlighted by
the survey to be discussed.
We've
learned the hard way that if you generate a long list of improvements they may
not be implemented. Just pick two and
nail them. Either the other problems
will still be there and can be worked in your next iteration, or they may no
longer be a problem. Pick two
actions. Implement them in two months
(sometimes change takes a while). But
reflect on progress every two weeks.
The
technique of discussing the questions with a wide range of responses from the
team is like planning poker described by Mike Cohn, with the exception that
you're not looking to form a consensus.[ii] You want to highlight the differences of
opinion to spark a team discussion that will lead to an improvement.
Keep it simple
Earlier
versions of the survey contained 30 questions. We were wrong. As process
consultants, we get excited about nuances of practices and methodologies, and
can talk about it for hours. But our
busy engineers have less appetite for process.
So we reduced this list to 15 questions. Keeping the Agile Pulse survey
to 15 minutes leaves time for discussion and action. Without action, discussion is waste.
XP:EF started
with questions based on XP, and then similar frameworks arose for processes
like the Rational Unified Process (RUP).
Now Scrum, OpenUp and Lean are interesting. So the Agile:EF and it's AgilePulse focus on
practices instead of an entire process. It's also key to keep questions at a high
enough level. So instead of asking if
the team does inspections, committer review, or pair programming, we ask ‘do
you review your code with the technique of your choice?'. The team decides how. We call this "Non Solo" - don't go out there
alone.
We found
people didn't want an entire methodology handed to them. They wanted to pick and choose
practices. Using a cohesive proven set
of complimentary practices is better, but giving the teams some choice in which
practices they adopt first creates buy-in.
This accommodates both teams who want to evolve into a methodology and teams
who start with a methodology and see which practices are moving slowly.
It's not a scorecard
If the
Agile Pulse survey is used as a ‘scorecard' for executives to judge teams, all
the numbers will magically improve and will disappear as soon as teams figure
out how to avoid collecting them. Use Agile:EF
for in-process improvement. The team
owns the data. The team owns its process.
On the
other hand, we hope teams will voluntarily share their data in the form of a
short experience report. The thought
behind writing one benefits the team, but is a larger benefit to the
organization. Many teams feel their
environment is a bit different. Reading
experiences from a similar team is a gold mine for the organization. Don't just share good news. We call them experience reports instead of
success stories because the pitfalls are even more interesting.
Share
The
consistent format offered by Agile:EF helps teams share experience reports across
our industry. It's frustrating to read a
great paper on agile, only to wonder if they were using a particular practice,
or if they were co-located or globally distributed. With Agile:EFs survey and context factors, coaches
can see that a team reporting that a practice failed may have not done a
thorough job with a related practice.
For example, a team might feel iterative development didn't work, but
their Agile Pulse may show they didn't do a supporting practice like automated
unit testing.
Different
audiences will want different amounts of formality and detail. Agile:EF serves them both. Busy teams can show their experience on one
slide. They can show their big picture
practice bar chart, a few context factors such as team size, audit
requirements, and team distribution, and finally outcome with both benefits and
pitfalls. Formal studies can add metrics
into the framework and write a rigorous academic study.
When
discussing survey scores, note that low numbers are not always bad. Some teams consciously chose to not adopt a
given practice. They own their process. The goal is for them to thinking about what's
best for the team. If they share an
experience report, it will be interested to read their experience report to see
if the practice was indeed unnecessary, or if they wish they had used it!
Next Steps
Figure 2
provides some examples to begin using Agile:EF. The scale is one to ten. Ten means you use the practice all the time,
five is 50 percent of the time, and one is never.
Figure 2:
Example questions for using Agile:EF
Where are
the other questions? The purpose of the
framework is not to dictate the ultimate list of questions. Coaches have the power to write their own
questions related to their favorite set of practices for their teams. They don't have to start from scratch. Use the Agile Pulse questions above as a
start. But for groups that want to
measure in great detail, XP:EF 1.4 is an instance of Agile:EF that specifies
many detailed metrics.[iii] Some teams like the formality and
completeness, but other teams prefer ultra light metrics. Agile:EF is a framework that accommodates
both; the team decides.
In
summary, use Agile:EF's Agile Pulse survey for lightweight retrospectives. Add Agile:EF's context, objective metrics, and outcome structure to
share your experience.
About the Authors
Bill Krebs
has worked as a developer, performance engineer, and process consultant since
1982. He's been agile since 2001. In addition to coaching teams, he teaches
agile to over 500 students a year and trains other teachers. He's presented at agile conferences, IBM
Research, and serves as the co-chair for the IBM Academy of Technology
conference on Agile. Bill is a certified
ScrumMaster TM and continues to learn from every agile user group he
can find.
Per Kroll
is development manager for RUP ® and the IBM Rational Method Composer, and the
project leader on the Eclipse Process Framework Project, an open source project
centered on software practices. Per is author of the books The Rational
Unified Process Made Easy - A Practitioner's Guide, Kroll and Kruchten, and
Agility and Discipline Made Easy - Practices from OpenUP and RUP, Kroll and MacIsaac. Per is a frequent
speaker at conferences and author of numerous articles on software engineering.
[i] "Toward an XP Evaluation Framework"
by Laurie Williams, William Krebs, Lucas Layman, and Annie Antón.
Available at
Are we
Agile? If we ask a leader we'll get one
perspective, but if we ask each person on the team we may be surprised by the
variety of answers. Since 2002, IBM has used
agile evaluation frameworks with dozens of teams to help them learn, improve,
and share their experiences with agile practices. The metrics in the Extreme Programming
Evaluation Framework, or (XP:EF) originally focused on XP, and similar
instances covered other methods.
