Using Agile metrics to manage projects and
strengthen organizational transformation.
Traditional program management offices (PMOs) are responsible
for providing checks and balances to the development and IT organizations regarding
budget and schedule. Oversight and
management that comes from the PMO drives certain behaviors in the project
managers (PMs) and therefore in the project staff. Similarly, the Agile PMO provides certain
checks and balance, but principally focuses on the holistic well-being of the
project. This difference in 'tone' not
only drives different behaviors, but also can help to support dramatic
transformation within the organization.
The driving force becomes encapsulated in the difference in perspective
between traditional earned value reporting and Agile's achieved value
reporting.
Devolution
of transformations
The collection of documented Agile
success stories are swelling, driving the process into the mainstream of
software development. [i]
However, in the past several years, we have also begun to see a phenomenon
whereby successful transformations begin to devolve fairly rapidly, to the
point where projects begin to slide and even fail. Many companies have even begun contracting
in for services to re-enable Agile transformation, trying to recapture their
early successes and perhaps making the transformation stickier the second time
around.
So, are all Agile transformations
unstable? I think not. What I think is more likely is that there may be active
resistance to change acting as the outside force.
This situation brings to mind the
old Newtonian adage: "Things in motion tend to stay in motion unless acted upon
by an outside force."
Typically, a process change is
typically not deemed successful until there is consensus amongst the team
members and positive results have been shown.
This suggests that active resistance would have been weeded out prior to
declaring the methodological shift successful.
The alternative might be a passive resistance, or friction force that is slowing down or stopping the
transformation. If we accept this, then
in order to keep the transformation moving, we need to find something that can
counter this resistance and encourage the teams to continue.
Consider, do the Agile projects in
your organization get positive reinforcement, negative reinforcement, or
neither? And, what form does this
reinforcement take, if it exists? In
most development and IT organizations, there is a responsibility that belongs
to the PMO to provide checks and balances on each project. It offers an independent review of budget and
schedule for senior management as well as a periodic critique for the project managers.
It is this critique that
serves as possible reinforcement for the transformation.
What is the
tonal difference?
There is what I would call a tonal difference between the
critique from a traditional PMO and an Agile one. Traditional PMOs focus on variation from plan
to date and quantify this as an earned
value, the implication being that by merely making progress against the plan,
the project is earning value for the company.
Conversely, an Agile PMOs focus is on variation from commitment and
quantifies the amount of value being released to the business. These are actually two very different
measures - one allows the team to see how it is doing against its expectations
and the other allows it to understand how it is doing against the business'
expectations. The later is quantified as
an achieved value.
Advertisement
The tonal difference is established as a project-centric view
for the traditional approach versus product-centric for the Agile
approach. Project-centric refers to the act of completing the plan for the
product and product-centric obviously
refers to the product itself. To better
understand this difference we need to ensure we understand the motivation
behind the actions and decisions each PMO takes, so we will look at some of the
specific traits of each traditional and Agile PMOs.
Traits of a
traditional PMO
In companies that use traditional
‘plan-driven' methodologies to execute their projects there is usually a PMO to
provide a source of check and balance.
In these organizations, the PMO becomes the most influential force in
the organization with respect to both the execution of programs and the
approach by which they are undertaken.
In the classical sense of you-are-what-you-measure, when the PMO
requests that the PMs come to the review prepared to discuss budget, progress,
and work performed to date - the essential variables for Earned Value
calculations - PMs begin to focus their work and their teams on these
measures.
This course of measurement only
represents lagging indicators,
telling the team how well it has done; not leading
indicators, suggesting where it will end up. Ultimately, the result of Earned Value
calculations is that we can tell the business how well or poor we performed to
date, but we can not accurately recommend to them how things will turn out. The unfortunate effect of this is that
everyone worries more about where they are as a function of where they
anticipated being, but not enough about what value they are providing for the
business as they move forward.
This lack of leading indicators
for the project forces us to continuously rely upon the original plan for
guidance. As the project itself wears
on, deviation from the plan tends to increase and the use of the original plan
as a barometer becomes more and more invalid.
Stepping aside from this deficiency to forecast, projects tend to get
graded on how close to the plan their progress to date lies. In an Earned Value environment, the
significant measure is the budgeted cost of work performed. As it states, this
is a measure of how close to the planned cost you were able to execute the work
completed.
The focus on lagging indicators
produces a set of behaviors in the Project Manager and subsequently the
team. Since the team wants to be seen as
successful (even more so than it wants the project to succeed), there is a
desire to meet the plan and to be able to correlate work completed to budgeted
costs. Because these objectives are
almost always ambiguous, the teams can claim success - producing another
behavior, one of obfuscation of completion status. This is manifested as the popular problem of
tasks being on schedule all the way until they are supposed to deliver, and
then stalling at 90 percent complete.
An age old complaint is that the
teams and the teams' management end up finding ways to ‘game' the system and
even though the PMO methodically goes through these monthly evaluations to
assess progress, it knows the progress is merely a sham. Only once delivery milestones are met or
missed does the business really understand the entirety and impact of the
situation. This ultimately ends up as to
why we continuously hear of projects being canceled after significant funds
have already been spent with no clear path to success.[ii]
Traits of
an Agile PMO
The Agile PMO also provides a
check and balance process for the project, but there are only two real measures
used by an Agile PMO. These measures have become synonymous with the tools
designed to help track them - the Burndown, which tracks how the team is doing
on their commitments and the Burnup, which tracks the achieved value to date that
these commitments are bringing the company (see Figures 1). Both of these graphs include trend lines as
their most substantial metric. The
graphs themselves are clearly plotting lagging indicators, but the trend lines
represent leading indicators. When the
Project Lead presents the
graphs to the PMO, if the trend lines do not imply that the iteration and the
release have a sufficient probability of success, then a dialog can ensue
focusing on how the team is trying to address the deficiency.
Figure 1:
Sample Burndown and Burnup Charts. Burndowns graphically
display the how the amount of work the team has committed to complete changes
over time (hopefully is lessens as the iteration progresses, but it does not
have to be monotonic). Burnups graphically display progress against an
estimated scope level in discrete terms of stories completed. Scope is allowed to fluctuate based upon the
business' needs and is usually represented on a Release basis.
The role of the PMO here is to be a sounding board for the
ideas the team has developed, as well as a periodic check to make sure that the
team is indeed using their tools as they were intended. Though there are certainly ways to ‘game'
this system as well, the constant juxtaposition of leading and lagging
indicators make it much more difficult.
Additionally, when maintained on a daily basis, these indicators allow
the team to immediately see the results of their decisions and actions in terms
of their productivity and schedule. This
in turn reinforces the team's accountability and encourages them to actively
participate in generating solutions to problems as they arise.
Why Agile
projects are often managed using either no PMO (no support) or a traditional
PMO (anti-support)
The majority of organizations
implementing Agile in their project organizations do not implement a separate
PMO for oversight of the projects.
Mostly, this is because PMOs have not been included in the Agile
literature and are overwhelmingly associated with ‘heavyweight' processes. In the attempt to offer something new, most
practitioners subsequently reject all things old. The difficulty with this is that when
projects get into trouble, most project managers are so deeply invested in them
they frequently ignore the signs that support, guidance, or intervention is
needed. They lose the independent
council that the PMO offers with its critique and without this, projects in
trouble tend to stay in trouble until acted upon by an outside force. And then it is often too late.
When Agile projects are perceived
to be unmanaged because their releases are not meeting expectations (or for any
other reason), the first step usually seems to be to put a traditional PMO back
in place (or push Agile projects back under existing PMO jurisdiction in mixed
methodology organizations).
Since it is well known that you
can not improve what you can not measure, the traditional PMO attempts to
inflict Earned Value reporting onto the projects. Unfortunately, this usually has a negative
effect as the only way to report Earned Value metrics is to collect them, and
the only way to collect them is to change the process or begin tracking the
project with a second set of books. Even
though well meaning, by applying emphasis on a non-complementary set of practices,
we get a methodological phenomenon akin to two sets of waves meeting
asynchronously - the waves and troughs cancel each other out and the water goes
flat and the nice smooth forward motion stops abruptly.
How to set
up an Agile PMO to best support the transformation
The instantiation of an
Agile-centric PMO at the outset of an Agile transformation can help bolster the
depth and breadth of the changes. Many
practitioners reiterate that Agile is a very experiential methodology and that
there is a fair need to suspend disbelief while trying the first Agile
project. Accepting these sentiments, it
makes sense that the creation of a support mechanism simultaneously with the
initiation of the transformation would be of great value.
Therefore the best way to
inculcate the team to the new methodology is to have one or more experienced
mentors on the team, constantly able to bolster individuals when doubts
arise. If this is appropriate for the
team members, why not for the project managers too? By establishing a PMO made up of a mentor and
some or all of the project managers working on Agile projects, both a support
network and an oversight board are created.
As the PMO, this group retains the responsibility to provide checks and
balances to the Agile projects. My
recommendation is to review monthly - a
period that will guarantee data from at least one complete iteration (lagging
indicators) and initial work and projections for the next (leading indicators),
as well as reactions to the previous iteration's retrospective.
Establishing a peer group gives
the PMs an informal network from which to seek help outside of the PMO reviews,
while also helping them all mature and learn from each others'
experiences. Once the peer group gets large
enough, the actual PMO review responsibility can be rotated through the group
with the mentor and several different PMs being responsible each time.
The actual content of the review
becomes simply reviewing the trendlines for the Burndown and Burnup charts and
asking for explanations on anomalous behavior.
If the charts imply confidence of meeting the business' expectations,
then the review is short and sweet. If
adequate explanations cannot be put forward or leading indicators belie
satisfactory conclusions, then a longer discussion is necessary to help the PM
identify interventions that will change the course of the project. It still remains the PM's responsibility to
steer the team, but as with any retrospective, having these independent
viewpoints often help make the path more obvious.
About the Author
Matt Gelbwaks currently serves as the Chief Agilist for Globant, a leading IT Outsourcing services
provider based in Argentina. Prior to Globant, Matt acted as a Senior Coach at
ThoughtWorks, one of the leading international consultancies specializing in Agile
development as well as served as Director of Change and Product Management at
Segway LLC.
[i] Such as the 2004 report ‘The Total Economic ImpactTM
Of Using ThoughtWorks' "Distributed Agile" Approach' by Forrester Consulting
for ThoughtWorks Inc., and the 2006 paper ‘Distributed Scrum: Agile Project Management with Outsourced
Development Teams' by Jeff Sutherland.
[ii] See the overly quoted Chaos Report from the
Standish group. There is now a series of
these, the original from 1996 and the most recent revision last year. The Standish group has revealed that we
software developers are doing much better now, because 35% of projects can now
be considered successful, up from 16% in '96.
Though a 100% improvement, these are still what I consider baseball
numbers - a batting average of .350 would probably get you consideration for a
batting title, but imagine an airplane manufacture where only 35% of their
planes would fly or a construction firm where only 35% of their bridges were
traffic worthy.
Comments (2)
Jason: ...
Great article. My company is going through a growth spurt and I've been thinking about the 'Agile PMO' for a few weeks but didn't really think something like that existed. It seemed a bit contradictory to me, but the more I think and read about it, the more I'm realizing it's the right thing to do in our situation.
1
April 03, 2008
P. Reilly: ...
Excellent article - my organization,which has a traditional PMO is in the early stages of Agile adoption. This provides some good points for discussion.
Using Agile metrics to manage projects and
strengthen organizational transformation.
