Performance
and quality metrics are not indigenous to traditional IT practices. When metrics are brought to bear to achieve
greater transparency or compliance in an IT project, they are imposed on teams,
grafted on top of day-to-day execution.
The poor alignment of metric to execution means project managers must
constantly translate work effort into progress measures. Because these acts of translation take a lot
of time and are not natural fits with execution, project management is often
opaque and inconsistent. By comparison, business-oriented
metrics are natural byproducts of the way work is performed in Agile
practices. Measures of progress, quality,
and functional completeness are extensions of day-to-day execution of Agile
practices. This means that instead of
chasing after management data, the Agile manager is able to concentrate his or
her efforts managing the people in a team to achieve the business goal.
Traditional: Metric-Driven
Management
In
traditional application development, business problems are decomposed into collections
of tasks that are performed by technical specialists. The user interface is developed by one group,
middleware by another, and back office functionality by a third. Quality assurance (QA) testers and database administrators
are not core members of a development team.
Instead, they are "shared services" supporting a multitude of teams
across provided by the IT organization. As
a result, a development team doesn't function as an integrated whole, but as a
collection of people in specialist roles.
In this
model, technical execution is explicitly defined at the expense of the business
goal. The business goal is abstract, an
assumed outcome of the completion of technical steps that form the project plan. Because people are executing to an
abstraction of the business goal, it is difficult to assess progress and
quality during fulfillment. This means
there is clarity before a project starts and once it is completed, but not
while it is in-flight. Subsequently, traditional IT delivery is not transparent.
One
common tactic used to get greater visibility into development activity is to
create very finely-grained task orders.
Each task's dependencies are mapped and estimated, and assigned to a developer
in a project team. The collection of individual
work paths creates an overall project plan.
The total time spent versus the total time estimated provides a metric
of "percent complete." This makes time
the fundamental unit of measure for determining team progress. As "time" is the coin of the realm, project
risks are mitigated by securing an excess volume of development capacity.
Advertisement
Quality
in this approach is assessed at the end of the development cycle. Once code is asserted to be development
complete, it is turned over to a QA team which subjects it to a battery of test
cases. Test cases are finely-grained,
written to exercise a variable block of underlying functionality impacted by a
single event. The quality of the software
only becomes visible once test cases are completed to a state of pass or
fail. QA provides little more than
Boolean assessments of fitness as opposed to comprehensive feedback on
functionality. This relegates QA to the role
of a low-value-added auditor as opposed to a high-value-added partner in delivering
a business solution.
Absent
from this model are the costs inherent in coordinating a lot of inter-related
tasks. Because no single development task
defines an end-to-end business requirement, each task requires a hand-off to
somebody else to advance it further in the business workflow. Each hand-off creates the risk of
misunderstanding. Misunderstandings
require rework. The more finely-grained
the task definition, the more hand-offs and the greater the risks. The same applies to QA. A post-development QA exercise requires data
to be created and correct, functional expertise available on-demand, and test
case definition to be clear and articulate.
When this is not the case, QA is not the turnkey event it is projected
to be. These operational realities of
development and QA execution defy accurate definition. As a result they cannot
be estimated, and subsequently become costs unaccounted for in a project plan.
Taking
this approach, the whole of the project is less than the sum of its parts. This results in a reactive project management, obsessed with tracking task completion
as opposed to managing to solution delivery.
Projects are managed by the "fuel
tank," tracking time spent against time estimated to complete technical
tasks. While the fuel tank is an
indicator of cost, it isn't an indicator of direction. To wit: a driver may consume enough fuel to
drive from Paris to Avignon,
but just because the fuel was spent is no guarantee the car hasn't ended up in Lille. This is the disparity of IT project
management today: IT measures expenditure in hours, but the business isn't
buying hours, it is buying software. IT shouldn't
concoct indicators that assess where development might be, it needs to measure where
it is.
Agile: Management-Driven Metrics
Agile practices
offer a departure from traditional project management. The day-to-day activities of Agile teams are finely-grained,
but oriented around business objectives as opposed to technical objectives. This means that meaningful project and
quality metrics are by-products of in-flight work.
To
understand why this is the case, consider how Agile practices work. All participants in development work together
as an integrated team. The core unit of
work, the Agile Story, is a finely-grained business-oriented statement of
requirement that can be delivered in a short amount of time, typically two
weeks or less. Each story is developed
in such a way that there is a high degree of certainty of technical
completeness of the work. Specifically, developers
write both code and unit tests to demonstrate that the code does what it is
expected to do. Newly-developed code is continuously integrated with all other
code, meaning the entire project is constantly being compiled and is executing
all unit tests. As developers complete
functionality - as often as daily - it is immediately subjected to review by a
business partner, and then to QA certification.
In Agile
practices, the team doesn't divide up the work into abstract, incomplete
packages of tasks that fit an IT perspective, but instead delivers complete
packages of functionality that fit a business perspective. This means that the same people who develop
the user interface will participate in developing middle and back-office
components. They will also test and
certify functionality by performing full "round trips" through all the
layers. This allows an Agile team to meaningfully
report progress in terms that the business understands. That is, instead of reporting progress of
"user interface components" or "middleware services," the team can report "this
functionality has been delivered to a state of functional completeness." Progress is measured in terms of stories
completed, which to the business is a more meaningful assessment of "percent
complete" as it is rooted in functional
completeness, not technical effort
required. In Agile, work is
denominated by what the business is actually buying: functionality, not time.
In
addition, relevant quality metrics can be produced in near-real-time. The Agile
practice of continuous
integration provides both quality metrics and gatekeepers on the
deliverable. Code
quality analytics, as well as unit, functional, and integration tests can
be automated in a build
pipeline that produces current assessments of code quality. Test scripts written by QA (including smoke, workflow
and regression tests written in a tool such as Selenium) can be executed in the
same pipeline. As a result, indicators of
quality at technical and functional levels can be obtained effortlessly
throughout the life of the project, not assumed as part of development and only
exposed once an entire project is asserted to be "development complete."
There are
a number of benefits to this approach. Because
progress toward the business goal is reported with day-to-day work, the Agile
project manager does not need to translate team activity into project status
reports. There is a strong foundation
for forecasting: the date and cost of future development can be predicted from the
rate at which work has been historically delivered. Progress is reported in a meaningful
denominator to the business: functionality (expressed as stories) as opposed to
effort (expressed as hours.) Finally,
quality is not a "feeling" of product readiness, asserted by developers and
users from random testing exercises. It
is demonstrable through the constant execution of a thorough library of tests.
These
metrics don't drive management, but are management-driven. With near-effortless accessibility, they
provide unvarnished visibility into what's happening in a project. That means the Agile project manager isn't
spending his or her time scouring data, and then translating and interpreting it
to ascertain what is happening in a project.
He or she is proactive, able
to focus on managing the people to achieve a business goal. Change in throughput and quality indicators
are quickly obtained and provide rapid and meaningful feedback on acts of
management. As a result, the impact of
management decisions - in team composition, in roles and assignments, in scope,
and so forth - is not guesswork.
Metrics are Indicators of the
Project, Not the Project Itself
Metrics
don't tell the entire story of a project.
Unit tests may all pass, but the tests themselves may be of poor quality
(e.g., they lack assertions). The
automated test environment might be different from the production environment -
different operating system, database or application server - and thus mask
technical quality problems. The business
participant may not scrutinize or lack expertise to correct functionality, thus
work reported as complete will not be. Numbers
alone won't reveal this.
Still,
the ability to obtain metrics aligned with business goals that describe the
state of development is of tremendous advantage to the Agile manager. He or she is better able to clearly
communicate the state of a project deliverable in terms meaningful to the
business. For far less effort, the Agile
manager can forecast both calendar time and cost with greater accuracy than the
traditional project manager. Thus the Agile
manager does not waste time chasing up what a team has or has not accomplished,
but is more engaged in managing the team toward achieving the business
objective.
About the Author
Ross
Pettit has 15 years' experience as a developer, project manager, and program
manager working on enterprise applications. A former COO, Managing Director,
and CTO, he also brings extensive experience managing distributed development
operations and global consulting companies. His industry background includes
investment and retail banking, insurance, manufacturing, distribution, media,
utilities, market research and government. He has most recently consulted to
global financial services and media companies on Agile transformation programs,
with an emphasis on metrics and measurement.
He is a frequent speaker and active blogger on topics of Agile
management, governance and innovation.
He is currently a Client Principal with ThoughtWorks.
Comments (4)
Derrick E.: ...
All I want to know is where can I get the drugs that this guy is taking? Performance and quality are new ideas in IT? It's not metrics, it's quality! Oh, wait, well it's metrics too! But metrics aren't the project...it's just an indicator.
Oh, and Avignon and Lille are in opposite directions. That's not a fuel problem, that's a lack-of-mapquest problem.
1
May 15, 2008
David: ...
Ross, You almost have it right. "what the business is actually buying: functionality, not time." is on the mark, but it is not the story (Agile or otherwise). It is the work that the software is supposed to do. If the amount of effort that the end user puts into the system accomplishes a give amount of PAID work (or a reduction of overhead in otherwise paid work) than tne software is accomplishing its task. If the end user is not getting more done or in less time then the software is not doing its job. Like any tool that the staff may use, like a truck that can haul two tons instead of one may be 'better', but is there two tons of stuff to be hauled? What a given manager may 'think' needs to be done may not be what the end user 'thinks' need be done. Keeping the end user team as part of the meaningfull fead back loop is imperitive. The business/IT metric needs be cast as a function of end user work, not IT effort. There have been plenty of 'solutions' looking for a problem for years.
The Agile collaboration schema has a lot going for it, how much office politics plays in the acceptance or rejection of the Agile schema is a bigger challenge.
2
March 25, 2008
John: ...
I think, this is just an over-generalized bla-bla-bla. Article just rephrases some of the Agile/Scrum basics without actually answering the HOW and WHAT IF questions.
In response, my comment is over-generalized as well :)
3
March 03, 2008
Mark: ...
Hello, Ross!
I have liked your article very much and agree that traditional viewpoint on metrics needs some refreshment. From my side I would also suggest to take look at SourceKibitzer - http://www.sourcekibitzer.com metrics solution (desclaimer: I am co-founder of this company). Unlike agile management-driven metrics you suggest, SourceKibitzer solution doesn't rely on any specific development process and can automatically collect numerous indicators with business value.
Would be interested to hear your comments on that.
Performance
and quality metrics are not indigenous to traditional IT practices. When metrics are brought to bear to achieve
greater transparency or compliance in an IT project, they are imposed on teams,
grafted on top of day-to-day execution.
The poor alignment of metric to execution means project managers must
constantly translate work effort into progress measures. Because these acts of translation take a lot
of time and are not natural fits with execution, project management is often
opaque and inconsistent. By comparison, business-oriented
metrics are natural byproducts of the way work is performed in Agile
practices. Measures of progress, quality,
and functional completeness are extensions of day-to-day execution of Agile
practices. This means that instead of
chasing after management data, the Agile manager is able to concentrate his or
her efforts managing the people in a team to achieve the business goal.
