How do you know if an Agile project has succeeded? The typical response I get to that question is a blank stare or moment of silence, depending on whether the meeting is in person or by phone. And if I do get an answer, it's usually accompanied by a "we're just starting" comment. The challenge is twofold: development organizations are notoriously weak in collecting metrics and reporting on their projects and, when using Agile processes, many teams are challenged to demonstrate if the change from traditional approaches was worth it. As IT costs and business pressures escalate, it is critical that a development shop can demonstrate its value to the business. Proponents claim that Agile processes help do this, but they must back up those claims with numbers.
I contacted a number of Agile industry experts to see how they are approaching metrics and business value with their internal and client projects. I posed four (not so) simple questions:
What metrics do your clients typically collect and report on?
What metrics do you advise them to collect and report on, to better reflect an Agile project's success?
Have you seen Agile teams measuring business value, and if so how?
Are there any best-of-breed tools or techniques that you recommend to measure Agile project success?
The responses aren't so surprising, but they do shed light on the complexity of the issues and the immaturity in the market today. Perhaps these comments will stir up some good discussions in your shop.
What do customers measure? What should they measure?
The list of traditional development metrics is obvious. We continue to see teams focused on quality metrics, such as defects per phase or release;
Advertisement
schedule metrics, such as actual effort/days versus estimates and delivery milestone achievements; and cost metrics, such as budgeted versus actual costs, internal (sunk) costs, and also production support financials. Basic application life cycle (ALM) management tools do a decent job of collecting this data, sometimes even in an unobtrusive manner. But as we know, none of these metrics demonstrates why and how an Agile process can affect project success and business value.
John Cunningham, President and Founder of consulting firm Band XI, has a terrific perspective on why Agile processes, Scrum in particular, are well-suited to achieving the team's ultimate goal:
"Most customers initially want your standard PMI project plan along with the ‘stuff' that it provides them. Since many of our projects are short duration, I am usually able to quickly convince them that the cost/benefit of such a heavy process is counterproductive.
I hate to do this... but let's use an analogy to football. I listen to all these analysts (professionals) tell us that the keys to winning are running the ball (increasing time of possession) and stopping the run (decreasing the opposition's ability to control the ball). Nonsense! These are not the keys. Scoring more points than the opposition is the key. I don't care if they are scored by interceptions that are run back or kick returns or long pass plays. Put points on the board. All that matters for the [win] is points. All the rest is noise and people talking to sound important.
We have had the same nonsense in the software industry forever. All that matters to us is shipping software that works reliably, is complete on schedule, and finished within budget. How you get there doesn't matter. If you spend all your time eyeballing tangential metrics, you are taking your eye off the goal. That's why we accept Scrum - it measures how much function got completed on time and with the resources available. Everything else is fluff and make-work."
Where do you start? Most teams transition to Agile metrics and simultaneously maintain more traditional metrics and reporting methods. Scott Ambler, Practice Leader Agile Development at IBM Corporation, suggests that teams start by collecting some basic data, such as:
1. Actual cost (hours, $, ...)
2. Some sort of functionality delivered (use case points, user story points, ...)
3. Defect trends
4. Stakeholder satisfaction
In this way, the team can communicate to the business in the terms it uses (basically money and systems capability) and still continue to track metrics that are core to development success, such as quality. Businesses have many choices for IT services. Make it easy for them to decide.
A measure of stakeholder satisfaction is one of the most important metrics that a team (Agile or not) should introduce. Delivering defect-free code doesn't mean delivering valuable code. Value is in the eyes of the customer; if you're not meeting their needs you're wasting your time and money. Ambler adds "more importantly, are they coming back to you to do the next project?"
How To Measure Agile Projects' Success?
Transforming an organization from measuring software quality and productivity to measuring business value is a difficult process. Matt Gelbwaks, Chief Agilist for SourceForge Enterprise Edition, recommends starting by turning the discussion with IT and business staff to one of priorities. Logically, the highest priority items will deliver the most value -- in the customer's eyes. For an individual project as well as for a portfolio of projects, the team can use this prioritization and begin to think about (and ultimately determine) value. Gelbwaks uses the example of the popular investment in a call center's single sign-on. This discrete software component has a direct impact on value. Companies can easily compute the correlation between the time saved (even a few minutes) by not having to log in to many different systems and the increased number of calls that an individual can field. Each call is value to a customer and potentially revenue to the company.
A major benefit to using Agile processes (or any iterative and incremental approach) is that you can quickly deliver a number of releases to the user community. Not only can you prioritize the features you release based on the value the user will receive, but you will also have quicker access to post-release information to best derive business value. For example, as soon as a software product can help drive revenue, you can begin to incorporate the project's costs and determine the ROI for the project. For companies with short funding windows, this is the best way to quantify results and justify investment.
More measures are not necessarily better. Erik Gottesman of Sapient outlined the company's model that tracks project health along five major dimensions: risk, quality, schedule, effort/budget, and compliance. This provides "just enough" information for both IT and business management to monitor development projects in the context of business value. Then, as necessary, managers can drill down into any of the categories to analyze specific data.
What About Tools?
For managing and monitoring Agile projects, the experts agree: less is more. ALM tools that are intrusive to developers and accumulate reams of technical data do nothing to help manage these types of projects. Many Agile developers cite uses of simple spreadsheets and homegrown dashboards to convey the information they deem appropriate.
Vendors in the Agile management space hear this loud and clear. New releases from Rally and VersionOne, work on open source tools such as XPlanner and the increasing number of contributions for Scrum tools, continue to pare down unnecessary information and emphasize value-driven metrics.
This is a space to watch. Expect to see software vendors and Agile consultants putting a lot of effort into value-driven metrics in the coming year. By communicating success in terms of business value, Agile teams will have no problem demonstrating why the change is both pragmatic and strategic.
About the Author Liz Barnett is the Editor in Chief of the Agile Journal and
Principal Analyst at EZ Insight Inc. Previously Liz spent 10 years as a
Vice President and Research Analyst at Forrester Research, joining
Forrester as a result of its acquisition of Giga Information Group. Liz
held management positions at Accenture, PepsiCo, and Atelier Research.
She also was the Research Director for the advanced software
development and advanced network computing research services at New
Science Associates, prior to its acquisition by Gartner Group. Liz
holds a patent for developing a distributed application
development/CASE tool. Liz earned her B.S. in operations research and
industrial engineering at Cornell University.
Error, missing joomlaboard config file!
Comments (3)
Piergiorgio: ...
Very good article and it's hard for me to have a good, automatic (VERY IMPORTANT), consistent metric about 'A measure of stakeholder satisfaction is one of the most important metrics that a team (Agile or not) should introduce'. What do you suggest? How could you get such a metric? PierG http://pierg.wordpress.com
1
January 23, 2007
Brendan Walker: ...
I think these measures apply to any project, regardless of how rigid or agile it is executed. Perhaps the only differentiation between agile and rigid projects is how the measures are taken. Without having to employ the use of specific tools what are people using to take these mesaures in an agile sense?
* Effort/Cost/Schedule is perhaps the easiest as simple timesheeting can do this, however this can become daunting to the 'workers' if to much detail is applied. * Defects can always be tracked using a bug tracking system. * What about measuring functions delivered - the timesheet again perhaps although is this making it daunting? * What about stakeholder satisfaction? What sort of measures are done here? Is it a simple measure based on feedback from the customer tracked over time?
All these measures are also in respect to the current project. Are we talking about effect of the project on the business as a whole? If this is the case don't we need to be doing things like taking measures before and after and applying them at the business level and not the project level. For example, what does the business care how much time is spent on a specific project task? They should be more interested in how much money you save or make them by the end of the project/iteration.
All this is good food for thought.
What I'd like to know is what tools/techniques are people using to take these measures, how often are they taking the measures and how much time do they take to complete?
2
January 10, 2007
Sanjay: ...
Good Article. Monitoring the defect trends from the field is one of the best indicators of a successful Agile transition. However, this takes time, since the team needs to montor this for a couple of releases before they can publicize the savings. The caveat to this is how stricly is the team adopting the Agile principles -- Are they making sure to automate all the acceptance tests, Do they stop to fix a defect as soon as it's found, do they consistently demo at the end of every iteration. I have seen most teams follow these diligently in the begininng and then start to taper off.
