“How long’s it gonna take?” My response: “Six weeks, plus or minus two days. No more.” In this article, I’ll give a rebuttal to Daryl Kulak’s article, “Let’s Stop the Wishful Thinking.” I will show why his beliefs about software estimating, while understandable, are questionable because of the advent of the Test Requirements Agile Metric (TRAM).

We Are Such Good Estimators!
The question posed to me along with my reply uses a real-world experience. The head of development at NEC was asking on behalf of a customer how long it would be until the final release of the customer’s software. The customer wanted it within three weeks. Although everyone felt it would be possible to release within that timeframe, I knew a six-week estimate would be more realistic. The use of TRAM helped me justify my statement, not just to the rest of the development staff but also to the client.

Storypoints
This is where Scrum’s “storypoints” would have failed us. Equating storypoints to an amount of time is ludicrous for a development team. As Mr. Kulak says, “It’s ridiculous. The power of the storypoint in estimating user stories is that it is vague. Keep that power.” Since we are trying to eliminate the vagaries, we will eliminate the storypoints.

You might ask yourself, “Oh, no! How will we estimate?” You do so by using the TRAM’s method of verification points. Verification points are a defined, rather than a fuzzy, metric. Each requirement is given a score based on the type of defect it would cause if it failed, as follows:

• Catastrophic: The defect could cause disasters like loss of life, mass destruction, economic collapse, etc. This severity level should only be used in mission- or life-critical systems and must have at least exciter (see below) priority.
• Showstopper: The defect makes the product or a major component entirely unusable. In mission- or life-critical systems, failure could be hazardous. This severity level must have at least recommended priority.
• High: The defect makes the product or a major component difficult to use, and the workaround, if one exists, is difficult or cumbersome.
• Medium: The defect makes the product or a major component difficult to use, but a simple workaround exists.
• Low: The defect causes user inconvenience or annoyance but does not affect any required functionality.

These scores are then modified by the priority level:

• Mandatory: The defect is highly visible or critically impacts the customer. It must be repaired prior to general availability release.
• Exciter: The defect has significant impact upon the customer and inclusion of this functionality would greatly increase customer satisfaction with the product.
• Moderate: The defect moderately impacts the customer and should be repaired before a general availability release, but it is not necessary unless at least medium severity. This level is also used for requirements that have not been prioritized.
• Recommended: The defect has some impact upon the customer and should be repaired before a general availability release, but it is not necessary unless the defect is scored at least high severity.
• Desired: The defect has a low impact upon the customer and should be repaired before a general availability release, but it is not necessary.

As you can see, while defects are still prioritized, some requirements will have the same priority. This is not a problem; rather, it is the developer’s choice to determine which defect to fix next or the requirement will be rescored.

The team decides on the requirement’s severity, and the product owner on the requirement’s priority. Since the estimation does not handle time in “man-hours,” but rather in “team-weeks,” the estimation has more value because all those little bumps in time are smoothed out. You don’t even need to show an employee’s time off, as other members of the team will be able to pick up his tasks.

Verification points, being a defined metric, are the same no matter who calculates them. New York, Denver, or New Delhi, a verification point is a verification point and means the same thing. This is not true of storypoints. With verification points, you’ll be able to determine which of two teams should develop faster by using simple velocity. The total verification points for a project are a good metric for determining the development effort’s overall size. You can accurately determine how many verification points can be cleared during each iteration by using a form of iterative development, over time. Cleared verification points represent deliverable software. Verification points cleared by the team per day, week, iteration, or sprint is a valuable metric that can be used to show how much effort was required per verification point, determine how rapidly a project can be completed, and estimate a project’s duration and cost.

At NEC, I implemented the TRAM on the mobility project to aid in determining how much functionality to attempt during each sprint. Fourteen weeks into the project, our customer asked us to make a final delivery of the project within three weeks. Management hoped that we could do that for them. However, we had only cleared 280 verification points of product during those fourteen weeks, giving us a velocity of twenty verification points per week. As there were 120 verification points of product still in the backlog, we told them that our best estimate for completion would be six weeks. It is worth noting that the TRAM analysis estimate was 100 percent accurate. We made the final delivery of the project in exactly six weeks. One thing that I was asked by management at NEC was, “What if we made more overtime mandatory to attempt to get the project out in three weeks?” We already had mandatory overtime on Saturdays for the prior three weeks, and the effects were not helpful. During the first week, the team worked an additional day and produced an additional day’s worth of product. In the second week, however, production started to fall. The team only produced 90 percent of what it had accomplished during a normal workweek. The third week, production slipped to 80 percent. Obviously the team was burning out. Rather than slip further to 60 percent, which is where the team was heading, I recommended cessation of mandatory overtime, which was then implemented. Velocity then returned to pre-overtime levels. This saved the company two weeks of development time and associated costs. For a team of sixty, this was a significant monetary savings for NEC, proving the power of verification points.

Verification points from the TRAM are a very good way to save your company time and money while producing very accurate estimates that will be useful to business people.

About the Author
Scott G. Ames has fifteen years in software quality, is a Certified ScrumMaster, and is the chief TRAM engineer at Good-To-Go!

Trackback(0)

TrackBack URI for this entry

Comments (3)

...
written by Daniel GreenLeaf, January 23, 2012

Scott,

This all sounds great in theory, but then again so does communism. In general, most IT estimates don’t focus enough on QA (including testing). This is a problem in both the Agile world and Traditional. In TRAM, at least from what I have seen so far, you take the opposite approach, and base all your estimates on QA.

You’ve swung the pendulum to far the other way. I can have Catastrophic requirements that have a high VP score and take little time to develop, or I can have an item with a VP score of 1, that takes weeks to develop. For example, imagine a 911 center operator application.

If operators can’t log in, people will literally lose their lives as emergency services can’t be directed. However, if I wanted a feature to open the callers location in google maps with a picture of the location, most people could agree this feature is much less important.

In this instance, creating a login screen may only take a couple of days, but the linkage to google with the display may take weeks. The fact is, any estimate needs to include all of the activities required to complete a project.
As an Agilest, I will not dissuade you from using a technique that works in your circumstance but, I would also caution anyone who saw it as an answer to all their estimation issues.

IT is traditionally bad at estimates (I have often wondered why we don’t have specialists in estimation as we do with every other specialty). If anyone is ever able to fix IT’s estimation problems, they will get my vote for man of the year.

• 
• +1
• 
• 

...
written by KaShNa, December 22, 2011

Hi -

This is just the website I was looking for. I am currently preparing for a PM case interview for an IT firm. Any suggestions on what I can read up? I have to come up with a project plan within an hour and discuss the solution with the panel for an hour. Thanks.

• 
• +0
• 
• 

...
written by Russell Pannone, December 14, 2011

At the heart of the challenge around estimating, especially early in the product (project) lifecycle is to step back and make a clear distinction between “estimating” how long it will take to do something and making a “commitment” to getting it done; while, at the same time making the distinction between the accuracy and precision of your estimate.

Here is a link to a an article I wrote about Differentiating Between Estimating and Committing:

http://www.agilejournal.com/component/content/6014?task=view

• 
• +0
• 
• 

Write comment

Show/hide comment form

Name

Comment

smaller | bigger

Subscribe via email (registered users only)

Write the displayed characters

Add Comment Preview