Continuous improvement is at the heart of Agile. Retrospectives provide opportunities to the Teams to inspect their process, and continuously improve it. Many Teams fail to take advantage of this and thus fail to improve.

Common Retrospective mistakes are

Ian Mitchell is Chief Scientist at proAgile Ltd. He can be followed on Twitter: dr_ian_mitchell

Ian Mitchell is Chief Scientist at proAgile Ltd. He can be followed on Twitter: dr_ian_mitchell

Ian Mitchell is Chief Scientist at proAgile Ltd. He can be followed on Twitter: dr_ian_mitchell

"Constant development is the law of life, and a man who always tries to maintain his dogmas in order to appear consistent drives himself into a false position".  

- Mahatma Gandhi

Contracts, and why you can't avoid them

The Agile Manifesto places customer collaboration over contract negotiation. Yet it must be admitted that it would take a hardline ascetic to work without a contract at all. Look at it this way. Without a contract of some form to bind parties into an agreement, the prospects of being paid start to look bleak. 

This might not faze a yogi who eschews material wealth, but to employees of the tech sector an unreliable pay check presents a serious barrier to productivity. Consequently, without a contract, the client's prospects of receiving a deliverable that is fit for purpose can be expected to diminish as well. No matter how uncool and unhip the drafting of legal agreements might be when compared to the visceral joy of crafting code, agile developers need something in place upon which the expectations for delivery can be predicated and managed.

In many ways Gandhi was a hardline ascetic, and an examplar of the sort of intellectual who can live without written contractual guarantees from others. Yet he was also a canny lawyer, and in retrospect we can see he was right to assert that constant development is the law of life. It can be argued that the agile revolution has been founded upon this very principle.

When good contracts turn bad

There is a definite tension between the need to express contractual obligations, and the need to embrace change.

On the one hand, a contract needs to be precisely formulated and exacting. A well-written one will leave no room for ambiguity in interpretation, and the authoring of such documents has rightly become the domain of legal experts. We may enjoy painting lawyers as grasping types who relish discord and the opportunity to harvest fees, such as when the parties to an agreement fall out and seek redress through the courts. However, the measure of a good contract is the extent to which it satisfies the stakeholders. In fact a well-written contract will help to keep people out of the courtroom. It should also be remembered that legal agents can incur negative publicity and professional liabilities from a poorly drafted agreement. So in truth, contract lawyers are motivated to produce terms that are defined as tightly as possible, and which are unlikely to result in acrimonious wrangling. On the other hand this drive towards unambiguous contractual specification makes change hard to accommodate. This is a problem. If market conditions change during a project - perhaps something which can lead to the business case being questioned or even invalidated - then the terms of the contract still hold. If the requirements are only poorly understood at the beginning of a project - but are subsequently clarified - then the original scope in the contract also still holds. The requirements and conditions which were enumerated within the contract are everything. Subsequent clarification and changing conditions count for nothing. Where is the client's best interest now?

What this shows is that although we need contracts in order to deliver products and services, it is horribly impractical to apply them in extremis. We need flexibility, and the traditional approach to squaring this circle is the Change Control Mechanism. 

Welcome to Hell

The Change Control Mechanism started with the best of intentions. Since contracts become out of date as requirements and the business environment change, why not formalise a process of amendment? The original contract doesn't have to be set in stone. It's perfectly admissible to replace one contract with another one. If that's too cumbersome, then it is equally possible to replace outdated portions of a contract with an up-to-date addendum, as long as it is signed and understood by the impacted parties. 

Unfortunately, the devil is in the details:

1. Changes must be recosted. The project budget will need to be revised accordingly. Since changes usually involve new requirements or rework, the price (referred to as the "commercials") will normally be revised upwards rather than downwards. It will, of course, usually fall to the supplier to drive through any such amendments. It should be noted that the time a supplier spends drafting and finalising contractual changes is also potentially billable.
2. Before a contract can be amended, a proposal must be drafted which outlines the recommended changes along with their justification and an indication of what the revised commercials will be. This proposal is known as the Change Request (CR) and there will typically be some sort of pro forma for this. Clients and suppliers can each have their own pro forma. However it is most often the client's pro forma which is adopted since they must review and approve the changes and associated costs.
3. In theory, it is the client who should initiate Change Requests since they ultimately own the product. But in practice a client is likely to communicate new requirements informally with the expectation that they will "ride" on the original budget. The onus is then on the supplier to initiate (and take ownership of) a Change Request.
4. From the client's point of view, there is a huge opportunity cost associated with Change Requests, or more specifically with the refusing of them. This is because the only alternative to approving essential changes with revised commercials might well be to choose another supplier. This may be far too expensive to be practical.
5. From the supplier's point of view, there is little opportunity cost associated with Change Requests. Very small changes may be absorbed by the supplier, but since the time spent raising and progressing CR's is itself billable, the threshold that must be reached before a CR is triggered by a supplier is low. That threshold is largely determined by political considerations, in that a surfeit of CR's may be viewed unfavourably at board level.
6. From the client's point of view Change Requests are "distasteful". This is because of the cost increases that typically accompany them. A client would instinctively prefer to progress changes informally so no costs are incurred, or perhaps - that old chestnut - under the guise of bug reports. However this does not serve the client well since a deviation from contractual conditions would leave them exposed, especially if they can be seen to have initiated it.
7. From the supplier's point of view Change Requests are great! Any change to the original contract can, in principle, be subject to a CR and to cost increases which the client may have little option but to pay. Suppliers also know, through experience, that change is inevitable no matter how sure a client thinks they are about requirements. In practice it is quite possible for a seasoned IT supplier to double or even triple the original contract size through Change Requests. What was that about lawyers being grasping types who relish the opportunity to harvest fees?

We clearly need a better way to "square the circle". To be more specific: it is clear that industry needs to replace the established Change Control Mechanism with a contractual model that supports Agile practice and a more evolutionary take on requirements.

Lifting the Curse

A few months ago Gabrielle Benefield (Scrum Training Institute) & Susan Atkinson (Gallen Alliance Solicitors) wrote a paper titled The Curse of the Change Control Mechanism. I reckon they're right. That's exactly what Change Control has become - a curse. The alternative is to look at the problem afresh, and to promote an "evolutionary" contract model which better supports change. 

I and other proAgile staffers have been applying such contractual models since 2009, originally as part of the Codeworks DEV programme. A key plank of that initiative was the incorporation of an Agile-friendly contract model which redresses the balance between stakeholders, and which holds clients and suppliers to be equal partners in successful ongoing delivery.

Evolving better requirements

Technically, the core of the problem that needs to be solved can be reduced to two observations:

1. Once defined, the value of requirements deteriorates over time. In fact requirements demonstrate the pathology of a "half-life" similar to that of radioactive decay. According to a Kansas University study, the half-life of a requirements set (i.e. the time before it loses 50% of its value) is typically about six months.
2. The ability to define project requirements improves over time. In other words, the uncertainty associated with a project diminishes as the project progresses - a pathology sometimes referred to as "the Cone of Uncertainty". This is because project members will learn things and grasp the problem domain better. However, this improvement is subject to the law of diminishing returns. The first few days may reduce uncertainty by 80%, but it could take several weeks to reduce it by 90%. 

In short we need to capture requirements quickly enough so they don't lose value...but not so quickly that uncertainty remains a problem, and not so slowly that "paralysis by analysis" becomes an issue. 

If we use an iterative approach then we can constantly track the changing requirements. Assuming each iteration lasts only a few weeks, we won't get even close to reaching that six month half-life, and the requirements set will be allowed to evolve. At some point, requirements will have been clarified to the extent that the additional return on investment in clarifying them further will not be justifiable. But if we are also releasing incrementally, by the time that happens we'll have delivered a viable system. And as we know, that "proof through ongoing delivery" is a key part of the Agile sell.

The Evolutionary Contract as a baseline for Agility

Seen in this light, the duties of an Agile-friendly "Evolutionary Contract Model" are threefold.

Firstly, the contract must define the Agile process that will be adhered to during the project:

1. It needs to stipulate that the process will be timeboxed. The precise terminology can vary - "Sprint" or "Iteration" perhaps - but the fact that it is time-constrained must be declared. 
2. The length of each iteration must be stipulated, such as 2, 3, or 4 weeks. 
3. The mechanism to be used for raising and prioritising requirements must be described, along with the roles and responsibilities of those involved. The fact that the initial first-cut of requirements (scope) is subject to ongoing amendment needs to be stipulated.
4. If requirements can be traded in and out of timeboxes, then the mechanism for estimating their difficulty or complexity should be indicated, so that like can be traded for like. Again, the roles and responsibilities of those performing the estimates needs to be elucidated. Will bugs be fixed gratis, or will they be raised as new requirements tickets?
5. It should be made clear whether or not the client (or product owner) can reprioritise timeboxed requirements once the timebox has started, or introduce new ones, and if so what the process for accommodating this will be. Some agile methods do not permit such interference mid-iteration.
6. Client responsibilities for making business representatives available must be specified, including domain experts and/or product owners.
7. The protocols of issue escalation need to be tabulated, and any special reporting requirements (above and beyond incremental evaluation) must be clarified, as do any special mechanisms for risk management.
8. The protocols of incremental evaluation and release must be described. Who will evaluate the releases? Will demonstrations be held in client or supplier environments? Is Continuous Delivery required? How will QA and UAT be done for each release? How much formality is required for sign-off? What documentation will be supplied?

Secondly, the relationship between cost, time, and scope has to be nailed down. There are two common Agile approaches to this:

1. Variable time, cost, and scope. This is sometimes referred to as "time and materials" contracting. The client does not know how much the project will cost, but will pay the supplier on an ongoing basis until a satisfactory product is delivered. 
2. Fixed time and cost, with variable scope. This is becoming more common since clients are increasingly price-sensitive and wish to know their financial liabilities up-front. Although time and cost can vary independently, it is usual for a client to expect full-time resources and for the product to be delivered by a certain date. On the principle that "time is money", the correlation between time and cost is typically invariant. In practice, this means that a client pays X which will fund Y iterations, and a product backlog of Z requirements will be reworked, prioritised, actioned, and delivered until the project terminates. 

Thirdly, the initial "first cut" of requirements needs to be specified:

1. Unlike traditional contracts, it does not represent the feature set that must actually be delivered at the end of the project. The first-cut is the starting point from which a product backlog will be populated.
2. It provides an indication to the supplier of the type of expertise that will be required in order to deliver a satisfactory product
3. The first cut can be mapped against a provisional list of milestone deliverables. It should be noted that this project schedule can be revised iteratively without the need for any change requests, since the mechanism of change is incorporated into the contract.

Special case contracts

There are two special cases of Agile contract for which the heads-of-terms will differ from those outlined above.

Firstly, Lean projects:

1.  A Lean contract will need to be modified to eliminate timeboxing. Instead of negotiating which requirements will be implemented in particular iterations, the product backlog will be constantly topped up and reprioritised. It may be possible to stipulate the expected Kanban velocity in the contract if the number of team members can be fixed along with the time per day that is to be spent on progressing tickets. 
2. Don't assume that an 8 hour day amounts to 8 hours spent on driving velocity. There are always ancillary tasks (such as the provision of assistance) which do not show up on Kanban. In some cases these tasks can be exposed as tickets, and it is recommended that this be done as much as possible in order to promote transparency. However, this is only practical in situations where a product owner can reasonably prioritise the ancillary task. It may be more practical to base any contractual assertions regarding velocity on (say) a 6 hour day.
3. One other thing. It can be argued that Lean projects are not projects at all, but rather operational activities, since it is common for them to function without a business case sensu stricto and without clear rules for termination. The termination clause of the contract will therefore need to be given particular thought.

Secondly, there are public sector projects:

1. These mandate special consideration for two reasons: they typically carry a greater burden of compliance, and they often involve public-private partnerships.
2. The requirements for compliance vary by nation, international agreements, and in some cases by state. For example, bodies in receipt of European Regional Development Funding will need to ensure that their contracts and tendering process are compliant with OJEU requirements, unless the project is very small. Contractually, the best way to manage compliance may be for public sector clients to pre-select a panel of suppliers who can then bid for work. The burden of compliance is then largely reduced to a one-off activity (formation of the panel) and the tendering process for contracts can be made comparatively light-weight.
3. Public-Private Partnerships can require contracts that encompass multiple parties. For example, if an incubator spin-off project is being actioned then we can expect a private start-up seeking matched funding, as well as the public body and the supplier. A tripartite contractual agreement may seem the logical choice but this can become messy. Clearer arrangements can be made if two separate contracts are formulated - one between the public body and the client, and another between the client and the supplier. The contract between public body and client is comparatively simple and will encompass match-funding arrangements and the responsibilities of the client to participate in the process. For example, the contract may stipulate payment in timeboxes. Once the client signs off an incremental release, the supplier will bill them directly and they will pay in full. The client then sends the receipt to the public body who will reimburse them by an agreed amount.

Contracts, flaming torches, and the pitchforks of the baying mob

Let's not forget that there is a meaty and earthy purpose to an agile contract. It doesn't just set the parameters within which work can be progressed. Nor is it there just to make sharp practice harder. The contract also explains to the signatories - albeit in perhaps rather dull legal prose - what their responsibilities are. 

Why point this out? Well, remember that certain parties might not actually know what they are expected to do in order to satisfy their side of an agile contract. For example, a client might genuinely not know that they are expected to participate once every four weeks in the evaluation and signoff of deliverables, or how seriously the exercise is meant to be taken. If they don't appreciate this - if assumptions are made, and these details are left unsaid or glossed over - then a client might well cut up rough when final delivery doesn't meet their expectations. A PR disaster may be in the offing. Upset customers might seek redress publically as well as privately. Flaming torches and twitching pitchforks might start wending their way up the hillside, baying for Agilista blood.

That's why it’s important to baseline expectations in contractual form. Again, the purpose is not to define the scope in detail - it is simply there to establish a first-cut of requirements and to define the mechanism through which changes can be managed, and value delivered on an ongoing basis. It establishes the terms upon which an Agile understanding between client and supplier can be encouraged to develop. 

Conclusion

Remember that a well written contract will actually keep the signatories out of the courtroom as well as away from each other’s throats. Even if the relationship between client and supplier gets no further than a mutual toleration predicated on acceptable ongoing delivery, we can be justified in claiming success.

"It may be true that the law cannot make a man love me, but it can keep him from lynching me, and I think that's pretty important." 

- Martin Luther King, Jr.

Note: I've uploaded a video blog on the above: http://www.youtube.com/watch?v=eDBpgZ1qYvw

Ian Mitchell is Chief Scientist at proAgile Ltd. He can be followed on Twitter: dr_ian_mitchell

Years ago an article appeared in Project Manager Today which elaborated upon why projects so often fail. It certainly ties in with my own experiences of process dysfunction and the symptoms I have learned to identify.

<!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-1610611985 1107304683 0 0 159 0;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-1610611985 1073750139 0 0 159 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} span.EmailStyle15 {mso-style-type:personal; mso-style-noshow:yes; mso-style-unhide:no; mso-ansi-font-size:11.0pt; mso-bidi-font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} -->

According to the market research company, Research and Markets, US spending on IT products and services is forecasted to grow to $514.5 billion in 2010. However, the estimated cost of IT failures to the US economy is $1 trillion! Now we’re talking about IT system failures and their net affect on the companies that rely on those systems. But you can bet money on the fact that the institutions and companies experiencing these losses know the IT company who built their system. Furthermore, they’re not going to be buying from that company again, much less recommending it to their partners and contacts. In any company, large or small, your reputation is your biggest asset – you want to be in that 8% group. If your IT projects are failing, then you are not going to win the market share you need to compete in that $514.5 billion market!
The following 5 principles lay out the steps you should be following, to make sure that your projects are completed on time, and on budget, successfully. Read them, learn them, know them, teach them and then make them stick.
Principle One: Communicate
• 40% of project managers cited poor communication as the leading cause if IT project failures
If everyone involved in your IT project doesn’t know what they’re working on, when it’s due, how to get it done and who the audience is, how can you possibly expect your project to succeed?

According to the market research company, Research and Markets, US spending on IT products and services is forecasted to grow to $514.5 billion in 2010. However, the estimated cost of IT failures to the US economy is $1 trillion! Now we’re talking about IT system failures and their net affect on the companies that rely on those systems. But you can bet money on the fact that the institutions and companies experiencing these losses know the IT company who built their system. Furthermore, they’re not going to be buying from that company again, much less recommending it to their partners and contacts. In any company, large or small, your reputation is your biggest asset – you want to be in that 8% group. If your IT projects are failing, then you are not going to win the market share you need to compete in that $514.5 billion market!

The following 5 principles lay out the steps you should be following, to make sure that your projects are completed on time, and on budget, successfully. Read them, learn them, know them, teach them andthen make them stick. http://www.elementool.com/ebook/CompleteProjectsOnTime.pdf

Ever since I came across Agile methods some years back, I have become a die-hard fan of it. I wonder, why Agile is not been accepted as Industry standard for Software development, even after having many advantages over other methods.

The answer to this question according to me is reluctance to CHANGE. Agile requires difference kind of mind-set; it is a paradigm shift as compared to traditional managed projects. Traditional Managers do not want to change from the way they are working for years i.e. they do not want to come out of there comfort zone and explore new ways to success.

I have come across many people, who by name of ‘Short Release’ become uncomfortable with Agile. I do not have any statistics, but I think, unless Agile is enforced by Customer, there would be few people / project / company, who would advocate Agile.

-Vinayak Mehta