This article was written by Peter Vaihansky and Jeff Sutherland

One of the common objections to using agile approaches is, "What if our project is too big?" Indeed, agile almost invariably suggsts small teams. However, our experience of the past several years shows that agile methods such as XP and Scrum can be successfully applied to large-scale projects. In fact, Scrum is the first process with well-documented linear scalability. When you double team size in a well-implemented Scrum, you can double software output, even when the teams are distributed and outsourced.[1][2] In contrast with historical experience large projects can be just as productive as small projects for the first time.[3]

This article discusses some of the highlights of a distributed Scrum project by SirsiDynix (Provo, UT) and StarSoft Development Labs (Cambridge, MA and St. Petersburg, Russia). The project focused on the new implementation of platform and system architecture for a complex Integrated Library System (ILS) called Horizon 8.0, which is best compared to a vertical market ERP system with a public portal interface used by more than 200 million people.[1]

Best current Scrum practice is for local Scrum teams at all sites to synchronize once a day via a Scrum of Scrums meeting. In this case, {sidebar id=1} we describe something rarely seen on large, distributed teams. At SirsiDynix, all Scrum teams consist of developers distributed across different sites. Any team member from any site can work on any team task. While some Agile companies operate in this geographically transparent manner on a small scale, SirsiDynix and StarSoft have successfully implemented fully integrated Scrum teams with 56 developers in the U.S., Canada, and Russia.

New best practices for distributed Scrum seen on this project include:

1. Daily Scrum meetings of all developers from multiple sites.
2. Daily meetings of Product Owner team.
3. Hourly automated builds from one central repository.
4. No distinction between developers at different sites on the same team.
5. Seamless integration of XP practices like pair programming and aggressive refactoring with Scrum.

While similar practices have been implemented on small distributed Scrum teams, this is the first documented project that demonstrates Scrum hyperproductivity for large distributed/outsourced teams building complex enterprise systems.

Let's look at the project in more detail and see how some of the obvious challenges were addressed.

Business Problem
SirsiDynix is a global leader in technology solutions for libraries. SirsiDynix has approximately 4,000 library and consortia clients, serving more than 200 million people through more than 20,000 library outlets around the world.

SirsiDynix was confronted with the requirement to completely re-implement its flagship product, a legacy library system with over 12,500 installed sites across the globe. The large number of developers required over many years in the midst of a changing business environment threatened to obsolete many feature requirements in the middle of the project. To complicate matters further, the library software industry was in a consolidating phase. Dynix started the project in 2002 and merged with Sirsi in 2005 to form SirsiDynix.

Fortunately, Dynix started the project with a scalable Agile process that could adapt to changing requirements throughout the project. Building a large product and facing time-to-market pressure, Dynix needed to quickly double or triple its productivity to exploit a business opportunity. The local talent pool was not sufficient to expand team size, and salary costs exceeded the constrained Ramp;D budget. The project could only succeed by augmenting Dynix' resources with outsourced teams.

On the other hand, outsourcing is only a solution if Agile practices are enhanced by capabilities of the outsourced teams. The primary driver was enhanced technical capability resulting in dramatically improved throughput of new application functionality. Cost savings were a secondary driver.

Partner Selection
Jack Blount, President and CEO of Dynix and later CTO of the merged SirsiDynix company, selected StarSoft because of its history of successful XP implementations and its experience with systems level software. The combination of high risk, large scale, changing market requirements; merger and acquisition business factors; and the SirsiDynix experience with Scrum combined with StarSoft success with XP led the team to choose an Integrated Scrums implementation. Jack Bloun's past experience with Agile development projects at US Data Authority, TeleComputing and JD Edwards where he had used Isolated Scrums and Distributed Scrum of Scrums models was a key factor in his decision to structure the project as Integrated Scrums.

System and Tools
The project uses a three-tier architecture and uses Hibernate as a database abstraction layer. Oracle 10g, MS SQL, and IBM DB2 support is provided. The JBoss 4 Application server is used with a Java GUI Client with WebStart bootstrap. It is a cross-platform product supporting MS Windows 2000, XP, 2003, Red Hat Linux, and Sun Solaris. Built-in multi-language support has on-the-fly resource editing for ease of localization. Other key technologies are JAAS, LDAP, SSL, Velocity, Xdoclet, JAXB, JUnit, and Jython.

The project uses the Jira issue tracking and project management software. This gives everyone on the project a real-time view into the state of Sprints. It also provides comprehensive management reporting tools. Figure 1 shows the Sprint burn-down chart and a snapshot of Earned Business Value on the project along with a synopsis of bug status.

Scrum Meetings: Addressing the Time Difference
One of the obvious obstacles to keeping everyone on the project synchronized is the 10-hour time difference between St. Petersburg and Utah. To create an overlap of three to four hours, the Russian team adjusts its schedule and comes in at 12:00 p.m., while the US team starts its work between 7:00 and 7:30 a.m. local time.

Teams meet across geographies at 7:45am Utah time which is 17:45 St. Petersburg time. Teams have found it necessary to answer the three Scrum questions in writing and distribute the answers by email before the Scrum meeting. This shortens the time needed for teleconference on the joint meeting and helps overcome any language barriers. Each individual reports on what he did since the last meeting, what he intends to do next, and what impediments are blocking his progress. The shifted office hours allow for enough overlap so that both sides of the project can stay online after the Scrum meeting to jointly complete some immediate follow-up items.

The team uses email exchange on the three questions before the daily Scrum teleconference throughout the project to enable phone meetings to proceed more smoothly and efficiently. These daily team calls help the people in Russia and the U.S. to learn to understand each other.

For greater focus and productivity, the Russian team starts its day with a local Scrum meeting. Everyone is using the same process and technologies and daily meetings coordinate activities within the teams.

Sprints
The duration of the Sprints was chosen to be two weeks: this was as long as SirsiDynix felt it could go without synchronizing and getting the complete picture of what is happening in the project. The team holds a Sprint planning meeting that is the same as an XP release planning meeting in which requirements from User Stories are broken down into development tasks.

The lag time for a Utah Product Owner response to questions on User Stories forces multitasking in St. Petersburg and this is not an ideal situation. Sometimes new tasks are discovered after querying Product Owners during the Sprint with additional feature details.

Code is feature complete and the team provides a demo at the end of each Sprint. If it meets the Product Owner's functional requirement, it is considered done. It is not deliverable code, however, and SirsiDynix wants to strengthen its definition of quot;donequot; to include all testing. Failure to do this allows work in progress to cross Sprint boundaries, introducing wait times and greater risk into the project.

Scrum Masters and Architects
SirsiDynix was able to achieve strong central control of teams distributed across geographies by centrally locating ScrumMasters, Product Owners, and Architects. ScrumMasters are all located in Provo, Utah or Waterloo, Canada, and meet in a Scrum of Scrums every Monday morning to coordinate work across teams. Architects are all located in Utah as well, and are directly allocated to production Scrum teams. The Architecture group also meets on Monday after the Scrum of Scrums meeting to control the direction of the project architecture through the Scrum meetings. A Product Owner resident in Utah is assigned to each Scrum team. The Chief Product Owner meets regularly with all Product Owners to assure coordination of requirements. This setup enables SirsiDynix and StarSoft to get consistent performance across all distributed teams.

Knowledge Transfer
At the start of the project, SirsiDynix brought a key senior domain expert to St. Petersburg for a week to train the Russian engineers in the object domain and the library workflow. The client also demoed parts of the product to the Russian team. After that, top product architects come from Utah to St. Petersburg for three weeks to coach the Russian team in how the product works. At several points during the project, key members of the St. Petersburg team travel to SirsiDynix offices for training.

Having people rotate between the US and St. Petersburg helps achieve two goals: to share knowledge and to create personal connections between counterparts on both sides of the Atlantic. Introducing people personally creates a friendlier atmosphere, and work related issues are resolved much faster.

Marrying XP and Scrum
StarSoft is an XP company and tries to introduce XP practices into all its projects. Pair programming is done on more complicated pieces of functionality. Refactoring is a planned activity for some of the Sprints and is not done in every iteration as in XP. As you can see from Figure 2, some radical refactoring has occurred as the project approached completion of its critical stage, without loss of functionality. Another XP practice, continuous integration, is implemented through hourly builds from the central code repository.

Figure 2: SirsiDynix and StarSoft: Lines of new Java code in thousands (2003-2006)

Key Conclusions and Takeaways
his project is still active today, and there is always room for improvement. In fact, even as you are reading this, there are new approaches being tried by the project team to further improve quality and speed (e.g., some elements of test-driven development are being applied). However, both from an independent observer's point of view and from supplier's perspective, we want to offer you the following points for consideration:

1. Based on the average number of function points generated per developer/month in a waterfall project (which is 3 for a project this size) [2], the SirsiDynix project (with 15.3 function point generated per developer/month) is almost as productive as the small Scrum project with a collocated team of four to five people (17.8 FP per dev/m). For a globally dispersed team, it is one of the most productive projects ever documented at a run rate of more than five times industry average.
2. Even if your own development organization is not Agile, there is a compelling argument for using Agile-capable suppliers if you need to outsource your development. We feel that shifting to Agile processes with your in-house team will yield greater benefits than outsourcing to a waterfall supplier.
3. It is extremely easy to integrate Scrum with XP practices, even on large distributed teams. This can improve productivity, reduce project risk, and enhance software quality.
4. As counterintuitive as it may sound, single teams with members distributed across sites can enhance code ownership and improve autonomy essential to team self-organization. One Scrum meeting a day has been critical to the cohesiveness of the project, and it includes all team members across geographies. Most outsourced development projects do not hold formal daily calls and the communication bridge is never formed.
5. Written communication prior to joint meetings improves communication and reduces misunderstandings due to cultural and distance barriers. (Communicating in writing as well as on the phone helps mitigate any risk of misunderstanding due to poor phone line or accents.) Project leaders in Provo, Utah, and St. Petersburg have a remarkably common view of the project because of the transparency and frequency of communications.
6. Automated communication of Product and Sprint backlogs throughout the organization combined with upward reporting of Scrum status to management can tightly align a global organization.
7. The issues of Product Backlog being quot;readyquot; for implementation in a Sprint and working software being quot;donequot; at the end of a Sprint are key areas where even the best teams need improvement. 

[1] J. Sutherland, A. Viktorov, J. Blount, and N. Puntikov, quot;Distributed Scrum: Agile Project Management with Outsourced Development Teams,quot; in HICSS#39;40, Hawaii International Conference on Software Systems, Big Island, Hawaii. IEEE, 2007.

[2] Jones, C., 2000, Software assessments, benchmarks, and best practices / Capers Jones, Addison Wesley (Boston, Mass.).