CASE STUDY: How BMC Is Scaling Agile Development

BMC is responsible for the Patrol® product line, which monitors the applications, networks and infrastructure of data centers at many of the largest Fortune 1000 companies. In late 2004, BMC decided to make an aggressive move to grow its revenue by enhancing its two existing Patrol product lines. A single new architecture would have the features and benefits of existing products, while simultaneously increasing scalability and performance.

The challenge was to enhance two highly visible and successful product lines, and do it quickly. Target delivery of the first release was less than a {sidebar id=1} year away, and the teams were split across three countries and six locations spanning 11 time zones. In a bold move, the company decided that the only way to meet the challenge was to adopt an Agile development methodology. BMC chose the Scrum method for planning and managing development activities, and the support of an Agile lifecycle management provider for software and services to accelerate the effort.

Transforming a Traditional Organization to Agile-Ready
If you look at the best practice recommendations for an Agile team - small, co-located, cross-functional with dedicated personnel - BMC's DSM group initially met none of these recommendations. A massive relocation and restructuring program was simply not an option. At the beginning of the transition to Agile, the development teams were:

• Geographically dispersed across six locations - Austin, TX; Houston, TX; Silicon Valley, CA; Pune, India; and Tel Aviv and Tel Hai Israel.
  
  
• Working often in relatively large teams of as many as 30 engineers.
  
  
• Delivering matrixed-in responsibilities with team members constantly pulled away to fight fires.
  
  
• Lacking sufficient product managers to provide the "voice of the customer" on a daily, just-in-time basis.
  
  
• Performing testing and QA activities in India with a relatively large team,  but the typical cycle of ship-test-report defects back-rework-check was simply taking too long to complete.
  
  
• Suffering from laborious, slow, and error-prone systems integration. There was no "continuous integration" environment for this large, complex and network-dependant application suite.

As BMC adopted Agile development practices, it needed to work within some constraints - such as distributed development resources - while addressing the issues under the company's control. With guidance from Agile coaches, BMC made a variety of changes to the organization to ensure success of the Agile process:

Designating an Agile Evangelist
This position was responsible for coordinating the training and rollout process, helping to resolve issues and facilitate communications. She also acted as ScrumMaster for the "scrum of scrums" as the need to coordinate multiple, fast-moving component teams became obvious.

Reorganizing Around Agile Teams and Components
As BMC got more familiar with Agile development, it reorganized the development teams around the Agile process by assigning full accountability for the successful implementation of components to specific teams. Each team included a ScrumMaster, developers, product manager, test/QA, and documentation resources. The teams had responsibility and accountability for delivering tested, quality components into the integration environment.

Making Continuous Integration Critical to System Quality
Over a period of many months, the teams built a continuous integration and test environment that reduced the time it took to produce a system-level build and smoke test from two months to one hour. Continually incorporating code changes from all component teams in a daily build was a substantial challenge that resisted the team's initial efforts. BMC achieved this objective by applying management focus and talented resources. Now, the component teams regularly know if new code they delivered to the integration test bed is working systemically, rather than waiting months to discover how their changes impact the whole.

Coordinating Distributed Teams for Round-the-Clock Development
BMC needed to continue leveraging QA resources in India while adopting Agile methodologies. Its solution was to create blended QA teams that matched one lead QA engineer co-located with the US-based development organization with several QA engineers located in India. The co-located lead QA engineer provided the necessary coordination and communication at daily US stand-ups, and then communicated back to the India-based QA resources on progress and priorities. The result was that BMC was able to gain the benefits of a round-the-clock QA cycle. Unit testing and acceptance testing of new functionality was done overnight with feedback available to the developers when they arrived at work the next day. While this required a substantial political realignment that crossed organizational boundaries, the India resources became empowered to become more directly engaged in the development and quality activities and create new bonds with their overseas team members.

Creating a New Role: The Requirements Architect
As BMC went through initial iterations, it realized there was often a gap between the level of detail in system-level features and the detail needed for iteration planning and execution. The use of simple "throw away stories" was insufficient to the current challenge. To bridge this gap, BMC created a role of requirements architect as part of each development team. The requirements architect was responsible for taking the high-level features defined by product management and decomposing them, on a just-in-time basis, into the more detailed requirements and stories needed to drive iteration planning and development. This new role increased the accuracy of development estimates and improved the collaboration between the development team and product management. Development efficiency also increased by eliminating efforts formerly wasted on detailing requirements at the beginning in an analysis phase that became moot by the time of implementation.

Scaling Agile Practices Across a Large Organization
In BMC's first year of Agile development, it rolled out Agile processes and tools to over 300 people. Along the way, teams learned some key lessons about successfully scaling Agile practices.

Accelerate Successful Adoption with Expert Coaching
BMC kicked off its Agile transition with two self-selected pilot teams. Within the first two iterations, the pilot teams saw many common impediments to producing fully-tested software in rapid iterations: team members were pulled off a project to fight fires; architectural blocks arose due to multiplexed resources; there was inadequate access to product owners; and development teams were spread across three to four cities. Despite these challenges, the pilot teams could see the value of Agile development and were excited to continue. With support from BMC executive management and Agile coaching from Rally Software Development, the teams were able to get their Agile efforts on track and word quickly spread, creating a groundswell of support for the initiative.

Rally's Agile coaches helped BMC with the three major requirements for successful deployment in a large organization. They provided:

1. An academic framework that provides a solid base of knowledge,
2. Consultation and mentoring from experts that have seen these situations before, and
3. On-the-job experience where each team can deploy and adapt Agile techniques to its unique environment.

As individual component teams became successful, the challenge shifted to coordinating across teams. Although iterations were humming along at the team level, it was initially difficult for managers to get a complete picture of how the release was progressing. They were unable to tell when particular features were complete as their requirements crossed team boundaries. Using an Agile project tool for managing the development lifecycle, BMC was able to guide the overall release objectives with themes and high-level features. Features were prioritized with rough estimates in a product backlog and then decomposed into requirements that were scheduled into the component projects where they were implemented with story cards. The tool enabled each component team to see how its commitments related to the whole program, while allowing the company to track progress at the feature level as each team worked through its iterations.

Creating a Highly Responsive Development Organization
The final measure of any development process is in how the resulting software  delivers value to customers. The move to Agile enabled BMC to create products in a way that was much more responsive to changing customer needs. Ultimately, Agile promotes a holistic cycle that ties BMC closer to its customers. Instead of working heads-down for nine months, developers got much quicker feedback on the usefulness of the features they were creating. Even more importantly, BMC has been able to quickly accommodate customer requirements at any point in the product life cycle.

Accelerating Time to Value
Like most enterprise software companies, the DSM group had historically delivered major new releases to customers every 12 to 18 months. With Agile development, BMC's DSM can now offer customers new releases three to four times per year. Customers have access to new functionality earlier than had even been possible before. Customers that request new features can now see those needs met in a matter of months instead of years.

Embracing Fast-Changing Customer Needs
In its traditional 12- to 18-month development cycle, defining requirements often occurred many months in advance of the start of development, sometimes resulting in a two-year lag from customer need to delivered product. This delay often produced stale requirements that were out of sync with the latest user needs and market opportunities. By dramatically reducing the time between releases, BMC is now able to define requirements just ahead of implementation, enabling the business to quickly adapt to changing market realities and take advantage of new opportunities.

Incorporating Proven Engineering and Testing Practices into the Agile Model
It is easy to overlook standard engineering best practices when focusing on the new planning, development and testing behaviors of the Agile paradigm. Sometimes standard BMC practices like performance testing, load testing, and monitoring memory usage weren't executed until late in the development cycle. Pulling these activities forward in the release cycle and maintaining an integration test bed is the responsibility of a newly-formed integration test team that insures that cross-team functionality is performing properly throughout the release cycle.

After the First Year of Agile Success: Next Steps
Already BMC's Agile development has improved key metrics that drive business growth:

• Development teams are more engaged, empowered, and highly supportive of the new development process.
  
  
• Individual developer and team productivity is up by an estimated 20 to 50 percent.
  
  
• Teamwork between product management and development is significantly improved, giving developers better insight into customer needs.
  
  
• BMC returns are improved by reducing investments and waste associated with unfinished or unshipped requirements, designs, and development work.
  
  
• Responsiveness to market changes and customer demands is improved, as BMC increased its ability to incorporate new requirements quickly at any point in time.
  
  
• Customers are receiving critical functionality sooner through more frequent releases.

As external validation of its efforts, BMC's DSM development team was given the 2006 Innovator Award by Application Development Trends (ADT).[1] Using Agile methods, BMC took a project that was less than a year away with teams split across 11 time zones and three countries and successfully combined the company's PATROL solutions into a single platform in just 10 months.

Agile is at BMC to stay.

[1] See http://www.adtmag.com/article.aspx?id=18430&page=5 for a discussion of the ADT award.

Israel Gat, VP Infrastructure Management, BMC Software.

Israel Gat's executive career has spanned some of the world's top technology companies, including IBM, Digital, Microsoft, EMC, and BMC. Digital's NetView, EMC's Cellera, Microsoft's MOM 2000, and the BMC Performance Manager are some of the products he brought to market during his tenure with these companies. In addition to his experience with industry giants, Israel is also well-versed in growing smaller companies and has held advisory and venture capital positions for companies in new, high-growth markets. Israel holds a Ph.D. degree in Computer Sciences from the Israeli Institute of Technology and an MBA degree from Clark University. He lives in Austin, TX with his wife and 12-year-old son.

Ryan Martens, Founder and Chief Technology Officer, Rally Software Development Corp.
Ryan Martens is an expert in assisting organizations transition from traditional development processes to Agile techniques. Before founding Rally - his fourth software start-up - Ryan directed the corporate adoption of Internet technologies within Qwest Communications and also co-founded Avitek, a Boulder-based custom software development firm, which culminated in an acquisition by BEA Systems in 1999. At BEA, Ryan was instrumental in growing that division to more than $50M in revenue within its first 12 months. Ryan received his Masters in Business Administration and his Bachelors of Science in Civil Engineering from the University of Colorado at Boulder. Ryan currently serves on the board of the Agile Alliance.

{mos_sb_discuss:8} 

Comments (0)

Write comment

Show/hide comment form