Home  Articles Continuous Quality - Shortening the Development and Test Feedback Loop Beyond Continuous Build
|
Continuous Quality - Shortening the Development and Test Feedback Loop Beyond Continuous Build |
|
|
|
|
Written by Zach Nies
|
|
Sunday, 08 June 2008 |
 Agile practices emphasize the value of bringing testing
forward in the development process, but has anyone ever explained to you how to
actually make that happen? Most likely not, at least from my experience. The
fact is most teams lack the process and automation needed to get continuous
visibility into the ongoing quality of their software releases.
In Part 1 of this series, I will outline how automation can provide
a great platform to pull quality and testing forward when the focus of the
automation is informed by a few key principles. In Part 2 of this article, I
will take the concepts presented in Part 1 and walk you through the details of
implementing an automation infrastructure, including specific examples.
What the Heck is Continuous Quality?
What do I mean by "continuous quality"? Quality isn't an
event that happens right before a project is done or before a product ships;
quality requires the continuous focus of the team. Just as continuous
integration enables the continuous feedback cycle around integrating code,
continuous quality enables the continuous feedback cycle about the quality of
the system or product being developed. Ultimately, continuous quality is about
reproducibly building integrity in at all levels of the development and testing
process through the effective use of automation.
As an engineer working in an IDE, you should be continually
aware of how your code integrations are affecting quality. As a tester, you
should have a say in the definition of done, visibility into the quality of the
stories under development, and control over your testing environment. As a
product manager, you should be able to understand and provide feedback, in
short cycles, about how a customer will perceive the value and quality of the
stories being developed. Finally as a team, you should have constant and
objective feedback about how quality is trending with the ability to "Stop the
Line."
Principle #1 - Agile's Discipline Around Short, Time-Boxed
Development
In waterfall development, the goal of the testing phase is
for quality to "come together" at the end of development, where I would argue
it is much too late to materially affect quality. Also, there is typically
little focus paid to the deployment or installation work until development is
finished. Focusing on quality at the end of the development cycle puts the team
in a position where fixing any issue often requires reconstructing the context
around code that changed months ago. The problem is that at this point,
everything is out of context. It's like asking what youhad for lunch three
Tuesdays ago. Do you remember?
The short time boxes of Agile development (where small
increments of work are taken from definition through acceptance) enable testing
to be pulled forward. Because the feedback cycles between development and
acceptance are compressed, issues are discovered and resolved while the code
changes are still in context. Do you remember what you had for lunch today? I
hope so. Involving testers in development from definition to acceptance is a
start, but there are ways to make the feedback cycles even shorter through
automation. Short time boxes AND automation help pull testing forward to the
point where quality is continuously understood and acted upon.
Principle #2 - Feedback Cycles and Lean Thinking
Now let's take a look at Lean thinking to understand
feedback cycles and how to compress them. According to Mary and Tom
Poppendieck, improving cycle times involves limiting work to capacity, evening
out the arrival of work, minimizing the number of things-in-process, minimizing
the size of things-in-process, establishing a regular cadence, and using pull
scheduling [1]. The basic cycle of development is captured in the picture
below.
In this cycle, testers and product owners specify functionality, developers
write code to implement that functionality, and the code gets checked into the
source control management system. At some point in the cycle, those code
changes are incorporated into a build.
Testers and product owners use this build to accept the functionality
they specified earlier in the cycle. Without short time boxes, this cycle can
take months or years to complete. Even with short time boxes, the challenge is
to compress this cycle to the point where feedback is continuous enough that
responding doesn't require the team to reconstruct context.
Why should we focus on cycles and batching? To deliver as
fast as possible, queuing theory suggests that reducing batch sizes, evening
out the arrival of work, and including slack can positively influence cycle
times. As utilization goes toward 100%, the cycle times of large batches is
considerably higher than for small batches.
Translated for a development team, this means as a team tries to
schedule 100% of their velocity into each time box, the size of the batches in
the development cycle will greatly affect the time it takes for the work go
through the cycle. The details behind this are beyond the scope of this
article. If you are interested in learning more about queuing theory for software
teams I would highly recommend reading at least chapter 4 of Lean Software
Development by Mary and Tom Poppendieck[2].
Automation and Lean thinking are the keys to enabling
quicker cycles and giving the team continuous feedback about quality in an
actionable way. Let's look at the ‘why' and in Part 2 of this series I will
address the ‘how'.
Unit Tests & Code Metrics
Unit tests and code metrics are a good place for a team to
start. By implementing continuous integration, where the code is built and the
unit tests run on a dedicated machine that is not a developer workstation, you
are most of the way there on this level of automation. You'll need a scriptable
way to create a build, a scriptable way to run unit tests, a dedicated machine,
and the willingness to build with each atomic check-in. If we go back to the
basic cycle of development, continuous integration alone can have an impact on
batching patters and the quality feedback cycle.
The above graphic shows how continuous integration affects
the feedback cycle. By doing a continuous integration build after each
check-in, cycle times and batch sizes can be significantly reduced. Also, the
continuous integration infrastructure is giving the team some continuous
measures of quality. Specifically, the developers are getting constant feedback
on the quality of the code in the SCM system - does the build pass and do the
unit tests pass? Through the use of code coverage tools and static code
analyzers, the development team can get a deeper understanding of the degree to
which each check-in is affecting code level quality. The feedback from
continuous builds along with the development discipline to integrate code
regularly can help a team to even out the arrival of work, minimize the number
of things-in-process, and minimize the size of things-in-process.
Automated Functional Tests
Automated functional testing involves having: a scriptable
way to package a build into a deployable or installable package, an automated
way to deploy or install that package, an automated was to run database
migrations, a dedicated testing machine, and an automated functional testing
framework. If we go back to the basic cycle of development, automated
functional testing provides continuous quality feedback for developers and
begins to remove batching between developers and the rest of the team.
The above graphic shows how automated functional testing affects the feedback
cycle. Now the team is getting constant feedback about the functional quality
of the code in the SCM system. By having an automated way to package and
deploy/install a build, the deployment part of development now becomes part of
the feedback cycle, and the team gets feedback when a code change breaks the
packaging or deployment of the build artifacts. Before this level of
automation, the ability to deploy or install a build is unknown until someone
manually makes an attempt. If something goes wrong, the code responsible for
the problems is likely to be out of the team's context. If the feedback comes
as a direct result of a specific check-in it is easy to understand what is
causing the problem and decreasing quality. The feedback from automated
functional testing can help a team limit work to capacity, even out the arrival
of work, minimize the number of things-in-process, minimize the size of
things-in-process, and establish a regular testing cadence.
Manual GUI and Regression Testing
The next area to focus your automation effort is the
infrastructure to allow testers to pull builds for manual GUI and regression
testing. This is the part that allows testers and product owners to be
completely in control of their testing environment. In teams that lack this
level of automation, a tester or product owner typically has to ask someone in
development or operations to make a build available, which can be a slow and
contentious process that is frustrating and distracting to all involved. To
implement this level of automation, you'll need a repository of build
artifacts, and a way to deploy or install from that repository. The remaining
areas of batching are removed with this level of automation.
With this level of automation, there is no batching in the
feedback cycle and, more importantly, the team can move to pull-based
scheduling for testing. Pull scheduling empowers each person to clearly
understand what he or she should do next to have the biggest positive impact on
the business and have the ability to act on that understanding[2].As
functionality is being developed, testers and product owners are in control of
determining, with developers, what build would be most effective to test and
get quality feedback to the team.
In Part 1 of this series, I outlined how automation and Lean
principles reduce cycle times in a way that allows quality and testing to be
pulled forward. In Part 2 of this article, I will take these concepts and walk
you through the details of implementing an automation infrastructure, including
specific examples.
About the Author
Zach Nies
brings close to 20 years of engineering and product development experience to
Rally's innovative products. Prior to joining Rally, Zach served as Principal
Architect and Director of Systems Architecture for Level 3 Communications and
founded a small start up which was quickly acquired by the publicly traded Creo,
Inc, now a division of Kodak. He also served as Chief Software Architect at
Quark, where he provided the overarching technological vision for the company.
Zach's product vision has won numerous industry awards, including Jolt Product
Excellence awards, Seybold HotPicks and the prized MacWorld Best of Show. Zach
has served on standards bodies such as the W3C's HTML working group and
currently serves on the board of directors for Agile Denver. At the age of 13,
Zach began commercially publishing software and, at age 16, started a successful
consulting business. A Boettcher Scholar, Zach received his BS with distinction
in Computer Science Engineering from the University of Colorado at Boulder. He
spends his spare time tuning his golf swing and spending time with his family.
[1] Poppendieck, Mary and Tom. "Managing the Pipeline."
Poppendieck.LLC. 7 Oct 2007. http://www.poppendieck.com/pipeline.htm.
[2] Poppendieck, Mary and Tom. Lean Software Development,
An Agile Toolkit. Boston: Addison-Wesley, 2003.
Comments () |
 |
|
|
|
Tags:
Click to add your tags...,
|
|
