Emergent Design: Leveraging Agile Retrospectives to Evolve Your Architecture
|
 Technological
debt is mistakenly thought of as a technical problem, but when system design
cannot change according to the needs of the business, it becomes a business
problem. Big Design Up-Front leads to
technological debt. Architecture must be
allowed to emerge according to the needs of the product and the business. We know iterative, emergent development
works; iterative, emergent design is no different. Agile teams should use Retrospectives as a
tool to determine current needs and enable emergent design.
The Case for Emergent Design: Understanding
Technological Debt
Technological
debt is mistakenly thought of as a software problem, but when system design
cannot change according to the needs of the business, it becomes a business
problem. Just like financial debt,
technological debt is a business liability - it affects the liquidity of the
business (see Figure 1). Think of good design in a system as a line of
credit: just as purchasing new equipment
for the business without generating income creates a debt against the credit
line, adding features to a system without improving its design is "borrowing"
against the capabilities of the system. There
can be good reason to do this; after all, credit is used to improve cash flow
and solvency.[i] The danger is in thinking of this temporary loan
as a permanent increase in funds. The
debt must eventually be repaid.
Failure
to reduce the debt (improve the design) in favor of making new purchases
(adding new features) eventually results in a large portion of effort going
toward interest payments (patches and bug-fixes), and reduced buying power
(ability to add new features).
Eventually, this can lead to a situation where all the effort is going
toward interest payments (struggling to keep the system alive), and only
trivial features can be added without major rework.[ii]
If the
business depends on new features for competitive advantage, this technological
debt has now directly affected its ability to thrive in the marketplace. Thus are the seeds sown for the opening line
in a long-standing argument between the business and its technologists: "why does it take you software people so confounded
long to add a simple capability?"
Figure 1:
Understanding Technical Debt
So why not
design the system well up-front? A well-designed system should have complete
requirements, a thorough design document, and analysis that shows it will meet
the needs of its users. Where is the risk of technological debt?
In fact,
the presumption that we can predict the future in terms of software components
is often quite inaccurate. We note
architect Louis Sullivan's decree that that functional use should take
precedence over aesthetics: we can build
beautiful design documents, have perfectly defined requirements, but in the end
if the software released does not generate value, then the effort was wasted.[iii]
This paints a grim picture indeed:
technological debt is actually created at the point that the system is designed, because it has not yet
created value.
This is
the premise of Agile product development:
if we design and release in short iterations, we can predict and realize
value at a much more rapid pace. In this
same manner, technological debt can be determined much sooner. The design becomes less relevant than the value that the system delivers. If we think in terms of value generated
rather than in terms of technical documentation, we can than assert good design
manifests as a well-valued system.
Retrospectives Enable Emergence, Emergent
Design Enables Low Debt
As Agilists,
we know that iterative, emergent development works; iterative, emergent design
is no different. One tool we can use to
enable this sort of emergent design, and keep our technological debt low, is
the Scrum Retrospective.
Retrospectives
are enablers of emergent behavior via the three pressures of innovation, selection,
and conservation. Biological life adapts
to its environment over time through these same three pressures: mutation creates new traits in an organism,
and its survival (or not) within its environment selects for the chance to
conserve its own traits by passing them on to its offspring (or not). Given
enough iterations of this process, characteristics emerge - in ways which no
human could imagine ahead of time - that enable the organism's perfect
adaptation to its environment.[iv]
Retrospectives
enable this same effect for the development team, their process, and their
product, with three traditional questions.
Asked as a matter of rote behavior with the answers then filed away
never to be seen again, these questions at best produce an improvement for one
Sprint, and at worst are little more than a chance for the Team to vent.
Applied
correctly, these simple questions evolve an Agile process that works perfectly
for the team and enables it to thrive amidst the complexity of its environment. By "correctly," we mean that the
Retrospective enables selection, conservation, and innovation not just for one
Sprint, but repeatedly across multiple "generations" of the Team:
- Ask "what went well." The ScrumMaster should ask the Team what
it needs to repeat such good effects; obtain formal licenses for the tools
that were used on a trial basis and worked well; share successful behavior
with other teams... in short, work to conserve
these good team traits.
- Ask "what didn't go well," as
a way to select out traits that
hurt the Team, and for each "didn't go well" item, be sure to ask:
- "What could we change to improve
this issue?" For each "didn't go
well" item, the Team innovates
to identify actions it can own, or that the organization must own. Team-owned changes either become Backlog
Items, or behaviors of which the ScrumMaster must keep the team aware. Organization-owned changes become Organizational
Impediments which the ScrumMaster must work to resolve.
We can incorporate
these same selective pressures when it comes to architectural decisions. The following design questions can be asked
as part of the usual Retrospective, or they can be conducted during a separate
Design Retrospective:
- Ask "what part of the design (code) delivered value?" The best designs are often those that
are birthed out of refactoring which, by its nature, implies design should
be iterative. These portions of the
code should be held up as "adding value."
- Ask "what part of the design (code) should not survive?" The Team targets those parts of the
system which made it difficult for team members to add value.
- For each item that made it
hard for the Team to work, ask "what
can we change in the design, or add to the design, to achieve the current
needs of the business within good design standards?" The Team applies its innovative powers
toward the problem of adding value beyond that of system features, and
generates work to do in future Sprints.
As for
output from these design questions, it should be up to the Team to determine if
having a Design Story or having a Design Task for each Story works better. Teams will differ on how they like to work.
The important part is not the mode of capture for the design, but how much
value in the end that the design has delivered.
The team
should also be allowed to choose how to apply the design questions at the
retrospective: all questions together or
a separate Design Retrospective.
However, to those teams who fear that not all team members are capable
of understanding architectural issues during, we point out that asking all six
questions together is an opportunity for the Team to help those who may not be
up to speed (see Figure 2).
Figure 2:
Evaluting Value and Debt Via Design Retrospectives
Note that
the six questions at the Sprint Retrospective, or three questions plus three
more at the Design Retrospective are not quite enough on their own to evolve
process and design. Some kind of
"heredity" or Team memory must be maintained.
Whether on a wiki or on the Team planning board, each of the Team
decisions must be kept visible. There
should be opportunity at the next Retrospective to review the changes that were
made and see if they helped, harmed, or had no effect. And the process repeats, Sprint after Sprint,
perfectly adapting the Team's process - and the system design - to its
environment.
The Architect as Technical Product
Owner
Agile
principles encourage us to deliver demonstrable product. If reducing technological debt is important,
the entire product must include not
just system features, but also the capability to produce those features
according to shifting business needs (see Figure 3). Briefly, this means considering "the product"
beyond the usual definition of the features of the software. Instead, the "entire product" must be viewed
as the composite of the software's features, the software's design (which helps
the Team make new features), and other capabilities.[v]
Figure 3:
Viewing the "Entire Product"
This view
implies a shift of roles for the Architect away from designing the overall
solution and leaving the coders to fill in the blanks. The Architect must think beyond assembling pieces
developed by "mere coders" into a unified whole. Nor should the Architect be content to use UML
diagrams, patterns, or high-level component design to constrain how the coders will work.
All these behaviors smack of Big Design Up-Front, and will lead to the
unhappy situation of technological debt being created before the software even
exists.
Rather,
the Architect should be thought of as the Technical Product Owner, with ongoing
responsibility for limiting technological debt to the business. Just as a Product Owner is responsible for
answering questions on what the system's capabilities should be, and sets
priorities based incremental estimates of ROI, the Technical Product Owner
should be responsible for specifying what the capabilities of the system design
should be, and setting priorities on work which will improve the Team's ability to add new features.
We have
actually seen a user story written in this spirit: "as Bob the Frustrated Operations Guy, I want
Engineering to be able to swap in different file output components, so that the
system can generate different file formats without waiting for a months-long software
development cycle."
A caveat
about this new Technical Product Owner role for the Architect: the ScrumMaster must pay attention to the
dynamic between the Product Owner and the Technical Product Owner. When we use Agile methods, the Team must move
as a unit and make decisions together.
Just as QA staff are the part of the Team focusing on quality, the
Architects are the part of the Team focusing on good design. But they are still part of the Team.
Two
failure modes to watch for are the Subordinate Technical Owner and Dueling
Product Owners. In the case of the
Subordinate Technical Owner, design work is continually deferred in order to
get "just one more feature" into the system.
We described the effects of this situation at the start of this article.
In Dueling
Product Owners, both the Product Owner and the Architect have equal ranking
power. This is good in that it ensures
that feature-oriented work and design work are both taken into consideration, but
something is wrong if the Team continually finds themselves facing a work
backlog containing "feature stories" and "design stories" of equal
priority. The Team may then either
prioritize these stories on their own (in effect, becoming another Product
Owner), or accept the risk of delivering both at once.
The need
for ongoing coordination between the Product Owner and Architect should be
readily apparent. Whether setting
explicit goals during Sprint Planning for "ability to react" and "eliminate
technological debt", or working to eliminate conflicts in the work backlog
before they reach the Team, both parties must maintain ongoing conversations.
If the
Product Owner, Architect, and Team all strive to be mindful of business value as
discussed here, the conversations will be fruitful ones. In all things: balance.
About the Authors
Derek W. Wade is a Partner and
Senior Consultant at 3Back LLC, a
CSM/CSP (Certified ScrumMaster / Certified Scrum Practitioner), and Agile team
coach. He has over 13 years experience across all roles of the software
lifecycle in several industries including Aerospace, Distribution, Internet
Commerce, Digital Mapping, and most recently Loan Financing and Employment
Testing. One of Derek's distinctions is bridging the gaps between developers
and business users, and his current focus is the "hard problems" of
the psycho-social aspects of team product development in corporate
cultures. He has led discussions on
software testing and transitioning to Agile, and is currently focused on tools
and behaviors which improve agile development for non-collocated teams.
Scott M. Barnes
is an Executive Vice President at Persis Consulting Company,
a CSM (Certified ScrumMaster) and mentor. He has more than 10 years management
experience, working in a myriad of market spaces. Bringing to bear over 13
years of technical expertise in various roles in industries such as Vehicle
Telematics, Location Based Services, Digital Mapping and Navigation, and
Enterprise Content Management, he has been a catalyst for change in business at
all levels. Currently focused on evangelizing Agile at the Executive level, he
has been successful at engaging businesses and encouraging growth based on
solid results. He has proven success in managing projects, consistently meeting
or exceeding expectations. Scott is working within the business community at
the C-level to bring to bear the power of Agile practices within businesses
that are ready for change.
[i] Shore,
James. "Voluntary
Technical Debt." http://jamesshore.com/Blog/CardMeeting/Voluntary-Technical-Debt.html.
Accessed February 4.
[ii] Foote, Brian. and Yoder, Joseph.
"Keep it Working." http://www.laputan.org/mud/mud.html#KeepItWorking.
Accessed February 11, 2008.
[iii]
"Louis Sullivan." http://en.wikipedia.org/wiki/Louis_Sullivan. Accessed March 4, 2008.
[iv] Darwin, Charles. "The Origin of
the Species." Gramercy, 1995.
[v] Rawsthorne, Dan. PhD. "Managing
the Work on an Agile Project." http://www.agilealliance.com/show/1433.
Accessed February 22, 2008.
|
