|
Original Broadcast Date : Wednesday, June 8, 2011 
Duration: 11:30 AM PT / 1:30 PM CT / 2:30 PM ET
Speakers:
Jeffrey Fredrick, Technical Evangelist, Urbancode, Inc.
Eric Minick, Lead Consultant, Urbancode, Inc.
In good economic times teams attempt bold experiments that promise to take them to new heights of productivity. In lean times like these few have the appetite for such speculative ventures. Any new investment needs to address an immediate pain and show immediate payback. But the pace of demands has not slowed, and the ever increasing need to do more continues. For build and release teams the demands come from all directions. The shorter cycle times of Agile means more builds, more deployments and more releases. The adoption of SOA means more complexity, more elements to juggle. The move to global round-the-clock 24-hr development means more teams to service and less downtime. And in this economy adding headcount is likely not an option, instead you’re told to get Lean and Mean. But how do you get to Lean and Mean without being stretched too thin?
This need to keep costs fixed while adding capacity makes improving efficiency key. Instead of attempting a whole scale change to how you develop software the focus is on reducing waste, allowing you to deliver more in the same time and for the same cost. Build and deployment automation offer fertile ground for dramatic productivity gains that will improve the efficiency of the entire team.
Join us in this webinar to learn how to develop and implement your own program for improving efficiency through build and deployment automation:
-
learn about the dramatic gains other organizations have achieved through build and deployment automation;
-
discover investigation tools for spotlighting inefficiency in the build and release cycle; and
-
hear about common build and deployment productivity blockers and the best practices for removing them.
About the Presenters:
Jeffrey Fredrick,
Technical Evangelist,
Urbancode, Inc.
An internationally recognized expert on Continuous Integration, Jeffrey Fredrick is an 19-year veteran of the software industry who has performed and managed virtually every role in the software development lifecycle. An early adopter of XP and Agile software development, the top committer for CruiseControl, and a 10-year veteran of leading Continuous Integration efforts, Jeffrey has consistently been at the forefront of the industry. He currently indulges his passion for improving how software is made as a Technical Evangelist at Urbancode, as the organizer of the Silicon Valley Agile Meetup, and as the co-organizer of the Continuous Integration and Testing Conference (CITCON).
Eric Minick,
Lead Consultant,
Urbancode, Inc.
Eric Minick is a lead consultant at Urbancode where he helps customers get the most out of their build and release processes with AnthillPro. He has 8 years of automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer. Eric has been at the forefront of continuous integration for 6+ years and has worked on all three generations of Urbancode's Anthill. Keep up with Eric's latest insights on the Urbancode blog.
Presentation Transcript
Good morning, good afternoon, or good evening, depending on where you're joining us from today, and welcome to this Agile Connect session, "Building Deployment Automation for the Lean Economy".
This web cast is brought to you by the Agile Journal and is sponsored by Urban Code.
I'm Patrick Egan, publisher for the Agile journal. And I'll be your moderator today for this webcast and other webcasts for Agile Connect. Well, in few moments, Jeffrey Fredrick and Eric Minick will join us to discuss How to Develop and Implement Your Own Program for improving efficiencies to Build and Deployment Automation.
But first, let me take care of a few housekeeping details. Our webcast environment is intended to be interactive, so you have the ability to ask a question at anytime by using the chat box on the right side of the screen. You also have the ability to chat with other viewers, and we'll answer your questions throughout the broadcast. We'll also address some questions at the end. If we don't get to your questions, don't worry follow up by email at the end of the session. We'll Also , I want to note that you can pop out the chat box, so it doesn't get hidden behind the slides, if you want to.
It'll allow you to move on the screen. Just use the little pop up button at the bottom of the screen. Also, if you want to follow along using Facebook, click on the Facebook tab on the box on the right side of the screen as well too. Also, if you'd like to share your experiences through Twitter, you can click on the tweet tab or use the Twitter icon that you can see above the presentation window.
Don't forget to use the hash tag #AgileConn11 to make sure that everyone else sees your tweets. We'll be broadcasting this tweet roll throughout all of Agile Connect and a live at 80P West in Las Vegas. Also, if you want to download a copy of today's presentation or any other presentation materials, click on the links button.
You'll have an option to download the presentation and click on the links for other materials that the speakers mentioned during the presentation as well, too. If you'd like to get a closer look at any slide. Just go ahead and mouse over the presentation window. You'll see a full screen button pop up.
Just click on that it will expand the screen to the full size of your browser window. And also, a recording of the entire live broadcast will be available within 24 hours, including the Q&A, so that you can listen to it again, or recommend it to a friend. Well, let me introduce our speakers today.
Jess Frederick is an internationally recognized expert continuous integration. He is a 19 year veteran with the software industry that performed and managed virtually every role in the software development life cycle. Also joining us is Erik Minick, a lead consultant at Urban Code, where he helps customers get the most out of their build and release process with Ant Hill Pro.
Eric has been at the forefront of continuous integration for over six years, and has worked on all three generations of Urbancodes and help products. Well if you guys are ready to go, Jeff and Eric the floor is yours, and I'll let you take it.
Alright, thanks a lot Patrick. And I thank everyone for being here today. Eric and I are really excited to talk about this topic. Just a little bit about our background and why we chose this topic. For myself, as Patrick mentioned in introduction, I'm the technical evangelist for Urban Code. I've been with Urbancode for about two years.
I've actually been doing Agile for quite a number of years, back before the term Agile actually existed. And that's what really got me into build automation and continuous integration. And I worked on the Open Source Project, Cruise Control. So I have actually quite a number of years both with Agile and its impact on development teams, as well as doing build and deployment automation infrastructure.
And I'll let Eric introduce himself.
Thanks Jeffrey, I'm Eric Minic and Really where I'm coming from, is, I spend a lot of my time, helping people implement new build and deployment practice. Get some automation in place. And I was really excited as I studied more on Lean and Agile to see how they fit together really nicely. And Lean gave us some tools for describing why things were doing so much better once we went Agile, once we automated.
So I really liked to address this topic.
Right . Now a little bit more about the topic. In our roles with Urban Code we end up talking a lot to build and release teams. And one thing we've noticed and a very common theme, as far as why people are talking to us, is because they're feeling an increasing amount of pressure.
And I think there's a few different places the pressure comes from. I mean, the first part is there 's an overall push to do more, and certainly Agile is a huge part of that. There's been just, if your development teams are going Agile, then one of the big signals for that is a faster pace of bills, and a faster pace of releases.
And I need to try to keep up with deployment pressure. And I'm sure you see this as well. Isn't it right Eric? Yeah, Agile does cause some pain for people who are downstream of the development team. So that's giving us a lot of pace and then we're also seeing those challenges from more component oriented architectures.
Lot of the smaller teams are driving smaller components. So whether it's a service oriented architecture or just another component orientated architecture, it's a good challenge. And then those teams are distributed, they're global, they'reinternational and that adds a lot of more pressure, I think you'd agree.
Yeah absolutely, another thing that's true is that, even for things that aren't Agile, I think that the expectations for what the norm is has changed dramatically. So were people might have had said before, "You know we're not Agile. We just release like once every eighteen months for two years." Now we often hear "No we're not Agile.
We only release like once every nine months." So, the expectation of what a slow release is, has been cut in half. So that you know, release frequencies kind of doubled even on normal projects. And yes, certainly the rise of the need to coordinate these different components and in increasing use of global distribution, all putting pressure on the build and release process.
And, of course this is still a time that we have not the best economy out there so there's not a lot of companies saying, "Oh well, you know, we're doubling the work, so let's double our head count." That's not something we're hearing, so from retrospective, the only real way to address this is, is looking into increase efficiency.
Take the people you have and work to get more done. And that's one of the reasons I like this stuff because we think there really is a solution to help do it. And that's what we're going to look at today, how to get to that in three major pieces.
So, we're going to break up this, our talk into these three parts. First of all this is an introduction into Lean Software Development and the concept of removing waste. From there we're going to move into then taking what we've learned about Lean and applying that view onto build and deployment automation.
Finally we're going to look at what happens as you step back and get this broader view, which is part of the Lean perspective as an end to end view and look at what happens when talk about what we would call Enterprise Continuous Integration.
So that's the topics we're going to cover and with that we'll jump into it with a description of Lean Software Development. You know, Lean is a topic that's, like Eric, I have a real affinity for. I've been doing Agile for a long time and I've seen Agile move through different stages, major, major, where part of the norm for what Agile was changed.
In the early days pretty much the dominant Agile methodology was XP. We're currently in an area where I'd say the dominant Agile methodology is Scrum. In the future, Lean software development will actually become the dominant Agile methodology, become provide the default language and the default viewpoint.
So I think it's important to understand Lean software development. In seeking to do that, I think the first thing to understand is that it's pretty much a very direct mapping from Lean Manufacturing. Lean manufacturing is there, in the roots of Agile, it inspired a lot of the XP practices, all the other Agile practices and Agile mindsets, but Lean takes it a step further by being a much more direct mapping.
The other thing that is really important about Lean to understand is one thing that separates it from some of the other agile methodologies such as XP or Scrum, is this focus that it has on principles rather than practices. And what I mean by that is, if you go read about XP, you'll see there's maybe "The 13 Practices of XP." And you should do all those 13, if you're doing all of them, then you are fully XP, and if you're not, then you're not.
Similarly, Scrum is defined by a set of practices that you do, and if you're not following them, then you say "well, we are doing Scrum but," the scrum, but instead of actual Scrum. Lean's a little bit different, its that rather than trying to tell you change all your practices, "adopt this externally defined set of practices" instead it says "adopt these principles" and Look at what you're doing through the lens of these principles.
Eric, do you want to expand a little bit more on that?
Yeah, and I think it's really why a Lean approach can be compatible even with something like SCRUM. Because you can take something like SCRUM and apply Lean principles to it. And use that to guide you as you evolve your practices. Because most of the Agile approaches do allow you to evolve. I think that's all I really wanted to touch on there.
It's just that you can combine the typical Agile practice with Lean principles in time and end way, it 's not one or the other.
That's a great point and certainly you can even apply those principles into a non-Lean environments as well, so it gives you a good starting off point, because you're not being asked to adopt a whole self set of practices. What it means is you end up with a very low risk approachto improving your efficiency.Rather than a disruptive, high-risk approach. So I think these allreasons why lean is a really good topic for people to be looking at when they're in a sort of resource constrained setting where they need to improve their efficiency in a low risk fashion.
And so we talk about increasing he real focus, and we saw this on the list of lean principles. The very first one was remove waste. And that's really the key in Lean of increasing efficiency, this focus on waste, so much so that one of the key concepts where defining lists if your will in Lean is this idea of the waste of his is also true in manufacturing as well and then you can see here we have a direct translation from what we listed as the seven ways of lean manufacturing into the seven ways of software development and these lists come from the Poppendieck's book on Lean Software Development Agile tool kit we'll actually give that reference later in the end.
That's where we pulled this from and you can see in some cases the exact same topic, such as extra processing, extra processes, I mean you can see this very direct translation.
of the Lean concepts. So, removing waste is really the key of motivation of Lean, the key principle by which you improve your efficiency. So because of that, Lean actually give us some tools to go about identifying waste, so that we can then remove it. Eric, do you want to introduce these tools for us?
Yeah. There are a number of tools that Lean provides. We're looking at two - spaghetti diagramming, and the classic spaghetti diagram, when we're looking at manufacturing is you'd lay down a floor plan of your factory and then you would put your pencil down on there where widget starts being processed and then you keep your pencil down, follow the whole path and often what you would find is that the widget is kinda taking some crazy steps, there is a lot moving from here to there.
And you end up with a drawing of spaghetti on top of your floor plan. And the goal is to get things nice and clean where you have a lot less extra motion. We 'll show how to apply that into human processes and in the software. And then the other concept is value stream mapping. We Which is really just trying to show us, when we look at the various tasks in a process, where we're spending time doing work, and where things are just waiting.
Because waiting is inefficient, so that we are trying to identify those work-wait ratios and things like that. We're going to do a very simplified view of value stream mapping as we go.
Yeah. So these tools that Eric's introduced, these are the tools that we are going to use to analyze the impact of build & deployment automation. But before we get into the build & deployment automation examples, which are a little bit more complex, we just want to illustrate the use of these tools with a very simple process.
This is kind of our touring case for these. And this is the example of a sign- off process that you might have. It's something that you're probably are familiar with.
But you have a project manager and he'd need to get spin off from the development manager, the QA manager, and operations, this release is ready to go, and basically he's just going to do some linear in serial fashion. He's going to email first the dev manager, email the PM, he emails back, email the QA manager, he emails back, and so on.
If we go ahead and look at this process through the lens of a spaghetti diagram, what we see is this picture up here. We have the three managers with the project manager acting as a hub. And the flow through this always goes...starts at the project manager and goes back to the project manager before it moves on to the next person.
So, pretty simple, pretty straight forward. If we didn't put that into a value stream map, what we see here now is we are going to start tracking the time that it takes to do this. Now, assume that each of the emails are straightforward and each will only take a minute. The most of the time that was actually spent sort of waiting .
Someone sends an email, then the question is what's the time it takes to someone actually read it and decide to reply. Just for the use of...for example here, we're saying on average it's 30 minutes before the manager actually texts, or, you know, Blackberry or something, and sends a reply. And you can see here we end up with a ratio of waiting and working that's actually pretty terrible, right?
It takes us 30 times as long in weeding as it does to working. So, if you look at our overall value stream, for every 156 minutes, we have six of those minutes working and 150 waiting. So that's a terrible ratio, and we want to try to address that.
So we can imagine how to do in some simple way - we are going to say, rather than in this flower pattern where everything comes back to the PM as the hub, let's have our managers simply go to the next managers. So, the PM will ask the Dev manager and the Dev manager says "yup, it's good with me. Pass it on to Q A".
QA's done, pass it on directly off to Ops and then finally all the sign offs are done, then we go back to the PM and, you know, voila, we've managed to pull out a couple of those steps. We've gotten our process time down, actually quite a bit from this simple change.
So, that's our really simple example. You can see the two tools we have are our spaghetti diagram to identity the wasted motion. We have the value stream map, which identifies extra waste in waiting. Other kinds of waste, but these are really good weapons. We're going to be the primary focus because it's such an easy win for so many people.
So, that's kind of our quick summary of Lean software development, as far as we're going to be applying it today. You know, if we were to give you an exam at the end of this, these would be the four that were going to be on the final.
Number one, Lean soft development. Hey, it's inspired by Lean Manufacturing. Pretty much a direct translation. Two, there is a really strong emphasis on removing waste. Three, that there is a list that identifies and categorizes the seven wastes of Lean software development. And finally, number 4, that there are these tools that Lean provides for actually visualizing the ways to help you identify them.
So, with that, we 're ready to move on to our second section which is now applying these tools into the build & deployment arena and we're going start off with this common scenario we see of having the designated build person that's right, Bob, the builder. Eric, do you want to explain, you know, Bob's role and how often we see him.
Sure thing. Yeah, and we see Bob a lot, well a little bit less and less these days. But basically, Bob's the build guy he's the one who knows how to do the build. He's the one with access to the build machine. These probably the one who wrote the build scripts and pretty much he is in a service role.
So you will be in a situation where a programmer, like Pam will make a bunch of changes, in source control, and say you know what, I want to get it built with what I just did. So, email Bob, say hey can I get a build of this and that, look at this tag in the source control. Bob will go ahead and get that email, perform the build, and then when it's all done, email Pam back andsay, "your build is done. You can get it here. Hurray, life is good." I think this a pattern that we see often, but it'snot necessarily the most efficient way of doing things.
Yeah, it's not but, you know, one thing, Eric, I think that's worth pointing out is.that 's not efficient. People have a good reason to do it, which is they say, "Look, we want to have a controlled build process. We don't want just any random build from developers' desktops." And so...
Right .
...in seeking control, people are looking to Bob as the solution. Isn't that right?
Yeah. And because Bob is good and Bob is disciplined, it's a lot easier for Bob to do things the same way every time than to have a whole group of programmers who are all going to follow the same set of procedures. It's a lot more likely if Bob does it. And hopefully that takes a lot of load off the developers, right?
They don't have to worry about doing all these builds. So, in the ideal case, it's a win win.
Yeah. So there's good motivation. And second is there are cases where someone will say, "Look, we don't have a Bob. You know, we don't do that. We don't have that inefficiency. We just make sure our tech leads are the people who supply the build and they just do it off their desktop."
You know, guess what? If it's your tech lead who's doing it, your tech lead is acting as Bob. Right? You know, you may call him Ted, 'cause he's the tech lead, but he's still acting as Bob. If ever the programmer needs to go go through them to get the build. So, that's the two reasons. Understanding one is that people do this for good reasons.
Then second is that even if you don't think you're doing it, you might actually be using this pattern because you have a different person acting as the gatekeeper. It's not about the role it's about the person who is acting in this role as the bottleneck, which I think we can see here if we kind of diagram it out. So this is the spaghetti diagram of the Bob-the-builder pattern. It's pretty simple, it's actually, you know, arguably simpler than the example we had earlier with the sign off process.
Basically, you know, if Pam wants to build, she has to go through our interface our human interface to the build system and when their person is done, they can say, "Yep, your build is ready to go." If we go look at this on a value stream map, we start to see the inefficiency this doesn't look inefficient this looks, you know, straight forward.
n the best case, this actually feels really pretty efficient, right look, I mean, if you look at the ratio, you know, Pam emails Bob, he gets right to it, he runs the build, he's working, and, you know, fifteen or twenty minutes later we have a build. Great! There's not really a lot of a lot of inefficiency.
Eric, you seem to think that there was, so why don't you explain what goes wrong here?
Exactly. So we don't really believe that Bob's going to read his email the moment we send it to him. Because, he's only able to do that if he's just staring at his computer hoping an email will come in, so that he can perform a build because he's that bored.
And in this economy, no one is that bored and if they are, they're really working hard to pretend they're not. What 's really going to happen is Bob's going be at lunch. Bob's going be in the meeting. Jeff, you and I have both been in the meeting when Bob's Blackberry went and he said "Yeah, going to have to do that when we're done with this meeting." Pam did not get her build right then.
She got her build when the meeting was over when Bob got back from the lunch, when all the more important builds were done first. So really, this whole idea that we can send it and it'll happen immediately just isn't true anymore. Yeah, and, your just looking here in terms of the email, but obviously this happens at every step of the way, right?
Every time where Bob does an action, and then he needs to wait, so he's going to start a build and then he's not going to sit there and stare at the console for ten minutes. He's going to be up surfing the web, he's going to go get a cup of coffee, he's going to strike up a conversation, and so on.
Every single time that there's some action he needs to take a potential point of interruption. And as you say, we know about these things first at hand because we know the interruption. We'll come and say, "Hey Bob, time for the meeting we have." He goes, "Oh right, I guess I'll get back to this build when I'm done." So the fact that you would have a very efficient process here.
It very often doesn't become the case because Bob's availability really makes him the bottleneck. And as we say, Bob's busy and Bob's got a lot going on. That was our whole starting premise here, is that given the effects of Agile, given the effects of build complexity, given the impact student build teams.
There's more and more work to do for these build people. They're more and more busy and we've seen it all the time. So, this fantasy that you're going to have a very efficient process with Bob as the hub isn't what works out in practice.
You brought up a good point. In that, with more and more work, more and more of a bottleneck and the time it'll take will be slower and slower.
So, when you need, you're build the most because in the most critical time is when this process is slowest.
Yeah, that's a great point. I know we've talked to people who, in doing their into their system go live and in the process they need to, they have some things that fail in validation staging, they need to set in bunch of emergency builds and suddenly we have all these builds trying to go through the same bottleneck at the same time and you know, it just makes that deployment process stretch out at the worst possible moment. So that's true.That unknown impact on waiting is what makes this so inefficient. So clearly we say, "Look, the solution here is we go ahead and use build automation to break the bottleneck." We're gonna allow Pam to come in now and directly trigger their build, and she can self service herself and get a build back immediately.
Now, I had said before that Bob is here as a control was to make sure that, you know, that, they were getting the official build and there wasn't sort ofrandom stuff happening to the developer desktop I make it clear here that people might look at this and go, "Well, what happened to separation of duties in this case?" But, I think, to agree that the key idea is that while Pam can trigger the official build process, she can't necessarily change the official build process.
Isn't where the build machine just runs it and that magic was configured by Bob in all likelihood. So, he's responsible for setting everything up, and then, you know, he'll just let Pam trigger it. Because he's never saying no to Pam he's just saying, "I'll do it later" you might as well just give her a button that she can press and the build will run.
It'll be logged any parameters that she passed in or anything like that.
Yeah, and I think that's a great point, is that people, especially if you're in the role of Bob, you might look at this picture and go wait a minute, what happened, you know, am I fired?
I think that's really the key idea is that steps have been set up by Bob. The invisible of hand of Bob is actually in this picture, enabling this you know, so Bob has better things to do than act as the machine to be waiting for Pam's request, he is enabling this kind of system and keeping it healthy and expanding over time.
So, the picture doesn't look that different. But it actually can have a pretty big impact on the build value string we take out a couple of those steps where we have the whole email back and forth process.
And if you look at the numbers, though, if you get rid of the best case, it doesn't seem that different right, we look at; before it was16 minutes of working hours saying it's 3 and the waiting before it was 2 and now it's 1. The real win here, though, isn't that we've reduced the totalamount of time, but that we've reduced the uncertainty right, we've now improved the predictability. And I think you have a good point here Eric for about people who are in six sigma kind of organizations.
Or you know, some sort of CMI process improvement. Having that predictability can be pretty important, isn't that right?
Yeah, to just know that when I press this button, in 15 minutes I'm going to have a build back. That's what Pam has now and to some extent, if she had that, even if it took on average as long as it took Bob to do it, even if it took an hour, that would be okay.
The really good news is that now we have made the normal be the same as the previous best case which is fantastic. We are actually faster, particularly when looking at wait time. But just the knowledge is really good. Pam knows she can grab a coffee, when she comes back, the build will be there, and she can go do that.
As opposed to, I might have it done in ten minutes, and I might have it done later this afternoon. Just a huge improvement. Yeah, they could do psychological ones, like you say, of their reliability...is a real confidence builder for the team and actually changes their behavior. And that's one I've seen time and again.
You get people who say things like, "Oh, just a second. I'm waiting for the build to finish and then I'll go to lunch," as opposed to, "Well, I guess I'll go to lunch now because who knows when I'm actually going to get the response back."
You know, in the...we'll give you in a real world example of people who have gone through this transition from, you know, a Bob driven system to a automated system. Eric, I'll let you talk this because you're pretty familiar with this client.
Sure. We had a team of basically four Bobs who were doing the builds and some deployments, mostly the builds for sixteen different projects. But then they were told, "No, we're doing some consolidation. You guys are going to have ten times as much work to do. And by the way, you don't get tired anymore.
So, figure it out." So that was a little bit challenging. So they introduced this automation so there would be some self service or best, they could just press a button and it would happen. and they were able to handle all those projects. They were able, you know, to do ten times as much work; and their big fear was that they weren't going to be as responsive but there were actually able to be more responsive you know, the average cycle time went from 90 minutes to 20 minutes.
That's fantastic! And they just are getting more and more work. The business loves them, they're getting more responsibility, they have a higher profile, they're given forming fifty projects. I think the latest numbers are higher than that now. Yep, it's very, very doable.
Yeah, and I think that's a great example of someone who's taking adversity and turning it into a real opportunity to change the game. You know, you still - no, not just like you're going to be a little bit more work. But you need ten times as much work that you're going to be servicing. Now, the idea of going in hiring, growing to forty people - clearly not that attractive.
And by maybe automation these people again were not gone, they were simply much more valuable, much more leveraged people. I've got to think that not only are they providing better service, but they all probably enjoy their job more as well, because they're not sitting there acting as robots, you know, trying to run builds manually anymore.
They get to actually do, you know, fun stuff building the system and if you like heroes for bringing in all these projects, and providing service to them. So, I love this scenario of taking what was a real challenging environment and scenario, and turning it around into a real win for the the company and themselves personally. That is sort of the build story, we're gonna go extend that out now into the deployment scenario and you know, keeping with our creative naming stories, of course, we gonna do deployment, it's gonna be Dan doing it and this should be deja vu.
Do you wanna do the very, you know, quick run through this, Eric?
Yeah, pretty much same story, Pam got her build back. Now she wants to deploy it in the test environment. She's going to ask Dan to do it. Dan will do it eventually, e-mail her back when it's done. So, pretty much the same story, and I think we'll see a very similar diagram here as well.
Yeah, you know, diagram real simple, you know, can we. a person acting as an interface to the machines. But you know, if it's a expect a person to go look at the value stream map. We are this thing is pretty similar you know, the big question is, with any human gated process, when are they gonna available to actually service my request and the reality is, everyone's busy.
We just don't find that many people out there these days, but tons of times sure come in, meet with us anytime because, you know, we have nothing to be doing, those people just aren't around much anymore. So, we're going to make the same kind of change here. We're going go ahead and remove the human element. Again, notentirely. They're justnot in the picture, but that invisible hand of Dan is still there, making sure that deployment automation is running smoothly.
I say it more deployment automation servicing board teams you know, going from 16 teams to 400. That's what's going on here. Then again, it's the consistencyis the real win. Although, you know, sometimes that removing those bottlenecks can be a systemic win especially in cases like this one. I'm going to go ahead handle this story Eric, cause this is one of my favorites.
I know it's your favorite.
It absolutely is! I love this story. You know we have this company we work with. They had an offshore testing team. The offshore testing team were contractors. And so, they were not allowed to log into the systems in the test environment directly. They couldn't log in and actually do the deployments, and as a result, they had to go and send emails to the on shore teams and hey, please deploy the new build whenever they were ready to take the new one.
Obvious problem for anyone who has worked with offshore, you know, we're talking about a minimum 24 hour turnaround here. So, actually the average turnaround time was about 20 hours. That was, that was kind of the norm for them. Whenever they asked for a new build, 20 hours later they hoped to have one.
Sometimes it was worse than this, right? Sometimes the build and the deployment didn't work out for some reason because of bugs in the buildup as there is a deployment in the stake and then suddenly you're talking about two days to get a new build. So that has a huge impact on the productivity of those testers.
What they are able to do, though, is to go and say great, we're gonna have deployment automation so they just push a button when they need a new build. So, rather than waiting for someone to come in, they can do it manually. They can just go, you know, trigger it and make their request. And suddenly they went from twenty hours to one hour.
Right? So a savings of let's say 19 hours for the entire offshore team. Right? So you're talking about 10 people here, so you're talking about 190 man hours saved. Fantastic savings. Just from adding the ability to push a button to get that done. I love that story because it's something a lot of people can benefit from, and that's such a clear case where you see the impact of delays.
Anything else you want to add to that, Eric? Or are you ready to move on?
Yeah, the only thing I'd add to the deployment narrative here, is that you're gonna do this not just to one test environment, you know. Any inefficiency you have in your deployment process is going to be repeated through each testing environment you have. Now, I think we've seen that the average is four or five different environments that a particular build will go through on its way out to release.
Oh, that's a great point. Yeah, it's not just like the one test environment, but there's performance testing and there's staging and there's functional test and system test. We have one customer, some customers that have like, you know, over ten environments, that they deploy to, right?
Absolutely.
Yeah, that's a great point, that these savings are just not a one time savings but you're saving in every step along the way. So we've kind of done now, I think we've slowly set the baseline. People should understand now the concept of the lean folks that are eliminating wastes. We've now applied that to build and deploy automation.
I think it's really the straightforward case; everyone can go and actually start with those scenarios. But, let's go ahead and move and say to let's to give that sort of whole end-to-end perspective and look at this from an enterprise, continuous integration or enterprise CI perspective. I think, to start out though, let's clarify what we mean by enterprise continuous integration.
The term continuous integration actually goes back to extreme programming. When their continuous integration, was not actually, had nothing to do with automation, it was really a manual process. They are trying to avoid integration help. So, the question was how often do we commit out code you know, are we all working on the same code all the time or are we diverging and have to integrate later and feel that pain.
So, that's really where things started when people realized that if they're checking it all the time it was nice to have this build triggered automatically, so this idea of automated integration came about idea even having someone have a button to trigger the build automation but they could actually just have the system run it automatically.
We give feedback to the team if anything happens. So, we have these two practices of living frequently and getting rapid feedback, both with the term CI. Most the time is now when people talk about continuous integration, they are usually talking about a CI system, and they're talking about this sort of automated feedback. And it's really from that perspective we're seeing. Take thatautomated, continous integration approach, but extend it beyond just the build, beyond just a test, beyond feedback to developers out into the full life cycle.
Now in, to kind of, clarify this and expand on this idea, Eric and I wrote a white paper, you can get it from the download section, from our website, on the enterprise CI maturity model. And if you, you squid really really close here at these words at the screen or you can just get white paper and the associated wall chart.
And we can kind of lay out both were we thinking the average person in the industry is, where the average team in the industry is and also we identified targets that you might want to shoot for, along the way.
and one of those targets is actually in deployment and what we think people should be aiming for is continuous deployment which brings together two pieces that the continuous generation build as well as automatic deployment to the first test environment.
You want to kind of explain this, the terminology and where this fits in the world of continuous deployment?
Sure. So, our scenario here is the developer commits a change that triggers a build. When the build's done, it's moved off to some sort of authoritative storage. And from there, deployment is automatically run in the test environment. When all that's done, we let the programmer know, the developer know what's going on.
The term continuous deployment itself is a pretty loaded term. Some people are using this to discuss, you know, a series of tests and deployments that carry all the way out through production just from a developer commit. And there are some variants on that like continuous delivery, which go almost all that way.
For our purposes here, we're just looking at that first test environment, but these same concepts drive all the way out towards production. And some of those are really driven by lean. Teams that have tried to get very, very lean don't want to have good code changes sitting on the shelf, not delivered to customers.
They're trying to figure outand how to get that to production as quickly as possible.
Right. You know, this idea of continuous deployment, and you mentioned going to the first test environment. What would you say to someone who says, 'you know, that sounds great, but I'm a tester and I don't want my test environment changed out from underneath me. You know, how would you address that concern in continuous deployment environment?
I mean, if the test cycle is interrupted by deployment then you don't want to be doing that all the time. And so you might be doing, you know, once or twice a day automated deployment like this on some sort of a schedule. Or that first test environment could blow under the developers just they can see their software running and they can validate it's doing what they think it should be doing before it goes off to the testers.
Yeah, and I really like that pattern that you just described of you have a test environment that's not the same as the QA environment. So you know the QA people are in a long running test cycle, you don't wanna interrupt them. We certainly agree with that, but the idea of - look, you can have this deployed automatically somewhere else so that anyone can goes see the current state of the build The current state of what's checked in at any given point time, that's hugely valuable, and, you know, rather than actually waiting for the test environment to be updated, you know, the QA environment to be updated, that anyone can go and see yeah, this is the current build.
And you might even use that to say, well, lets, you know, let's go ahead and see if that's something we're ready to take into our official test environment. But you have the ability to get feedback much faster without waiting for QA, so I like that. And we know that even just a simple pattern can actually, you know, bring this dividends and so, you know, we are gonna talk through this customer scenario of continuous deployment, Eric?
Yeah absolutely. So, these guys really couldn't test their software without getting it in some sort of environment. So the challenge was, how do we get it out into that first test environment quickly. They had kind of the Bob the Builder, they had Dan the Deployer, they had those people. They were frustrated by the inconsistency some time six sigma shop.
But they knew that on average it took about two hours to get all those emails done. With this sort of thing in place, where they commit a change and it's just sent out to the first test environment as quickly as possible, they're able to drop that cycle time to seven minutes. And it was really six minutes to eight minutes as opposed to twenty minutes to four hours, again that consistency in addition to speed.
It was really nice for these guys.
Yep, so, you know, huge wins there, and these people who are real happy to no longer be in the, you know, Bob and Dan Button Pusher. So, you know, we mentioned before the seven wastes of software development. We are gonna run through a scenario here, now that we have this whole Enterprise CI system, putting all the pieces together, and we are going to say how we address five of the seven pieces of - seven wastes of software development.
So, this is the scenario we're going to look at And basically what we have here is our programmer Pam is going to commit a new feature. That feature needs to be built, it needs to be deployed, it needs to be tested to verify that that's working. But the tester is gonna find a defect, and we're gonna go back to that cycle again.
So let's start by looking at the spaghetti diagram. Eric, do you wanna step us through each of these, step by step?
Yeah, it's a lot of fun. It was a lot of fun to put together the example. This is where we actually see spaghetti take its meaning. We're able to look at this and just go ugh: It doesn't feel good as opposed to some of the other ones where it doesn't look so bad. When you bring in the cycle where we go through it twice, this really doesn't look good and I think you can spot where's some problems.
And we can look at this and some of other tooling to really break down what we don't like.
Yeah, and I think you especially appreciate that Eric because you did all the heavy Visio work here. So, you did a lot of work to make this look as simple as possible and still looks like a pile of spaghetti. If you go apply this now to the value stream map and we bring in this, the normal kind of things people who do be a nightly build.
Maybe they only deploy it to the QA environment twice a week. That's pretty typical. You know, people take - maybe two builds a week out to QA. Then you're going to have the time spent actually finding the bug, reporting it. It takes a while for it to be triaged actually and then back to the developer.
And then it gets fixed.
We're talking about something that takes about on average about eight and a half days for a shop that's doing all the stuff manually, to actually be able to run through the cycle.
I talk about this with lots and lots of people and I get a lot of confirmation that people go, "Yeah, that's about right, that's about how long it will take us to to work through the cycle."
So what we want to do now is show you what it can look like if we go and take out, you know, address with different parts of automation. First off, we're doing that continuous deployment, then both times that we build and deploy, we're gonna be, you know, attacking those waiting elements. If we have automated testing, if we have unit tests, if we have, you know, static analysis, if we have automated functional tests and so on that can find the defect then guess what?
We can go in and automatically find the defect and let Pam right away without that delay. And if we can then let Pam know right away, right after she actually made her change. We don't have to go through the whole triage cycle, right? Because we don't say, "Well, where'd this bug come from? We say, "Well, look, Pam made a change and Kurt, you know, triggered this test failure, can he go fix it?" So that's going to be addressed.
So we're really identifying each of these different elements that with these long delays, and attacking them through automation. Lucky take Pam less time to fix it, because she's still doing that work. She remembers the changes she just made, as opposed to finding out three of four days later. So, this is our revised view of the world.
We don't have a Bob and Dan anymore, we're going through the cycle. We find a defect, we only have to go through and test it once. I am guessing this is little bit work on to draw, isn't that right, Eric?
It was, and I'd like to point out all that messiness, over on the right, that's only automation. So that's the stuff that was configured once by Bob and Dan and now just happens the same way every time. The simple stuff that is over with the people.
Yeah, that's a great point. You know, people look us and say, "Wow, there's still that complexity." And we go, "Yeah, it's machines doing it, not people," so you don't have the delays, you don't have the possibility for errors and now you know what Bob and Dan are doing when they're not in this picture. They can actually make this kind system work and actually, expand it, make it, you know, put more features in it, so over time it gets better and better.
As nice as this picture is, the real impact I think you see is in the value stream map.
Huge, huge reduction in time here, right? You know, because we're not waiting around for things to be deployed. We're not waiting around for, you know, getting the feedback on the bug. You know, we have this huge impact, where we've sort of taken eight and a half days to go through the cycle, we're talking about making just that full cycle in about a day.
Most of that you'll see is actually waiting for Tom, the tester. Because we're assuming that you didn't, you know wire this directly into the manual test environment so Tom is still gonna wait for his half a day to go by for him to get the new build and actually go though them get around to actually doing his work cycle.
That's where actually most of the time is spent. If you can do things to make that more efficient, you you'll drop down this time any more, I think when we look at what, you know, where the savings come from, you know, we can kind of look in, go back to that list of wastes. Eric, you want to kind of walk us through how we've address the different pieces of "the wastes of software development?"
Sure . So we haven't touch the partially done work or the extra features; that's still on the team, we've really, you know, limited the extra processes. There's probably not as big of a triage meeting. There isn't lot of discussion. We know who's to blame. You know, because Pam is notified so quickly of the problem, trying to get back and remembering what she was working on lowers the expensive task switching.
Obviously that waiting and the motion, that's what you saw with our spaghetti diagrams and value stream--and then on the defect side, because that first bug was caught by the automated test, Tom wasn't doing nothing that time, he was testing something else. He's finding some other problem. And so we are going to see higher quality result.
I want to expand on the impact, the defects a bit more, right. And I'm gonna use an example developed by Michael Bolton - not the singer but the Agile testing consultant. He has this great example and he says, you know, imagine that testing a feature takes two minutes. Right? But that when you find a bug, it takes ten minutes to go investigate and report it, because you need to do all the set up and capturing all the details.
So, in a 90 minute session, you can run through 45 features but only if you don't find any defects, right? And so, if you start finding defects, suddenly you have this big impact in your scheduled testing session. And so you kinda say this is just for the tester to get through their tests. For instance, if you find 9 bugs, if there's one bug in every 5 features, then you are actually doubling the testing time to get through that.
So anything you can do to pull these defects out and catch them earlier is going to have a huge impact. And that's only just looking at impact on the tester. There's also that impact on the engineer time to investigate, the time to retest, and of course, then you have those bugs that are hidden behind other bugs.
The ones that you find later because you couldn't get to them because the bug was in the way. There's all kinds of other costs. But even these have a big impact.
Putting them all together, you end up with the overall impact and this is from I triple E survey saying that, you know, if you do builds at least daily and you run integration test so you can find defects sooner. You'll end up with a huge impact on both productivity, you are much more productive, and you end up putting much better defect rate.
So, much higher quality.
I think that really shows you the impact of that. With that, we've kinda gone through our different topics. One thing I wanna say is people always ask us, hey, you know, I know you have these tools, can we use them to do this? But I wanted, I don't know if we're gonna have time for many questions, but I wanted to at least get this one out because people always ask this question.
We essentially have two products, we have Anthill Pro which will automate the end to end cycle including all the, you know, builds and new test that sort of development, automation infrastructure and we also have Urban Deploy which will help you if you just wanna deploy in a reliably consistent fashion across all your environments, so I just wanna get out there.
And we also mentioned the different resources, one book we mentioned was the Poppendieck's Lean Software Development: An Agile Toolkit. There are some resources on our website and also available to download on the online site. Check out our blog and we kept talking about bottlenecks and if you wanna learn more about the importance of bottlenecks in software development, go read The Goal, which is about lean manufacturing but you'll see how it pretty much directly relates to bottlenecks in software development as well.
And with that, if anyone has any questions we don't get to today, feel free to email Eric and I, we love talking about this stuff. With that, we'll open up if we have time for maybe one question.
I think we have time for a quick one. Great presentation as usual, Jeff and Eric. And again, if you have any questions, go ahead and use that "ask a question" box on the right-hand side of the screen and We're gonna jump over real quickly to the next session in Agile Connect and I know that both Jeffrey and Eric will stay online in the chat session for a little while, so feel free to keep chatting away.
But, here's the big question for you, I'm talking about Bob and Dan. Is there any value in having a combined Bob and Dan? Someone who handles both build and deployment and acts as a single point of contact, someone like a configuration manager who controls whats in all environments as well as what's in source control.
I know a lot of people attempt to use that pattern but the problem is very often that the same kind of knowledge doesn't translate well across all these. You can put one person in place as a gatekeeper but they can't really always facilitate the problems. So, for example if you're doing your JTune deployments, the person who needs the web secure administration knowledge to, you know, deploy a component, is not the same person that has the knowledge about your branching strategies in ClearCase.
You know, or if this person has the knowledge in doing your Ant scripting. So, while there can be value in a single point of contact, I think it's hard to certainly combine all the knowledge in a single person, It's different in some cases. So it would be exception of the rule. Eric, anything to add to that?
I think the only other concern is around separation of duties. But, that'll be an organization to organizations so concern.
Alright well thanks you guys. Again great presentation and all of the resources that book, Eric and Jeff mentioned, you can find it over the links tab there. A lot of good things that you guys were talking about earlier especially like the idea of Bob the Builder, maybe we should replace those guys with Bill and Ted instead.
But, I look forward to seeing you all in our next webcast and thanks to everyone who helped put this presentation on today. Bye bye everyone.
Trackback(0)
Comments 
Write comment
 |
TrackBack URI for this entry
Show/hide comment form