|
Broadcast Date: Thursday July 14, 2011  
Time: 10:00 AM PT / 12:00 PM CT / 1:00 PM ET
Duration: One hour
Speakers:
Bob Aiello, Editor-in-Chief, CM Crossroads
Leslie Sachs, COO, CM Best Practices Consulting and coauthor of "Configuration Management Best Practices: Practical Methods that Work in the Real World"
Francesca Matteu, Moderator
Implementing Application Release Automation to support IT Governance and Compliance can be very challenging and, in some instances even impossible, without robust tools and processes. There is a lot more to helping your IT organization establish ITSM and ITIL than just setting up a change management database. This Webinar will discuss release and deployment best practices that can help your IT organization handle the demands of rapid application deployment across all of the environments required by the application lifecycle (e.g. Dev, QA, Staging, Production). You will also learn how to assess and manage deployment-related risks that many managers find especially challenging to handle. Implementing a robust deployment framework may be the most important step that your organization will take to improve productivity and quality. This Webinar will get you started on the road to successful automated application release and deployment best practices!
About the Presenters:
Bob Aiello,
Editor-in-Chief,
CM Crossroads
Bob Aiello is a Consultant, Editor-in-Chief for CM Crossroads and the author of CM Best Practices: Practical Methods that Work in the Real World, Addison-Wesley Professional (http://cmbestpractices.com). Mr. Aiello has over 25 years experience as a technical manager in several top NYC Financial Services firms where he had company-wide responsibility for CM, often providing hands-on technical support for enterprise Source Code Management tools, SOX/Cobit compliance, build engineering, continuous integration and automated application deployment. Bob has served as the Vice Chair of the IEEE 828 Standards working group (CM Planning) and is a member of the IEEE Software and Systems Engineering Standards Committee (S2ESC) Management Board. Mr. Aiello holds a Masters in Industrial Psychology from NYU and a B.S. in Computer Science and Math from Hofstra University. You may contact Mr. Aiello at bob.aiello@ieee.org, link with him at http://www.linkedin.com/in/bobaiello or visit his corporate website http://yellowspiderinc.com
Leslie Sachs
Leslie Sachs is a New York State Certified School Psychologist and the COO of Yellow Spider, Inc. (http://yellowspiderinc.com ). Leslie is the coauthor of CM Best Practices: Practical Methods that Work in the Real World, Addison-Wesley Professional (http://cmbestpractices.com ). Ms. Sachs has over 20 years of experience in the psychology field and has worked in a variety of clinical and business settings where she has provided many effective interventions designed to improve the social and educational functioning of both individuals and groups. She may be reached at LeslieASachs@gmail.com or link with her http://www.linkedin.com/in Lesliesachs.
About the Sponsor:
Nolio the leader in application service automation, is committed to helping customers meet the growing challenges and complexities of releasing and managing applications across the data center - increasing application uptime and reducing IT operation costs. Nolio customers including enterprises, online services and SaaS businesses rely on Nolio ASAP to automate application deployment, maintenance, remediation and recovery processes for physical, virtual and cloud infrastructures - delivering automation solutions for application-centric IT. Learn more.
Video Transcript
Good morning, good afternoon, or good evening, depending on where you're joining us from today. Welcome to this Agile business web cast series broadcast, utilizing release automation to support IT governance and compliance. This web cast is brought to you by CNC Media and is sponsored by Nolio. I'm Francesca Matteu and I'll be your moderator today for this web cast.
In a few moments, Bob Aiello and Leslie Sachs will join us to discuss Release Automation in IT Governance and Compliance. But first, a few housekeeping details. Our live cast environment is intended to be completely interactive, so you will have the ability to post your questions at any time during the broadcast, or chat with other attendees about the session.
We will address audience questions at the conclusion of the web cast during the Q and A period. If we are unable to get to all of your questions during the presentation, someone will follow up via email. To ask a question, simply use the live chat window located on the right of the screen under the tab "chat." You can also undock the chat box from the screen using the pop out button at the bottom right corner of the chat box.
That way it won't get hidden behind the slides. If you'd like to share your thoughts about today's presentation, please use the Facebook tab, located to the right of this screen. Log into your account, and you will have the ability to share your experience with your friends. You can also share your experience through Twitter.
Click on the Twitter tab to the right of the screen and join the conversation. Your tweets will show up in the Twitter roll to the right. Make sure to use the hash tag #agilelc .
If you would like a copy of the presentation materials for today's session, you will find a button to download a copy of today's presentation and other files under the links tab at the right of the web cast window. A recording of the entire live broadcast including Q ; A will be available in about twenty-four hours at this same location, so you can come back again or recommend it to a friend.
finally, if you ever need to get a closer look at any of the slides during the presentation, just hover over the mouse, hover your mouse over the video player, and click the full button, expand to full screen. Okay, so let me introduce our speakers joining us today. First, Bob Aiello. Bob Aiello is a consultant, editor-in-chief of CM Crossroads and the author of CM Best Practices.
Practical methods that work in the real world. Mr. Aiello has over 25 years experience as a technical manager in several top New York City financial services firms. Where he had a company wide responsibility for CM often providing hands on technical support for enterprise source code management tool, compliance, build engineering, continuous integration and automated application deployment.
Second, Leslie Sachs. Leslie Sachs is a New York State certified school psychologist and COO of Yellow Spider Inc., Leslie, is the co-author of CM best practices, practical methods that work in the real world. Ms. Sachs has over twenty years of experience in the psychology field and has worked in a variety of clinic and business settings where she has provided many effective interventions designed to improve the social and education functioning of both individuals and groups.
With that introduction I'd like to pass the mic to our speakers. The floor is all yours Bob and Leslie Alright, thank you, Francesca, and together with Leslie, we're thrilled to be here talking you today about utilizing release automation to support IT governance and compliance. And I think we'll go to the first slide.
This webinar we're going to establish release and deployment best practices, assess manage deployment related risk. And the third thing were going to try to do is implement a robust employment frame work or at least tell you how to do it. And we're going to have real strong focus on IT governance and compliance.
So, those are the three goals that we have. Establish those release and deployment best practices. Assess and manage risk and then look at things from a wide deployment framework point of view. So the first thing we have to do is understand what IT governance and compliance are all about. The whole point of IT governance is to provide transparency to senior management to help them make the right decisions.
It's very important and enables the organization to establish essential IT controls to protect the assets of the firm. And by IT controls, we're talking about putting up those guard rails to make it much harder to make a mistake and help you be successful. We also want to identify and mitigate risk.
Risk is not a bad thing. Very often risk comes hand in hand with high profitability. So we're certainly not gonna say that we want to be risk aversive. We want to identify those risks, though. We want to mitigate them. And in many organizations, you really have to comply with regulatory requirements.
If you're in a banking situation or any of the financial services firms, even many hedge funds in many different environments, you have important regulatory requirements that can really put you in a position where you can actually be shut down if you don't comply with those rules. And most importantly, we don't want to slow things down, we want to actually enable rapid, iterative development and we want to make sure that we can improve our productivity and quality.
Leslie, have you got something to add there?
Well just to highlight the fact that when you talk about transparency, we should be aware that people have a natural tendency to not always want to be so transparent. Believe it or not even this day of Facebook where everybody's putting certain things out there - when it comes to a work situation, there is often a tendency to hold back typically because there is another personality factor which comes in, which is that most people, as much as they like credit for what they have done, they are more motivated to Avoid being held accountable for a mistake, so there's double edged sword here that you need to both create a climate in the organization where people are encouraged to be transparent, for the sake of productivity amongst everybody.
And also, to realize that, that accountability is important for compliance.
Okay, those are all good points.
Back to you Bob.
That was Leslie's column on personality matters. Thank you, Leslie, Leslie's column on personality manners covers many of these issues on detail. You you can read them on CM crossroads. So, part of providing transparency decisions need to be made based upon accurate information. So let's answer this sorts of questions that Come up, which is what information is needed, how is the information provided, how do we identify and manage assets, those are the things that we build and there now important to the company, but whether it be a payroll, or trading system, we need to understand our assets.
and what change might impact those assets, particularly, when we are doing releases and deploy,we have to know how many releases were going to put out there, and how do we allocate our resources, and can I meet my business goals. You know, we often see that there's an interesting dynamic that goes on, which is many of the people in the trenches say, hey, management makes decisions, they don't understand what's going on in my life, they don't understand the impact of their decisions, and at the same time, we sometimes fail to realize how important it is to provide a transparency, so senior management can have the information they need to make those decisions.
Now all of this is really essential in an environment,where we have to comply with audit requirements and IT governance, and audit, often uncovers material weaknesses in existing controls, the auditors, come in, the identify risk, the most important thing though, is what we have to make sure that what the auditors are finding and what they are reporting on is relevant, and for somebody that has, kind of been in the trenches in this area.
For a very, very long time, I often spend a fair amount of time with auditors, talking to them about what is really important, to find in an audit, and what is kind of superficial, and perhaps, not the most important thing, and perhaps not very relevant. You know, a simple example is an organization needs to be able to tell, exactly what version of their code is running in production at any particular time.
They need to be able to find the source code, and rebuild any release at any particular time, and having these sort of abilities in controls are all a part of having an effective and comprehensive deployment framework, so most of all you want to make sure that this is more than just a rubber stamp, and essentially audit, is your verification and validation, for your IT controls, and its kind of testing your controls and making sure that they do what they need to do.
Now, ISACA is one of the organizations that is out there doing great work on meaning how to really implement IT governments...IT governance. IT governance is essentially two things, and this is from their materials. IT is delivery of value to the business and mitigation of IT risk. So the first of these is driven by strategic alignment of IT with the business.
That means that you set up your IT environment, your technology services, in a way that is consistent with the businesses that you have to support. So, for example, if your business is in the business of providing financial services, well, maybe you want to outsource the things that are not really the most important to your organization, but hold close to the chest the essential competencies that can make you a success.
The second is driven by embedding accountability into the enterprise. So you want to make sure that everybody understands what their role is, what their part is in being part of the solution. And, of course, one would identify the challenges and the kind of concerns that IT governance needs to address.
So this is from the IT Governance Institute, it's another one of my favorite resources. And I encourage everybody who's out there if you think there's an organization, or a resource that I should highlight in these presentations, feel free to contact me offline, we will provide you with that information in a few minutes, and, but.
IT governance institute is one of the best places to get the real good material on how to implement these practices, so you want to consider aligning IT strategy with your business, we talked about that. Cascading strategy and goals down to the enterprise, you want that consistency throughout the whole organization.
And the structures of the organization, who reports to who, you know your testers cant report to the same guy who is responsible for getting the development, out the door, and on time, because, sometimes there is a conflict of interest, and you have to insist that their is an IT controlled framework, the one that is most commonly used, of course in the industry is the ISACA COBIT Framework, and that's one of the ones that I have been very involved with, and idle.
is used a great deal in the service management world, and it very much helps you with measuring ITs performance, and of course you can't improve your practices unless you can measure them. Now these things are not arbitrary, in many cases you have to comply with specific regulatory requirements.
One of the most common is section 404 the Sarbanes Oxley act of 2002 commonly called SOX and the way that you do that is by following the model, the COBIT model, from ISACA, and that was derived from the work that was done by an organization called "KOSO". In the health care arena we have HIPPA and CFR21.
If you've got credit cards and typically you worry about PCI. A lot of times contractually,their requirements were SASS 70,or CMMI level 2-3,and what that means is many government agencies will require that their. factors have all the competencies in level 2 and a select number from level 3, they commonly refer to that as levels 2 slash 3.
Another example is implementing a quality management system and we could have a whole webinar just on that and that's till 9,001, colon, 2008, so establishing these IT controls, protect the assets of the firm, it helps you establish the best practices to prevent mistakes, and that's something that I really want to focus on, this should not just be about passing an audit.
It shouldn't just be a matter of were going to, you know, have these practices on Tuesday, get audited on Wednesday, and thank God we go back to the way things were on Friday, you know, one of the things we got to make sure what we do is include a separation of duties, just as a very practical example.
So, for example, if I am the developer, someone else should rebuild my code and their version is a version that we should be promoting and deploying, so that we are sure that the bill is repeatable, that it's traceable. and that it can be handled by somebody else if we have to make any changes in the future.
Separation of duties is one of the key aspects of these controls. That's where you have an independent build package and release, using a specific build engineer, and one of the goals is to keep the developers out of production. Now, why is that? It's not that we don't like the developers, they do fantastic things for us, but we want to make sure that they can't accidentally make mistakes that would bring down production.
More importantly, we want the transparency that everybody knows how to support the application. Now, in a lot of places where I've worked my entry to the organization meant that the developers lost their ability to go into production. So, sometimes they were a little resentful. They were use to being able to do a deploy, and if they made some mistakes, fixing them real quick before nobody noticed, so, one of our goals is to keep developers out of production so we can have a repeatable process.
Now, sometimes you can just use a different computer account. I call it a virtual build engineer. And most importantly you've got to to have the right practices in place. A good example is the configuration audit, where at a physical and a functional level make sure you can identify the exact version of the code and that it is performing its needed function in the right way.
So we trust but we always verify.
Now, risk can- and Leslie I'm going to ask you to give some input on this in a moment. Risk is always something that is hard to get our arms around. We'd all like to avoid thinking about risk. Risks are related to human error. An example of a common risk is critic man risk, which Leslie will explain and having a lack of traceability and repeatability are the kind of mistakes that we often see, so Leslie, you want to add to that?
The critical man factor relates to the issue of a person who is the lynchpin, and oftentimes, at the one hand,it is nice to be the only person, but. people get nervous, that someone's going to step on their toes, and take that away from them. In a work setting, oftentimes it's preferable or people feel that it's preferable to be that independent person and their reluctant to train somebody else in what their doing because they feel that that's their security.
Yeah, I mean I love it when I'm the only guy that can do something. I feel needed, I feel important. I love this stuff and it makes me feel.
Not so good for the organization.
That's right, you're right on Leslie, that's the point. So assessing risk - there's many unknowns, we don't know what's going to happen when we walk across the street tomorrow. Complexity is something else we're going to talk about. It can't always be tamed, and you have to mitigate the risk by automating what you can, and also having a plan b, and most of all released planning is essential.
Leslie what do you think about that?
I think that you're right on target, you can go more into that. So, complexity is one of the things we want to go into a little bit ...Yep, I'm here. And complexity is one of the things that we want to go into more, which is, How do we manage the complexity? So, the whole goal here is to enable rapid iterative development.
You want to make sure that you can do things faster, that development is supported by effective IT controls, and that you've got practices in place, such as either Agile scrum or an iterative waterfall. So this isn't just about slowing things down, it's really about, How do we manage risk, get our arms around the complex issues, and actually be able to go a lot faster?
Most of all, we want to be able to improve our productivity and quality, develop our code faster, and prevent defects. And what I always tell people The right practices in place actually helps your developers fly. So, we're in kind of a paradigm shift here. Now, we want to improve our productivity, so the only way to improve things is if you have some measurement to be able to judge against, so some of the measures - and I pulled this right off of our sponsor, Nolio, we should certainly thank.
They had some great metrics at their website so I decided to grab them and use them for today's presentation.
The number of major releases you've gotten out the door, the duration of the roll-outs, the number of the released back-outs, so if things went wrong and you had to back it out and the number of bad things that happened by a new release.
And also the punctuality, to be able to get the release done when you planned on and the amount of work it took, and of course the proportion of automatic release distribution, which means the number of releases you were able to do without having to do a lot of hand holding. All of these things are good metrics to be able to understand exactly how to make this happen So, the first thing is you want to establish release and deployment best practices.
Now, I've been hinting that throughout this whole presentation and that's kind of the whole point of this, is giving you practical advice, from our own experience, and implementing this, so were going to talk a lot about best practices, one of them I mentioned them already is having the ability to have somebody else independently build your release, and assessing and managing deployment related risks.
You know, having an honest discussion that folks, you know, don't feel so threatened. Leslie, I am going to ask you to talk about this in a second, but, you know, a lot of times people feel very uncomfortable talking about risk, and so, they end up just avoiding the conversation. Leslie, do you want to add something to that?
Well, it's just that you always want to have an organizational environment, where the team player has realized that the sum is greater than all of its parts and they have to work together, and if one person. Is monopolizing certain information or being reluctant to tell others what's really going on then the whole process is going to suffer.
And that's what a lot of this governance and compliance is about, again, it's making things clear to everybody whats going on,so that when something isn't going correctly, you can step right in there and fix it before the problem gets bigger. And those are all good points. And you'll notice Leslie is taking a very physiologically tuned approach to this.
And that's kind of our message here, having good technology backed up by a strong approach to handling people issues will make you much more effective. But at the end of the day, the psychological issues are important but implementing a robust deployment framework is key. And that's also what this presentation is about, because you need to be able to have release and deploy best practices that are structured in a way, and handled.
all of the different aspects, but then kind of brought together and coordinated by one central deployment frame work.
So let's give some examples. In continuous integration, which is common in both agile and non agile environments, you've got an automatic bill package and deploy. Many organizations are getting better at designing applications for testability and deployment and making the release itself a non-event.
So these are some of the best practices that you want to be thinking about as you put together your own deployment framework .
One of the techniques that I always tell people to do is to use "staging." Staging means you copy the configuration items - that's all the things you built and you're about to deploy - to the target machine, or at least some secure shared storage area, well before the deployment window. And you want to make sure you that you've got the whole deployment automated, including, and don't forget, the ability to fall back.
If you staged your code and then the operations or your deployment folks hit the button and they moved things forward If something doesn't go right, being able to fall back to the previous release significantly reduces your risk. And if you're able to confidently fall back to previous release it makes it much easier to have a robust deployment frame work.
So do all the hard work up front. Automate as much as you can, have a good check list, have good controls.
And most of all make sure you got that configuration audit which gives you the ability to know exactly what version is running in the environment after you're done. Now, it seems like everybody is using continuous integration and it certainly is the best practice. It's a service to the developers.
And, you know you're being successful, and the developers really become dependent upon continuous integration. So that's a good practice, and I think a lot of people are really sold on that one. But I want to focus on this framework for understanding the knowledge around the build, release, and deployment.
And one of the things I find is there's so much complexity out there, people can't really get their arms around it. We're all so specialized, and very often the system folks, they understand the infrastructure, but they don't really understand the application so well. So, you know, the fact that you can have a framework, get your arm around it, to break it into pieces that are more manageable, and provide a service to the developers, it's just really, really important to, you know, think about what your deployment frame work is.
Now, part of this is conceptual. And I'm trying to give as many examples I can, but part of this is also having the right the tools in place, and we'll talk about that as well.
So one of critical things here is testing in QA. I cant tell you how often I've found people...they do a great job putting together, you know, all the steps and they forget to be good testers. And you know, Leslie, I think You can talk to this. Very often it's hard to test our own stuff. So the last step -
Oh absolutely.
in any situation will always be smoke testing. Go ahead, Leslie.
I was just going to agree with you that.
I think your microphone is cutting out there. So it facilitates regression, functional, and performance testing and instrumenting the code will help you with that as well. So the key here is to use testing, as a key resource, and make sure that you realize the importance of including testing in your whole deployment framework.
So, it's not just an after thought, it has to be an implicit part of the whole process, so, we need to be able to build the application once and be able to deploy everywhere. I've seen some organizations where they built it once for development, then they build it for QA and they built it for production Oh, that is awful.
I mean, it's so easy to make a mistake and have a different version of the code running in production than the version that you tested in QA. And you know, the key here to automate and control the configuration for each of your environments and to make sure you you've got verification and validation for the environment.
So that means that, did your verification as, did things past your test. And validations is did you test the right things. And don't forget, a lot of our work now is in virtualization and also in SAS where you're working with services that are essentially located. And I worked with many SAS environments and cloud computing as well.
It's kind of raised the bar a bit for the kind of practices that we need to deploy. And very often we're doing development on Windows, but we are deploying on, Linux, and that's a very common use case, so being able to understand all of these issues with platforms and deployment, can help you be a whole lot more effective.
So one of the messages that I have in this presentation, please be open to using standards and frameworks and I, I'm gonna. I'm gonna take a little snipe at some of my colleagues out there.
I see a lot of people out there, smart guys there are, you know, got a guy the gals who think they know their stuff but they don't ever bother to look at what it says in the industry standards and frameworks.
They sort of written it off without every reading the document.
So my own group VIEEE working group 828 recently updated our planning standard.
My good friends at the EIA recently updated for 9 from A to B, and I saw 1007.
There's lots of good stuff going on with iso standards being updated. And of course, the frameworks, Saka, ITSF,
and I didn't list it, but our old friend, the CMMI.
So these are best practices as advocated by industry experts
and it's very, very important to make sure that your utilizing these forms of expertise.
So, you wanna access and manage risks.
So We need to be open and honest, identify the root causes, mitigate them, manage the process. So, you know, for example, one form of risk is so many moving parts that it's very easy to have trouble coordinating them. Or, to be able to have knowledge that is highly isolated. In other words, one guy knows how to do a disk, but he doesn't know anything about the application that's running on that disk.
So those are forms of risks that we have to kind of, you know, get our arms around and really understand, you know, how to handle it.
So, Mary and Tom Poppendieck ask a question that I think sums it all up. And that is, How long would it take your organization to deploy a change that involves just one single line of code? One line of code! If one line of code changes and it takes you two days to do the deploy, something is wrong.
And that is in many organizations, that in fact is the case. So, I think, you know, the folks from Lean, Mary and Tom Poppendieck keep bringing up a great issue. So I'm going to spend a little bit more on risks, a little more time on risks, because that is a key part of IT governance and compliance and the focus of the webinar is, how to do this stuff at practical level, pass your audit.
If you've got a date with an auditor, and you have these risks identified, you will come away from that audit very happy.
And your auditor will be very unhappy because he won't have anything to write up. So missing a single step can result in problems. I find a lot of times people don't realize the risk of missing a step. Sometimes you want to put in controls so you can't do that. You check off each one and you have a second pair of eyes.
And mistakes cause risk. So having your structure set up in such a way that it's easier to not make a mistake. I often give the example of the cockpit of an airplane is set up, it's designed to make it almost impossible impossible to make a mistake. And that's a great thing, you don't want pilots making mistakes.
We don't do the same thing with software, we expect software people to be really smart and never make mistakes, that's completely unreasonable. So, manual steps result in mistakes, manual deploys take way too long. And if you don't have a way to back out, that is a really bad risk. So those are the kinds of things you want to identify as much as you can.
Mitigate those risks.
Some of the other sorts of things that I find we run into, is we're we're doing great, and we had 7 developers. We had another 1 or 2 resources in, somehow or another the same team that was super makes a lot mistakes, often that's related to communication. But being able to scale up is just essential.
So if you have problems where you can't add a couple more people to the team. Then you have issues related to scalability. And you know, one of the things I always tell people is overcome that defeatist mindset, too often people start deciding that we're just going give up another weekend. In other words we're going to start on Friday, we were, just going to assume it's going to take us 2 days to do the release, and they just have this defeatist mindset.
They can't seem to get over it.
So, one of my messages and I'm going to bring in a couple of colleagues just a moment to kind of get their views on this, at least I'm going to quote them. We want to make sure that we have a mindset where we can be successful. And that we can do this in, in a shorter period of time, and manage that cognitive complexity, manage the reasons why things go wrong.
Leslie, I want you to talk a little bit about cognitive complexity if you can. Let's see if your microphone's working this time.
Can you hear me all? There we go.
Yeah.
So what is important to realize is that the cognitive - human brains although they're very talented, they do get overwhelmed. And we need to take that into consideration when we're setting our IT processes. And even, as you were talking about, the smartest guys they can do things and they can do things under pressure, but each time you make a a strong demand on somebody's cognitive processes, you're increasing the chance that they're going to make a mistake.
So the principle that we're considering here is the fact that, although there is a significant amount of complexity that's always inherent to IT processing and to building and deploying, we want to try to decrease as much as possible how much is in the moment and try to automate as much as we can so that we simplify the process at any given moment.
When you teach anybody a new skill, whatever it is, you always break it down into smaller pieces, because it's easier to do one at a time and focus on them in isolation. And so that's the goal here, is to break it down to have less steps, thereby less chance of error.
All right, that's great. Now, Leslie has more examples of this and Chapter and in our book on Configuration Manage and Best Practices annals doing our column Personality Matters. But the sorts of things we often run into is people not understanding the build. Too many moving parts. And most of all how do we automate something we don't fully understand?
So here's some great examples of complex technologies I find often get people pretty flustered.
So Tomcat and Apache are alright, that's not so bad, Webster can get pretty challenging, J Boss working with data base dependencies, data base warehousing and my favorite:OSGI , where you have these plug and play compatibilities change. A lot of what's going on in terms of configuration management.
So it's good to look at some of the things, I've given you some examples here for the idle framework of service asset and configuration management, SACM and the definitive media library we've got lots of materials written on how to implement this stuff at a practical level. But considering those kind of best practices is really important.
So, there's so much out there. We've gotta organize our best practices. Parts of solutions are often automated at varying levels. So you might have a piece of it that's automated, and getting started is what I often tell people is the hardest part. As long as you get started and you start building pieces of it, here's where you get back to needing a framework to organize all the different pieces of your automation and managing the process is really essential.
So, Demming always says, "Build quality in." And I always tell people we've got to be the shoemakers children and make sure that we use these principles for ourselves. We have to do verification and validation on the very deployment framework that we're building. So we need a way to understand how to build, package, and deploy and organize all these different pieces.
So it's an iterative process. You're not going to do this immediately straight out of the box. You've got to make sure that you also have a feedback loop to tell us how we're doing. So a lot of what I want to talk about for the remainder of this presentation, and I think it's just really important for folks to get their arms around, is the fact that you need a framework to help you organize it.
And I call it...it's kind of a skeleton. We put all the parts on there and it's also in some ways the conductor of the orchestra, so that, you know, you can run parts of it and see what parts have completed successfully and understand what parts need to get rerun. So, do we need automation? Well, of course.
I mean, sometimes developers, you know, joke with me, "Ah, real developers don't need to automate. We can do everything real fast. We just need smart people." And the one I like, this is the line that I always love, "It's too complicated to automate," which is absolutely nonsense. So, you know, once again we got to get over that mindset.
So let me bring in my colleague Dean Leffingwell, who says, "Making each product a successful and routine a good end, an event that is indeed planned and eagerly anticipated. Yup, one that happens almost on autopilot. That's from his books Agile Software Requirements, and Dean calls this the agile release train, and I think that's just a great way to look at it.
My colleagues Jez Humble and David Farley, they talk about the deployment pipeline as an automated implementation of your applications build, deploy, test and the lease process. So, we're seeing I'm not the only one that's saying that we can make deployment easier, we got to understand our practices, and the whole aim of the pipeline according to these experts is makes building, deploying, testing, releasing software visible.
Hey, there's that transparency word again that's implicit in IT governance. Software visible to everyone involved. It improves feedback, so the problems are identified and so resolved. Hey, this is IT governance. Wow. it's all part of the aim of the pipeline. And we enable teams to deploy and release any version of their software to any environment at will, through a fully automated process.
And this is from their book "Continuous Deployment." So some of the anti-patterns that Jess mentions is deploying software manually, deploying to production like environment only after the development is complete. So that means to push things forward you should always have deployment practice to serving the developers, then QA, then UAT.
Use the same practices in every environment. Don't just wait until the very end to try to automate things. You really have to start up front. And manual configuration of production environments are anti-pattern. These are things you should not do. They are practices that will contribute to failure and add to risk as we've talked about before.
So another great practice is get your change control board going and make it a valued asset. The CCB can drive the higher release process. It's more than just gate keeping though. You want to make sure that you've got the right experts, the subject matter experts, your SMEs to help form the Change Advisory Board.
This is the group that tells the CCB what's gonna happen if they make a certain change. So make sure you get yourself a CAB. That's the change advisory board. That's described in the Idle framework, and we've written about that a great deal as well. And you want to make sure you're managing releases and updates and respond to variations as needed.
You're going to have variations and you want to make sure that you coordinate the whole process.
So just some of the lessons learned that I have found to be most relevant. I mentioned before: look at release management and CM as the way that pilots look at the cockpit of an airplane. They make sure those controls are absolutely fool proof. People can't make mistakes. Automate everything, even a one time command.
Don't type into the command line. Make a script with date and time on it. Keep it in a script. This way you can go back and see exactly what you did. Traceability, repeatability, built quality in. We talked about that. Verification validation. These are some of my favorite lessons learned.
So Agile CM is also a key area that a lot of folks are really putting a big focus on and is very valuable. You want to be able to rapidly build, package, and deploy. You want to be able to support iterative development, deliver functionality to the end user, and maximize continuous integration with deployment to a test environment.
So if you're using a CI Server, you got cruise control, whatever product you have out there, is a great continuous integration servers on the market, make sure that you also deploy to an environment and actually run some tests against it. Testing is part of CM. So high rate of change. High rate of change can be challenging.
It's a potential for significant pain. And once again I want wanted to quote someone else here, I want you to just hear my views, so Jess Humble says, if it hurts, do it more frequently and bring the pain lean forward.
It sounds like my trainer when I was trying to learn how to lift weights.
I can lift all of two pounds now.
So smaller iterative releases are really critical they're much less risk, you get better at it.
So if you go for a big bang release, if you store everything up for a once a month release frankly frankly that adds a lot of risk.
If you get your release management process through a framework, to a point where it's easy, then there's much less risk. You want to make releasing a non-event.
That means it's something you know is gonna happen and you're looking forward to it.
But, you know, everybody knows that if you're gonna take a few hours and they're still gonna get to enjoy their weekend.
So it's easier to figure out what went wrong when you have smaller iterative releases as well. It's the same concept in continuous integration.
So one key aspect of this is looking at the whole whole application life cycle, or what we call the ALM.
The ALM is a wide focus, tools are center stage, and the integration, what we call multiple tool chains is essential.
And you automate in a work flow solution.
So you actually have that traceability to make sure people are following the process. this should sound a little familiar because once again in IT governance you want to make sure you have visibility into whether or not everybody followed the process.
so automated processes they're much easier to test.
You have that trace-ability.
Did I make a mistake?
I always tell people I make a lot of mistakes. I make tons of mistakes.
I make more mistakes than anybody.
But I can always show you exactly what I did and that makes it much easier to go back, fix it in the automation and then we don't make that same mistake again.
And we also can see exactly what we did from one point to another.
If you're doing things manually at the command line You really are not gonna accurately remember how you did the deployment last month. So this gets really tricky when you have complex environments. What version of Java are you using? What are your operating system patches? You know, which ports open?
This stuff can get pretty hard to handle manually. So, one concept that has come out in the deployment framework is to have my agent handle that. In other words, you have an agent that's running on a PC, another one that's running on your AFAX box, another one that's running on your Linux box. So, the front end of the deploy could deployment framework is automated in a consistent way.
And this back end agent handles the platform specific issues. And that makes it much easier to set up these deployment frameworks. It helps your scalability because this way you can also do a thousand PC's at the same time. So agents -- that's software that's running on the target machine that helps you with the deploy -- is also one of the best practices that you'll find.
and some of the tools that are out there. And not surprisingly by our sponsor Novio. They have this type of technology as well. It allows you to do parallel deploys and understand the status of every step of the deployment and, you know, be able to stop them, restart as needed. So, just coming up on our conclusion.
We're gonna get to take some questions, next. And, of course, my colleague Francesca's going to come back and ask us some of the questions But, you know, just in conclusion I think governance and compliance is essential. We shouldn't think of it as being an afterthought. Having the transparency to know what's going on in your organization, the number of releases that are being completed, the number that failed.
We talked about some of those metrics. Providing that governance senior management is really important to enable them to make the right decisions. And believe it or not, I've had senior managers that have said to me "OK" Now that I see what's going on, I think you need more staff.
I'm gonna give you some budget.
So there's things that senior management can do for us if we give them the real data through IT governance.
Compliance in many organizations is not optional.
That is, you know, playing by those regulatory requirements.
If you're a bank or other financial services firm, or medical, or defense, you know, you've got serious regulatory requirements and they are not optional. So you want to make sure that, you know, you are considering what the impact of IT governance and compliance is on your organization. And, you know, I always talk people that you should not just be an opportunity way you passing we have an opportunity here rather to enhance quality and productivity.
You have all these processes open here.
You're looking at all of your tools.
You're looking at how you do your bill deploy, why not make it worth your while and help yourself get from two days down to a couple of hours to do a deploy?
I have literally seen deployments go from a week down to an hour. Now that's dramatic. I mean sometimes you only get from, you know, a few days down to, you know, a half a day. Fine, I'll take it. But the point is you can enhance both your quality, because you don't so many mistakes. You get it right and also your productivity.
And one thing that's really key here is you make this easy you can take more releases. That helps developers a lot because if they can give you bug fixes in smaller increments, there's much less risk of there being a problem, as a result. Makes it much easier to manage that. So, I really want to focus on the fact that you gotta have to have a framework.
You can't this in bits and pieces. You need something to help you really organize your thoughts and the processes and you can't do this all at once. It's an iterative process. You make a small change just to get started and and, you know, that will really help you address the first issues and then you can take take it from here.
And making it an iterative process means you can show some good results.
And then, you know, over time time get a little better at each point.
Now, you know, I can't go further without mentioning that Nolio is our sponsor for this.
They've got a great it's a point of framework, and we're going to be putting up their contact information in just a moment,
so you can take a look at that, but I really want to come back to Francesca and see if she's got some questions for us and I bet she does.
That's right Bob, but first I want to thank you for that great presentation that you and Leslie gave.
And I want to remind our audience members if you have any questions for Bob or Leslie, please type them into the chat box located just to the right of the screen being, and we'll get through as many as we can with the time left.
So, with that being said, our first question is, this is great take one from our audience. My management doesn't really support setting up these practices. So, how do I get started without top down support? Excuse me. Well, honesty, that's a tough question. And frankly that's a good example of a risk.
Because if your senior management does not support what you're doing, the chances of you being successful are slim indeed. And so what you may need to do to get started is really market, so really put on your sales hat and sell your senior management on supporting this effort. That said, I've been in lots of organizations where the real change came from the bottom up.
And it was the developers who said, hey, this isn't the right source code management solution for us, let's go, let's evaluate the products that are out there and start using something else that can help us do our work better. And you know, sometimes, frankly that's you know, it's the folks in the trenches that really understand which way to point their rifles and you know, do the right thing.
But, you know, I got to say, at the end of Today having senior management support is, as far as I'm concerned, it's just an absolute requirement so, what I always say to people is that it's important to go from both the top down and the bottom up?
What do you say?
I up, you know, okay, lovely if you want to add to that .
Well, I just wanted to highlight what Bob was saying about going from the bottom down because the same way you want to make sales pitch and have the upper management be on your team.
The fact is that is the quote "little people, or the people doing the actual work, know better than anybody what works and what doesn't.
And if you can convince them that this is going to help them and make them better at what they do. And they're gonna come out both more productive and feeling comfortable with their work and being able to present a better product to their upper management. They're going to come on board and that's going to make your job of showing the upper management why this is so valuable, even easier.
So, you know, it certainly is important if you're getting flak from the bottom to have the upper pushing. But the fact is is the real work is being done at a certain level. And if you get those people on board with the project, it filters up just as fast.
Okay.
Alright. Good point.
Given that, how do you overcome resistance to change?
Boy, you know I was in one place Francesca, where the manager brought me over, literally brought me over a stick, and I remember it was a hockey, it was a small hockey stick. And he said, "Look, you know, keep this by your desk and Make sure people know that you're allowed to use it.
And he was being facetious, of course.
But frankly, you know, I've been in places and you know that I hate to rat out some of the government agencies are the worst for this when you see all the people in the meetings with their arms crossed across the chest, and you know, they don't want anything being done differently.
They've been there for 20 years, and you're coming in, and what do you got?
What are you trying to tell him how to do, release his fan cluster for, and meanwhile, you know, they can't make a one line change in production without blowing their systems up.
So, frankly, Overcoming resistance to change can be really difficult.
And, you know, this is something, we took a whole chapter to talk about this in our book I think for me, one of the ah-ha experiences was really learning to market what I was doing. and really, you know, become a salesperson.
And I'm gonna ask you know, Leslie to come in and give her input on this too, but you know, I think there's a people side to this, and maybe Leslie can speak more to the people I often had, you know, some trouble there.
What do you think Francesca?
Can we get Leslie in on this one?
Of course, Leslie what do you think? Absolutely, I had some things I wanted to add,
two things came to mind right away, one is the old adage that you catch more flies with honey than with vinegar and that works here too.
If you're gonna just on a personal level make people comfortable, they're going to open their ears and their mind to what you have to say more than if you try to bull bully them.
And the other thing is an important lesson that I learned from my father, who our book is actually dedicated to.
And he always taught us that you.
Any good business interaction has to be win-win.
He would not have approved of, you know, some of the political statements where they pit business against individuals and vice versa,
because he really felt that good business is everybody has to win something, and that's the key to negotiating.
And so, in terms of resistance, what you want to be doing is showing all the players, how there is something in it for each of them. It's not you coming in and imposing another system that's more work for them, it's you giving them more tools for their job to be easier. And that naturally is going to break down the resistance.
The other piece of it that's a personality slash psychology aspect to resistance, is that you have to, you know sort of have a little insight into the fact that there are different types of people, and you try to have some way for each of the different personality types to have a role in the team in the process that suits them.
Because you're going to get a lot less resistance if you're working with their natural tendency than if you're fighting with them. And it's not as hard as it sounds, if you look in our book, you'll see there are specific examples. But that's the key principle, is that you want to kind of get a feel for each of the people, and find a way to involve them in the process that suits their natural way of processing.
OK. Excellent advice. Those are all good points. Yes, excellent. Excellent advice and Wise words of wisdom. Now, next question; why is it essential and or advisable to have a different resource do the bills? I'm gonna take this one.
Okay. So I'm gonna. Aw. We can share the
Leslie, you go first. Go ahead, Leslie. You can go first.
Okay. No, I just wanted to point out because we kind of skipped over it before with the separation of duties, that one thing that I've noticed is that in addition to the fact that some people don't want to acknowledge when they have made a mistake. Sometimes they just don't see it, and anybody who has done editing knows this.
It's much easier to edit somebody else's writing than your own, because besides the psychological reluctance to pick up on your own errors you just sometimes don't notice them because your brain will fill in for them, and having another set of eyes is just so useful and so you always want to have a different entity involved with the build and the release versus the developers.
Bob? Yeah, I can just tell you from my own experience I've often been this independent bill person and the first time I tried to build it always fails.
And it fails usually because I'm not telepathic and the developers thought that I automatic We knew to set environment variable scribble scribble to x y z pound slash z colon slash.
And I'm like, wow, look I love surprises guys, but you know, in configuration management, it's good to know how to set those environment variables.
And usually it's because they forget to put it in the release notes.
They forget to communicate it because they did it Three months ago they set it on their machine and they didn't think about it again.
So having an independent person rebuild the release on a different computer using a different computer account. The first time it always fails. It's not a bad thing. You find out what are those things that I forgot to check into source code managing? What are those dependencies that I forgot to get my arms around?
Talk about risk. You know, I was working with a team doing a trading system, not too long ago. They lost the Anscript to do their build and they were building java classes on the side and just sort of hacking them in to the jar and deploying that jar into production. Yikes! So they knew they couldn't rebuild the release.
They knew they couldn't find their code. So they just sort of like, you know. And this is a training system. It's not the sort of thing you want to really cut corners on. So for instance I know we got a couple more minutes. But I think this a real good thing to make sure people understand. We're not saying that anybody Is not smart.
They're not doing a great job.
You know, we're putting in these extra steps. To test our selves to verify that we've got all the right things there.
And, you know, I've got to tell you, one instance that someone asked me this question What if my loved one, I was asked this on a job interview one time, what if my loved one was on a life support system and I did the release management for it What would I do to make sure that I never made a mistake that, God forbid, could jeopardize the life of my loved one? And, you know, when you look at things from the perspective of how do I make absolutely certain that I can't make a mistake, you know, then I think these practices start to make a whole lot more sense.
Trust but verify.
Francesca, back to you.
Alright, so we've got time for one or two two more questions so let's get right to it.
Quickly, do you have any recommended best SCM for cloud computing?
Wow that's a great question and this is something I would user love to interact directly with the person who asked this question and anybody else who is interested in this topic.
This is something that is Really really hot right now and I don't have all the answers right now, but I sure do.
You know, I think this is, you know, real finger on the pulse time, so. I've recently become very involved with some cloud computing efforts and one of the things that I've noticed is the. The risks are far greater when you don't have control over everything and very often people don't have as many practices.
This is in place because they're saying to themselves, you know, the vendor is managing it and,
actually, you need these practices in place more because things are begin done on the side. Have control over. So, the first thing is with cloud computing, you want to make sure that you really consider all of the platform dependencies, all the change control dependencies.
What's the vendor SLA? There's a lot more, and I think that this is something I'm going to ask the person, I think I'm going to try to reach out directly. Maybe this is something we could get into a whole article or perhaps another webinar, and really address it in more detail, but back to you, Francesca.
Well that's excellent, so we'll give you that person's contact information and the final question, what are some of the lessons learned from implementing all of these practices that you've spoken about today?
Well, I gotta tell you. It, I've made every mistake you can make in the. I've got a whole chapter in my book on mistakes that I've made, not because I feel guilty about them, but because it really important to take a look at what these lessons learned are.
Some of the things that I've learned is to provide that transparency to the whole Organization is to what you are doing and that's implicit in what IT governance is all about. Make sure you really use industry best practices and not just try to reinvent the whee land that's, you know, those are the standards and frameworks that we should all be consulting and I'm going to, you know, really put on the line here.
I've got a lot of colleagues here that haven't read the materials from the saga. They haven't read the materials from Idle.They haven't read the I Tripoli standards, the EIA and I'm asking them, how can you consider yourself to be a professional if you haven't looked at all the best practices that are out there?
So that is really critical, you know, looking at what other people have done and what their suggestions are and even if you disagree, that's fine, but making sure that you're open and you know that's a mistake that I made early on, it's a mistake that I certainly won't make again. And I think that being implicit in developing robust deployment framework is really making sure you're looking at what the other experts have to say and that, those are the.
sorts of lessons that I certainly learned the hard way. And I hope from sharing with you in this webinar that you won't make the same mistakes. Well said, well said. Leslie, really quick, before we end this event today. Do you want to add anything to that or any final thoughts?
Just to reiterate that as much as there's a technology piece of it which is so, so important. Ultimately there are people driving the technology, so you have to remember that, treat them like people, get them on board, help them understand, cut through the intellectual and cognitive hurdles that they have.
But remember that "good job" always goes far, saying "thank you" is important.
And Bob, do you have any final notes or comments?
Just that you know, this is a team sport. I hope everybody will reach out to us on CM Crossroads and the Agile Journal and we. Love to continue all of these conversations online and share our practices. I often tell people, like, you know, I know a lot about this stuff, not because I'm particularly smart, but because I've gotten out there and shared my ideas.
I hope you all will do the same and come out and interact with us in our virtual community where we get everybody together sharing what works and different ways of doing things a whole lot better. That's it for me. Francesca, Thanks so much.
Oh, thank you, thank you to both, to you, Bob and Leslie. And, we're out of time. Thank everybody for joining us. Please join us for our next Agile Journal Business Webcast Series Broadcast, taking place. 19th, which will be sponsored by CollabNet. And, again, a big thank you to everyone involved and also to everyone who attended today.
Have a wonderful day.
Trackback(0)
Comments 
Write comment
 |
TrackBack URI for this entry
Show/hide comment form
thanks so much for your comment. We enjoyed giving the presentation. There was a really funny note however. We checked our audios before the broadcast and didn't realize that Leslie Sachs's audio had an incorrect setting which resulted in Leslie hearing everything on a 3 second delay - yikes. Now the funny part is that we tested - but failed to test in a way that was consistent with how we would be working. Make me laugh because - What business are we in?