– Today on DevOps Defined, we’re talking continuous integration Let’s go What’s up everybody? Ken M. Middleton here, your DevOps Recruiter with another episode of DevOps Defined Remember, if you get anything in these videos, don’t forget to like and subscribe so we give you a new video each and every week So today we’re talking about continuous integration To do this, ladies and gentlemen, today we have a very special guest We have none other than Donovan Brown, yes, Mr. Microsoft himself Donovan’s going to share a number of things today in regards to continuous integration One, he’s going to tell us the benefits of CI and why each and every company should absolutely do it Two, he’s going to share the different types of CI tools out there and what you should consider or be looking for in regards to these tools or as the future progresses, in regards to what’s going to be important Third, he’s going to talk about the one thing that a lot of people do that they think is smart but absolutely ruins your chance of actually implementing CI into their environment A lot of stuff to learn, so let’s get into it – Hello ladies and gentlemen, it is Ken M. Middleton here, your DevOps Recruiter here with another episode of DevOps Defined Got a very special guest today I know a lot of times I say I have a special guest, but I really do have a special guest today, known as the Man in the Black Shirt, known as maybe Mr. Microsoft, Mr. DevOps himself, as you can probably guess who it is one thing you might not know about this person, of course he’s an avid programmer, but something that I found out that I was very interested in that he’s ranked #11 in air hockey, which I definitely have some questions around that! (laughter) So that being said, you can guess who it is, welcome to the show today Donovan Brown, from Microsoft Donovan, how are you doing today, sir? – I’m doing pretty well I appreciate you having me on the show, man I’m excited – Dude, so happy and appreciative of you too Today we’re going to talk about something very interesting and at the base of DevOps that is very near and dear, it seems, to Donovan’s heart is continuous integration With that being said, first, before we go into that, Donovan, if you will because I’m always interested, talk about your background in DevOps and how you got involved in it, and your, sounds like 5 1/2 year journey with Microsoft with evangelizing it to the masses – Right, so I’ve been at Microsoft for about 5 years, but before I joined Microsoft I was at a small consulting firm called Notion Solutions for 7 years and so I was doing DevOps long before they called it DevOps I’m a certified scrum master and as a member of that consulting team, we would fly all over the world installing and implementing a tool back then called Team Foundation Server – Right – What Team Foundation Server had in it was agile planning tools, and continuous integration and source control Being a certified scrum master, you would stay on staff with those customers after we installed the software, teach them how to use it, and help them come from the waterfall world into the agile world After you start producing an increment of shippable code, which is the goal of agile, the next thing is to ship that code into production You don’t want to do that manually You want to automate as much as you possibly can, and that’s a lot of what DevOps is about, right? DevOps is about people, process, and products It’s about all three, but a lot of times you see a lot of focus on the products that we use and the continuous integration and the continuous delivery, infrastructures, code, and all those types of things When I came to Microsoft, it was really starting to get a name People were calling it DevOps and we were trying to figure out how is Microsoft going to position ourselves so that we can go out there and do some great things for developers of any language, targeting any platform We took Team Foundation Server, which used to be something you would install on premise, and we basically moved it to the cloud It’s since been renamed to Azure DevOps, but it is a suite of services that allow a team, again programming in any language, to be able to automate the deployment, automate the compilation and testing of the products, and automatically ship it into production for you That’s what I’ve been doing now for the last 5 years, 5 1/2 years at Microsoft – Awesome, awesome So let’s talk about this thing called CI, continuous integration If you will, let’s just start at the base What is continuous integration, Donovan? What are the benefits of it, as you see, in a DevOps environment, watching companies do it? – Great, so continuous integration is the practice upon every commit to the repository, so a developer is writing code, or a operations person is writing infrastructure’s code, they’re going to then commit that to a repository That repository is where you version your code It’s where it’s all safely kept Well, we can actually detect when that check-in has happened There’s a trigger that fires, and what CI does, it says, oh, great! There’s a new change, let me go download onto this virgin, clean machine Not a developer’s machine, a separate machine, Let me download all the code, and what I’m going to do is I’m going to try to compile it I’m going to try to test it, and then I’m going to package it so it can be deployed The benefits of having that is historically, if you’re working on a team, and believe me, it’s unbelievable, in 2019 there’s lots of people that are going to watch your show that do not have continuous integration – Wow – For me, it’s just unbelievable that in 2019, you’re still on a team like that I’ll talk about some things that you can do to correct

that right away But what happens historically is we would want to a release They’d say, okay, Donovan it’s time to do a release, and I would have to make sure that source control had everyone’s changes in it, I personally would do a get latest, make sure that my machine was clean, I would compile the code and then I would package it and test it and get have a zip file or a binary, and say, okay, who do I hand this to? Who do I email this to so you can go off and do the next manual step? There’s a lot of downsides to that First of all, my machine might not be perfect My machine might not be this perfect environment to where I can make that it’s built even correctly Maybe I have old versions of libraries on my machine that are going to make it behave differently in production than what we expected it to be There’s a lot of bad things about having it built by an individual Not to mention, I, Donovan Brown, have to stop what I am doing because I could be in the middle of fixing a critical bug, but they want to do a release, so now I have to find a way to safely put my current work aside, so that I can go and then build this code for everyone else so that we can manually deploy it out into production So as you can see, just describing that sounds miserable – Yeah – When you put continuous integration in place, no one has to stop what they’re doing to produce a build Every person commits their code to the repository Our build machine identifies that there’s a new change, automatically downloads it onto a perfect machine, compiles it, tests it, and packages it One of the cool things about that, if you go back to the scenario where you do not have this, I can say without any doubt in my mind that if you live in a world with no continuous integration, you have come into work someday, and you’ve done a get latest and the code does not compile in your machine anymore It’s called breaking the build, a lot of people make the person who breaks the build bring in doughnuts We usually have all these cool raspberry pies (laughter) That will tell you when the build is broken so that people know better, but if you break the build, you’ve now prevented everyone else on your team from being able to work today Because now we’re all sitting there, like we all did a get latest Our code is broken, it doesn’t compile anymore because Donovan broke the build Come on man, what’s going on? How do you keep doing this? But the real question is: how do we not know that you’ve broken the build? That’s where CI comes in If CI is in place, when Donovan checks in the broken code, it’s that machine that goes off and tries to compile it, and it instantly says I can’t compile this code anymore Something is broken, and it notifies everyone on the team, do not do a get latest right now, because the build is actually broken So you keep the code that you have, you’re able to keep working, while I get a notification saying that I just broke the build I go make whatever changes are necessary, and I put those changes back into our repository The build triggers again, and hopefully the build succeeds and sends a notification to everyone saying the build is healthy, everyone can get the latest piece of code – It sounds like Donovan it just saves you mounds and mounds of time. Oh my God! – It’s a no brainer! What’s even crazier than that is most of us give you continuous integration for free! Microsoft, CircleCI, AppVeyor, there’s just a whole list of vendors who are like, here, use this service, it costs you absolutely nothing to do, so why are you not doing that? That’s the question that I get a lot from people Donovan, I’m all on board, I just saw you speak, I love what you said but they won’t let me install continuous integration at work I’m thinking, what do you mean they won’t let you install? Well, they think it’s going to take too much time, it’s going to cost too much money I’m like, well your 1st mistake, believe it or not, was that you asked (laughter) That is literally your 1st mistake was that you asked if you could do it, because what’s happening is, we’re all under a lot of pressure to ship There’s always somebody who’s like, Donovan, we needed it yesterday, we needed it yesterday If I’m always feeling like I’m behind the 8-ball, or I’m always running behind, I definitely don’t think I have the time to go off and do something else, right? I just need to keep on chugging at this backlog of work that I’m trying to do and I don’t have time to set up continuous integration What you’re not realizing is, a 15 minute investment is going to pay dividends for the rest of your career It literally could take as little as 15 minutes to set up You could get started with our products or others for absolutely 0 dollars and 0 cents, and no one else on your team has to know that you did it That’s the kicker, right? If I … the reason I would have to go ask for permission is because I’m going to inconvenience everyone, or I need everyone to stop what they’re doing while I go off and do this particular work But if it costs you $0 to get started and you don’t impede anyone else’s progress, why do you then need to go ask for permission? It doesn’t make any sense to me So let’s think about this, let’s say that we’re on a team There’s 7 of us on a team, and I try to come to you and convince you all that we needed continuous integration Like, we don’t have time for it, we’ve survived this long without it, let’s just keep doing it the way that we’re doing it because we’ve got deadlines coming up What I would’ve done instead, is I would have gone off to one of these services, I would’ve gone to dev.azure.com I would’ve created a free account, and I would’ve set up continuous integration against our repository No one on my team would even have known that I’d done it They would keep committing code to the repository Azure DevOps is going to light up and start building the code for me and let me know if the code is safe or not for me to get So I just imagine Let’s imagine it’s Friday and then all of a sudden, Thursday night, somebody checked in a build that was broken and Friday morning, the team comes in Everyone does a get latest like we always do, except me I don’t do a get latest. Why? Because I’ve got an email that says do not touch that source

it’s broken from the CI system So now everyone’s at the water cooler but me, and they’re all talking and complaining about the person who broke the build while they’re over there, frantically trying to get the build fixed, and eventually I hope someone’s going to say, “Where’s Donovan? Why is he not over here?” Like, no, Donovan’s still at his desk working They’re going to say, “Well, why are you still working? Didn’t you do a get latest?” No, I didn’t do a get latest because I knew the build was broken How did you know the build was broken? Because I used a CI system What’s a CI system? You completely change the entire conversation You’re no longer trying to convince them with your words that this is a good idea You’re letting the results speak for themselves The fact that I’m still working, and you’re over there getting water because you can’t work right now is why CI is so important The next question they’re going to ask is, “Can I get in on that?” Absolutely Give me your email I’ll send a notification You’ll know next time So stop asking to do your job Your job as an engineer is to continuously deliver a value, and implementing CI is going to allow you to do that That is your job, so why are you asking to go do your job? To me, it’s as ridiculous as when you get to a coding problem, I can write it as a for-statement or I can write it as a while-statement, let me go ask my manager which one of these I should choose I could write it as a switch-statement or I could write it as an if-statement Let me go ask my manager if I could get permission to write a switch-statement That would be ridiculous, right? To me, it’s that ridiculous not to have CI in your system It is your job to do – Well Donovan, let me ask, why are they asking? Because to your point, I’m under the impression that it’s part of what you do – Right – Why do you think some engineers are asking? And then I’ve gotta ask, and you just kind of alluded to this, is it the time factor and the cost is why the leadership is thinking not to do it? – It’s two reasons I think the leadership is more worried about the time commitment, right? Because the leadership is thinking about milestones, we have these deadlines we have to meet, I don’t have time for you to go off and try to figure this stuff out What I tell everybody is the amount of time that it took you to produce the PowerPoint, or do the research to go convince your peers, you could’ve been done already That was a total waste of time and then just be told, “No.” It’s one thing, like, if I go and tell you, “Hey I want to set up CI,” and you say, “No.” and I do it anyway? That’s bad If I just set up CI and I didn’t even ask for it, and you reap the benefits, it’s all goodness Right? So asking actually slows you down by asking for permission to do something that I think is your job But it’s the time and the money Those are the two things that I hear people talk about They think it’s going to take too long or cost too much Well I’m here to tell you, we give it to you for free and it takes about 15 minutes, and 15 minutes is generous! Let’s say for example, you’re brand new Never done this before, it might take you longer than 15 minutes, you still don’t have to ask permission I take 15 minutes out of my lunch, I try to set up CI on Monday, I get halfway done and then I put it aside. Go eat 45 minutes for lunch, do my job. No one in that building knows anything is different than it was on Monday On Tuesday, give it another 15 minutes Maybe by Friday, I get it working, and then from that day forward, I reap the benefits Now if I was going to take 1/2 a day off to go to try to figure out CI, you probably have to ask permission, because whatever it is you’re supposed to be doing that day probably isn’t going to get done because you’re off doing your little pet project But just take 10-15 minutes a day to go do a little bit of research and figure this out because no one’s going to fault you for that small amount of time and you don’t have to go ask permission to go spend 15 minutes of your own lunch making yourself better and making your team better – And when you do it, it sounds like from your description, that it’s not going to change, once you implement it nothing’s going to change in your environment, besides the people who take advantage of the benefits of CI – Exactly! Your peers will never know! Your peers will never know Your repository looks exactly the same, they don’t know that we’re monitoring it, they will have absolutely no idea that you’ve done this unless you tell them or they see the benefits of it themselves – Wow, that’s a no-brainer. That’s crazy – It is! What’s crazier to me is when I go to a conference in 2019 and I say, “Raise your hand if you’re on a team that doesn’t have CI.” And hands still go up! Come on, man I’ve been writing software for 20-some odd years I live right down the street from the old compact computers, it’s HP now, that’s where I started I remember how hard it was to set up a CI system I remember how long it used to take to set up a CI system We didn’t have one when I first joined the team We had one when I finally left, but creating make files and setting up a machine, and getting the network stuff Our world has changed You click a couple buttons, and CI is up and running It’s just amazing – It’s done. What are the main tools now? I mean, I’m sure you’re a little bit biased, but what (laughter) we want to talk about the major tools and the ones that you see work the best in an environment and let’s talk about from a cost-perspective versus effectiveness-perspective, let’s talk about both of those What do you see as the main tools out there that people should take a look at? – The #1 competitor that we have, believe it or not, is Jenkins, which is a free product, right? Jenkins is free You can go and download Jenkins, install it, and start building with it immediately Jenkins has little bit, I think, more of a learning curve I install our competitors’ products, clearly, so I can understand the true cost of ownership and if you have Jenkins running right now, and it’s been running for years, you have now forgotten how hard it was to set up, because you’re just reaping the benefits, right? You’re like, Jenkins is awesome

We’re able to build, we can make code, it’s fantastic But like, do you remember times zero? Do you remember when you installed it for the first time? I do because I re-install it, like, monthly to see like, has it gotten better? What are our competitors doing? So the initial set-up, getting all the plug-ins in place, learning a new tool, takes a little bit of time when it comes to Jenkins Azure DevOps, which is the product we have at Microsoft, one of the features is called Pipelines, and Pipelines integrates with GitHub right out of the box, it integrates with Bitbucket, it integrates with Subversion so wherever your source code is today, chances are we can go ahead and grab it It’s absolutely free to get started For example, the 2 of us can invite 3 of our friends, go to dev.azure.com and create an account for free and all 5 of us can use everything that we have for free, including our CI system, work item tracking, our release management system, our testing framework, even our package management, all of it You get it all for free for the 5 of you, and then the next person you add is $6 a month Right? It’s just ridiculously cheap Now, if you’re using open source, you’re in GitHub and it’s a private repo, we give you 10 unlimited Pipelines for free, unlimited build minutes, and access to Mac OS, Windows, and Linux Now that’s really important because depending on the language you’re programming in or what machines you’re targeting for your customers They might be on Macs, they might be on Windows, or they might be on Linux, and you want to test your code over all those sort of things You don’t want to just test it on Windows and assume it works on Mac You want to test it on Mac, you want to test it on Linux, you want to test it on Windows, and to do that you’re going to have the machines running all those languages I don’t know of any other vendor that has all 3 platforms You’ll find vendors that have 2 and then you’ll have to go to another vendor for the 3rd one You might have a vendor for 1 of each of them, but now you’re making 3 different build scripts, right? Because if I use AppVeyor for 1, and Travis for 1, and Circles for 1, yeah they all use YAML, but the YAML isn’t the same, the way you speak isn’t the same and now I’m maintaining 3 different build scripts for 1 project – Wow – When you use Azure Pipelines, since we give you Mac, Windows, and Linux machines, it’s one definition, but you just tell us where do you want us to run it, and we’ll run it on all 3 of them or any choice of those that you want, so it makes it a lot easier, and again, if it’s open source, we’re the most generous open source offering in the world right now Because we just bought GitHub at Microsoft, right? So we’re like, come on, welcome to the family We’re going to obviously give you the best offer we could possibly give you from Microsoft If it’s open source, it’s free So Jenkins is hugely popular, you were asking Azure Pipelines is becoming very, very popular You’re also going to hear things like CircleCI, Travis, and AppVeyor What you can do is just go to any GitHub repository and you’re going to see these little badges on their Read Me file, and those badges are the CI systems that are compiling their code, and you’ll see multiple of them there and you’ll get a good idea of the most popular ones just from the badges you’re going to see on GitHub – Gotcha, gotcha. So it sounds like for, with Azure DevOps, it’s just easier to set-up and it integrates, it sounds like almost with everything immediately (laughter) – Pretty much! We’re trying to change the perception that we only work on Windows and .net, right? – Right – We can help you with your mobile application, we can help you with your Go lang, we can help you with your Python, we can help on your Mac, your Linux, or your Windows machine – Talk a little bit, because I’ve got to be honest, you’re exactly right, I’ve been under the impression for a long time, whenever I saw TFS and now Azure DevOps that it was a .net shop But you’re saying, any language it doesn’t matter, you guys can integrate, open source or not – I cannot remember the last time I did a .net demo on stage Can’t remember, right? I go up there and I either do Node or I’ll do Java, just to prove that point Just so that people can see that I can actually do any language and any platform There’s a video of me once, I did it once in Germany and I think I did it once in Australia, or no, in New Zealand, where I was so confident that I could do any language, any platform, that at the beginning of this talk, I let people go to an app, go to a website and vote, tell me what language you want me to program in, tell me what platform you want me to target, and at the end of the demo, I’m going to do whatever it is that you want me to do, and I just go off and start telling them about everything They’re voting, and I turn around and I literally had a Mac on the desk, a Mac, a PC, and a Linux machine They got to pick which machine I use, they got to pick what language I programmed in, and they got to pick where I deployed my code, like I don’t care what you pick This is what I’m here to prove. I can do any of it And over and over again, I just kept proving, whatever you want me to program at Microsoft we can help you – That is awesome Now you said you were in New Zealand? – I’ve been … Yes, I was in Auckland, New Zealand for Ignite 2 years in a row and I think it was the 2nd year that I went to Ignite in New Zealand, you can find that video where it’s the 1st time I do this I was so excited and just buzzing to get on stage and just, let me show you what I can do There’s a cool little app with a little South Park picture of me, it’s hilarious, and the little guy helps you vote and pick what you want me to do and at the end of it I’m like, alright let’s go – Let’s go! Alright, well I am going to put that in the show notes so people can go and click on that video because I think they’ll be very interested in it That’s awesome man Kind of to wrap it up, Donovan, as we talk about the end, what do you see as the future of CI? As you look through the next 2020 and beyond, what are we looking at as far as the features benefits, where do you think it’s going? – Yeah, one of the benefits has to be reaching all of the

different platforms at once We are gone from the days of … I work at Microsoft, I go visit a lot of our customers, very few of the customers I visit only do one language Right? They all do multiple languages They’re all into mobile development, so you’re going to have to make sure they have a platform that allows you to target Android and iOS and allows you to hit that Apple Watch For that, you’re going to have to have a Mac, right? And if you want to do some Windows development or some pure .net, not that core, you’re going to need a Windows machine for that If you’re doing .net core, you can now build your docker images inside of your build system but you’re probably going to need a Linux machine for that You can get away with it on other platforms, but So you want to make sure that you’re picking a platform that’s going to allow you to build on all of that, and the future is more and more of us are going to start providing you with every single platform that you need We’re just the 1st ones to do it Also, thinking about how you’re going to integrate every part of your product into your CI/CD pipeline A lot of times you’ll go to a customer and all they have is the front end, their website, but their database is still done manually, their mobile app is still done manually, it’s like come on, your entire solution needs to be in that pipeline I think that we’re going to see a lot of people enabling and making it much easier for you to do CI for your mobile apps, CI for your databases, CI for your docker images and stuff like that I think we’re ahead of the game there because we already allow you to do all that I am going to speak at VSLive here in New Orleans, and the demo that I do does everything that I just said with a single pipeline I build an Android and iOS application, a database project, a .net webservice, a .net core front end for docker I do it in 1 build because I have all 3 platforms in Azure DevOps, and I push 1 button and next think you know, I’m deploying containers, and databases, and mobile applications, it’s amazing So it’s going to be awesome – How much time, oh my goodness, how much time do you save by doing that? And just, the lack of mistakes, right? By automating everything? – Exactly. Exactly because you do that, a human being does that and I only have to mess up once, right? And I could bring down a company There’s a story about Knight Capital Group where a poor guy who was deploying to, I think, 8 servers, he did 7 of the 8 servers correctly, he screwed up on 1 server, he lost $464 million in 45 minutes and bankrupt his company That’s what happens when you send a human being to do a machine’s job A machine would have done that perfectly, 8/8 times This poor guy messed up 1 time, and lost half a billion dollars and bankrupt his company – [Ken] Oh my goodness – So that’s why CI is important Automate everything – There you have it, ladies and gentlemen! I think that is a great story to end it at Alright, Donovan, anything you want to add as we wrap it up, anything you want to tell the audience out there, any last words about CI? – Yeah, implement it! Stop asking for permission! If you need help, if you need help setting it up, if you get stuck, you can actually find me on Twitter I’m @DonovanBrown My entire team, we’re called The League We’re all on Twitter, we’re on Twitter all the time If you use a hashtag, #LoECDA, if you use that hashtag in your tweet, all 5 of us will come and read it, and if we don’t know the answer, we’ll @ someone from Microsoft from the product group that can answer your question for you So, if you don’t have CI or you want to switch CI, come and reach out to us on Twitter and we’ll come help you – Say that hashtag one more time? – #LoECDA. It stands for the League of Extraordinary Cloud DevOps Advocates – LoECDA, I like it It’s probably going to get a crazy influx of #LoECDAs – (laughter) Yeah we’ve had it used, I think the maximum was 900 times in a month I think was the highest it’s every been – God, love it man But Donovan, thank you so much for your time, my good man I appreciate it greatly! – My pleasure – Alright man, take care Alright, there you have it Thanks Donovan for being a great guest and hopefully everyone learned a lot about continuous integration, different types of it, and it helped to give you a little more knowledge there So as always, if you’re interested in watching more videos of DevOps Defined, take a look right here as I’m including other videos you can watch from the different series, and as always, if you’re looking for more career tips when it comes to DevOps, I’ve included them here So as always, thank you everybody for listening Hopefully they helped a little bit and happy continuous improvement