{"id":1982,"date":"2011-02-06T16:13:31","date_gmt":"2011-02-07T00:13:31","guid":{"rendered":"https:\/\/mathpirate.net\/log\/?p=1982"},"modified":"2011-02-06T16:13:31","modified_gmt":"2011-02-07T00:13:31","slug":"continuous-integration-and-you","status":"publish","type":"post","link":"https:\/\/mathpirate.net\/log\/2011\/02\/06\/continuous-integration-and-you\/","title":{"rendered":"Continuous Integration and You"},"content":{"rendered":"<p>Continuous Integration is probably the most important development in team programming since the rise of source control systems.\u00c2\u00a0 It&#8217;s not a recent development, either, but while source control usage is universal, Continuous Integration tends to meet with calls of &#8220;It&#8217;s too hard&#8221; and &#8220;We don&#8217;t need it yet&#8221;.\u00c2\u00a0 Even in my company, where the benefits of Continuous Integration are obvious and clear, there always seems to be some resistance or avoidance when it&#8217;s time to put a project into the system.<\/p>\n<p>Well, it&#8217;s not that hard ((You people know seven programming languages, yet you&#8217;re afraid of a build script?\u00c2\u00a0 It&#8217;s not hard.\u00c2\u00a0 You&#8217;re just lazy.)), and yes, you do need it right now.<\/p>\n<p>Before I begin, I work at a web company.\u00c2\u00a0 Most of our software\u00c2\u00a0are fairly small (code-wise) sites, services and\u00c2\u00a0tools, that sort of thing.\u00c2\u00a0 We don&#8217;t really have any monolithic enterprise systems that take\u00c2\u00a0seven hours to compile with thousands of dependencies.\u00c2\u00a0 This posting is written from my experience in this world.\u00c2\u00a0 If that&#8217;s not your world, some of these comments might not apply to you, although the basic concepts should still be able to translate into what you do.<\/p>\n<p>Continuous Integration, in its most basic form, is a system that watches source control check-ins, and when it detects a change, will build the code.<\/p>\n<p>And deploy the code.<\/p>\n<p>And run the automated tests against the code.<\/p>\n<p>And build other projects that depend on the module that was just built, and deploy and test those, too.<\/p>\n<p>You may have heard of Continuous Integration before, but didn&#8217;t look into it because you think it&#8217;s an &#8220;Agile process&#8221; and that you have to be doing test-driven design and extreme programming and all of that other nonsense in order for it to work.\u00c2\u00a0 Well, you don&#8217;t have to be doing Agile to do this.\u00c2\u00a0 Even if you&#8217;re in a barrel and going over a hard-core ISO-9000 Waterfall, you&#8217;ll benefit from at least part of it.\u00c2\u00a0 Hell, I&#8217;ve even set it up for some of my own personal projects, just so I don&#8217;t have to deal with the hassles of manual deployment.<\/p>\n<p>Here&#8217;s why Continous Integration is\u00c2\u00a0Awesomeness:<\/p>\n<ul>\n<li><strong>Broken code detected automatically and immediately.<\/strong>\u00c2\u00a0 Accidentally check in a file with only half a function written?\u00c2\u00a0 Forget to check in a new file or library?\u00c2\u00a0 Within a few minutes, CI will fire off a build and choke, so you&#8217;ll know right away\u00c2\u00a0that there was a problem with what you just checked in.\u00c2\u00a0 You don&#8217;t have the case where two days later, someone else checks out your code, discovers that it&#8217;s broken, then wastes an hour trying to figure out what&#8217;s wrong before calling you over to help.\u00c2\u00a0 ((Except they can&#8217;t, because you&#8217;re in Cancun for the next two weeks.))\u00c2\u00a0 Plus, there&#8217;s the &#8220;public shaming&#8221; factor working in favor of quality software.\u00c2\u00a0 When someone breaks the build, everyone finds out about it, so people tend to be more careful about what they check in.<\/li>\n<li><strong>Automatic deployments.<\/strong>\u00c2\u00a0 If you&#8217;ve got web sites or services, you can&#8217;t use them unless they&#8217;re running somewhere.\u00c2\u00a0 Obviously, you can install the site by hand every couple of days, but then\u00c2\u00a0you have to go through the tedious steps of logging in, finding the latest build package,\u00c2\u00a0uninstalling the old code and installing the new code, repeating those steps for every machine you need the code on.\u00c2\u00a0 You could easily waste up to an hour or two a day doing this.\u00c2\u00a0 (On top of that, you&#8217;re running code that&#8217;s out of date most of the time.)\u00c2\u00a0 CI can do it for you.\u00c2\u00a0 Your machines are\u00c2\u00a0always up to date and you&#8217;re not wasting any of your time to make that happen.\u00c2\u00a0 Best of all, pesky testers won&#8217;t bug you asking for an updated build or begging you to push your latest fixes.<\/li>\n<li><strong>Automatic testing.<\/strong>\u00c2\u00a0 There&#8217;s test automation, then there&#8217;s automated test automation.\u00c2\u00a0 Automated tests that don&#8217;t run on their own don&#8217;t run often enough.\u00c2\u00a0 You&#8217;ve gone to all the trouble of making a computer do all the work of testing for you, so why do you still have it where someone has to push the button to make it go?\u00c2\u00a0 Think that&#8217;s job security for you or something?\u00c2\u00a0 If you have automated tests, make them run in a CI system.\u00c2\u00a0 That way, the developers check in their code and a few minutes later, they get an e-mail that tells them if anything broke, and you don&#8217;t have to do anything.\u00c2\u00a0 You don&#8217;t even have to be in the office!<\/li>\n<li><strong>Always up-to-date.<\/strong>\u00c2\u00a0 Testers are always looking at the latest code, so they&#8217;re not filing bugs about things you fixed last week.\u00c2\u00a0 The upper-ups are always looking at the latest code, so they&#8217;re not complaining about things that you fixed last week.\u00c2\u00a0 And the teams that rely on your component are using the component as it actually is, not as you designed it two months ago.<\/li>\n<li><strong>Code builds are clean.<\/strong>\u00c2\u00a0 Everything&#8217;s built automatically out on the build servers, not on your machine, where you&#8217;ve modified config files and replaced critical libraries with the debug versions.\u00c2\u00a0 The build servers are clean, and becaue what they&#8217;re building is constantly being deployed and tested, you know that the build packages they&#8217;re producing work.<\/li>\n<li><strong>Integration and deployment aren&#8217;t afterthoughts.<\/strong>\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 In ages past, I worked on a project which had a three month release cycle.\u00c2\u00a0 The first month and a half was strictly local\u00c2\u00a0development, on the developer&#8217;s machines.\u00c2\u00a0 Nothing was running anywhere other than the Cassini server in\u00c2\u00a0Visual Studio.\u00c2\u00a0 Like many systems, there was a front end and a back end.\u00c2\u00a0 Of course, the back end wasn&#8217;t running anywhere, so the front end was using a mock service that worked like the actual back end would, in theory, at least.\u00c2\u00a0 Then came the day of the first drop.\u00c2\u00a0 Developers spent hours building and installing everything, then turned it over to QA and went home.\u00c2\u00a0 The next day, QA was only able to log a single bug:\u00c2\u00a0 &#8220;<em>[Drop 1] NOTHING WORKS. (Pri1\/Sev1)<\/em>&#8220;\u00c2\u00a0 ((And, of course, QA got blamed for slipping the release date because they didn&#8217;t get their testing done according to schedule.))\u00c2\u00a0 You see, while everything worked according to the spec, in theory, none of the developers actually talked to one another, so neither side mentioned how they filled in the holes in the spec.\u00c2\u00a0 The back end was using a different port, because the original port conflicted with something, and the front end was using lowercase parameter names and expecting a different date format.\u00c2\u00a0\u00c2\u00a0 This sort of trainwreck deployment disaster doesn&#8217;t happen when you have continuous integration in place.\u00c2\u00a0 Not because there aren&#8217;t miscommunications, but because those miscommunications are discovered and resolved\u00c2\u00a0immediately, so they don&#8217;t pile up.\u00c2\u00a0 The front end developers point their code at the live work-in-progress back end servers.\u00c2\u00a0 If there&#8217;s a port change or a date format mismatch, it can be detected and corrected right away.\u00c2\u00a0<\/li>\n<li><strong>Testers get something to test immediately.<\/strong>\u00c2\u00a0 In the story above, the testers were forced to try to write their tests against a phantom system for the first month and a half, based on partial technical specs and a handful of incomplete mockups.\u00c2\u00a0 After that, we were writing tests on code that was out of date.\u00c2\u00a0 It doesn&#8217;t have to be like that.\u00c2\u00a0 With Continuous Integration,\u00c2\u00a0 whatever is checked in is deployed and running somewhere, making it at least partially testable.\u00c2\u00a0 Even if there&#8217;s no functionality, at the very least, there is a test that the deployment package works.\u00c2\u00a0 The argument that testers can just\u00c2\u00a0pull the code out of the repository and run the service or website on their local machine and run all of their tests against that is ridiculous.\u00c2\u00a0 It&#8217;s a waste of time, because every tester has to pull down the code, configure their machines, build the code, then bug the developers for an hour or two\u00c2\u00a0because things aren&#8217;t working right.\u00c2\u00a0 Even after all that, there&#8217;s no telling whether some bugs are actual bugs, or just byproducts of the way it&#8217;s set up on a particular tester&#8217;s machine.\u00c2\u00a0 Worse still, the tests that are written aren&#8217;t going to be run regularly, because there&#8217;s no stable place to run them against.<\/li>\n<li><strong>Deployment packages and test runs are easily archived.<\/strong>\u00c2\u00a0 The system can automatically keep a record of everything that it does.\u00c2\u00a0 Need to compare the performance of builds before and after a change that was made two weeks ago?\u00c2\u00a0 You&#8217;ll be able to do that, because you still have the .MSIs archived for every build over the last two months.\u00c2\u00a0 Need to find out when a particular piece of functionality broke?\u00c2\u00a0 You&#8217;ll be able to do that, because the tests ran with almost every check-in and the results were saved.\u00c2\u00a0 You don&#8217;t have drops and test passes that are done two weeks and a hundred check-ins apart.<\/li>\n<li><strong>Progress is always visible.<\/strong>\u00c2\u00a0 Ever have a manager above you, sometimes far above you, want to see your progress?\u00c2\u00a0 Ever have to race to get something set up to demo?\u00c2\u00a0 CI takes care of that.\u00c2\u00a0 You&#8217;re always deploying with every check in.\u00c2\u00a0 A demo of the current state is always a link away.<\/li>\n<li><strong>It saves time and saving time saves money.<\/strong>\u00c2\u00a0 Use this point with skeptical management.\u00c2\u00a0 The first time you set up a project in a CI system, yes, you&#8217;ll spend several days getting everything running.\u00c2\u00a0 However, if you add up all the time that your team will waste without an automated build\/deploy\/test\u00c2\u00a0system,\u00c2\u00a0it will easily pay for itself.\u00c2\u00a0 You won&#8217;t spend any time on getting a build ready, deploying the build, firing off the automated tests, reporting on the automated tests, trying to get something you&#8217;re not working on set up on your box, explaining how to set up your code on someone else&#8217;s box, creating a demonstration environment\u00c2\u00a0to show off what you&#8217;ve done\u00c2\u00a0to the upper-ups after they asked about your progress in a status meeting, integrating with a service that&#8217;s been under development for months but that you&#8217;ve never seen, trying to figure out what check in over the last three weeks broke a feature, wasting a morning because a coworker checked in broken code, or being yelled at by coworkers for checking in broken code and causing them to waste their morning.\u00c2\u00a0 And, of course, the second time you set up a project in Continuous Integration, you can copy and paste what you did the first time, then only spend a fraction of the time working out the kinks.\u00c2\u00a0 It gets easier with every project you do.\u00c2\u00a0<\/li>\n<\/ul>\n<p>Like almost anything that&#8217;s good, there are some points where it&#8217;s painful.<\/p>\n<ul>\n<li><strong>When the build goes bad, people can be blocked.<\/strong>\u00c2\u00a0 Everyone&#8217;s relying on the code being deployed and functional.\u00c2\u00a0 Now that you have an automatic push of whatever you&#8217;ve checked in, there&#8217;s always the chance that that what you&#8217;ve checked in is garbage, leading to garbage getting pushed out.\u00c2\u00a0 However, the flip side to this is that the problem is exposed immedately, and there&#8217;s a very real and pressing need to fix the problem.\u00c2\u00a0 Otherwise, the blocking issue may have gone undetected for days, perhaps weeks, where it would be far more difficult to track down.<\/li>\n<li><strong>It never works right the first time.<\/strong>\u00c2\u00a0 Never.\u00c2\u00a0 Whenever you set up a project in your Continuous Integration system or make major changes to an existing project, it won&#8217;t work.\u00c2\u00a0 You&#8217;ll spend an hour tracking down little problems here and there.\u00c2\u00a0 Even if you get the build scripts running on your local machine, they&#8217;ll fail when you put them out on the build server.\u00c2\u00a0 It&#8217;s just part of the territory.\u00c2\u00a0 However, tweaking the build to work on a different machine does mean that you&#8217;re more aware of the setup requirements, so when it comes time to ramp up the new guy or to deploy to production, you&#8217;ll have a better idea of what needs to be done.<\/li>\n<li><strong>Everyone knows when you screw up.<\/strong>\u00c2\u00a0 Everyone gets the build failure e-mail.\u00c2\u00a0 So don&#8217;t screw up.\u00c2\u00a0 Where I work, we&#8217;ve taken it a step further.\u00c2\u00a0 We used to have a &#8220;Build Breaker&#8221; high score screen, which would keep track of who had left builds broken the longest.\u00c2\u00a0 When the build broke, everyone involved would make sure they fixed it as soon as they could, so they&#8217;d avoid getting the top score.\u00c2\u00a0 However, we also have a watcher set up that will send an e-mail to the specific person responsible for the build break, letting them know that something went wrong.<\/li>\n<\/ul>\n<p>Now you know why it&#8217;s good to set up Continuous Integration, as well as some of the things that will go wrong when you do.\u00c2\u00a0 So, all that&#8217;s left is for me to give you some advice based on what I&#8217;ve learned in my own experience.\u00c2\u00a0 I&#8217;m not going to go so far as to say that these are Continuous Integration Best Practices or anything like that, just some tips and tricks you might find helpful.\u00c2\u00a0 ((Plus, I just wanted to say &#8220;Continuous Integration Best Practices&#8221; a couple of times\u00c2\u00a0to get more hits.\u00c2\u00a0 Continuous Integration Best Practices.\u00c2\u00a0 Okay, that oughta be\u00c2\u00a0enough.))<\/p>\n<ul>\n<li><strong>Do it.<\/strong>\u00c2\u00a0 Just do it.\u00c2\u00a0 Stop complaining about how hard it is or how it takes so much time to set up or how we don&#8217;t really need it at this point in the project.\u00c2\u00a0 Do it.\u00c2\u00a0 DO IT NOW.\u00c2\u00a0 It&#8217;s easier to do it earlier in the project, and the overall benefit is greater.<\/li>\n<li><strong>Add new projects to CI immediately.<\/strong>\u00c2\u00a0 Create the project, check it in, and add it to CI.\u00c2\u00a0 Don&#8217;t wait, don&#8217;t say &#8220;I&#8217;ll do it when I&#8217;ve added functionality X&#8221;.\u00c2\u00a0 Get it in the system as soon as you can.\u00c2\u00a0 That way, it&#8217;s out of the way and you won&#8217;t try to find some excuse for putting it off later when the testers are begging you to finally do it.\u00c2\u00a0 I view CI as a fundamental part of collaborative software development and the responsibility of any developer who sets up a new project.\u00c2\u00a0 If you&#8217;re a tester and the devs won&#8217;t do it themselves, do it for them.\u00c2\u00a0 You need it.\u00c2\u00a0 Trust me, you do.<\/li>\n<li><strong>Use what makes sense.<\/strong>\u00c2\u00a0 Where I work, we use <a href=\"http:\/\/confluence.public.thoughtworks.org\/display\/CCNET\/Welcome+to+CruiseControl.NET\">CCNet<\/a> and <a href=\"http:\/\/nant.sourceforge.net\/\">nant<\/a>.\u00c2\u00a0 Not necessarily because they&#8217;re the best, but because they&#8217;re what we&#8217;ve used for years and they make sense for us.\u00c2\u00a0 make is ancient and confusing,\u00c2\u00a0we don&#8217;t use\u00c2\u00a0Ruby for anything, so Rake would be a bit outside of our\u00c2\u00a0world.\u00c2\u00a0 If you can do what you need to do with a batch file and a scheduled task, then go for it.\u00c2\u00a0 Although, trust me, you&#8217;re going to want something that can handle all of the automatic source control checkout stuff and allow wide flexibility in how it fires off builds.\u00c2\u00a0 And don&#8217;t pay for anything.\u00c2\u00a0 Seriously, there&#8217;s free stuff out there, you don&#8217;t need to be paying thousands of dollars a year just because something that has a great sales pitch.<\/li>\n<li><strong>Consistency is the key.<\/strong>\u00c2\u00a0 On a major project\u00c2\u00a0about a\u00c2\u00a0year and a half\u00c2\u00a0ago, we standardized the set up of our projects and build scripts.\u00c2\u00a0 Ever since then, we&#8217;ve reused that model on a number of other projects, and it has been amazingly effective.\u00c2\u00a0 As long as we give parts of our projects standard names (<em>project<\/em>.Setup, <em>project<\/em>.Tests, etc.), we can copy and paste a small configuration file, change a few values in it, and in just a few minutes,\u00c2\u00a0we have that project being built, deployed, and tested in our CI system.\u00c2\u00a0 An additional benefit is that\u00c2\u00a0our projects now\u00c2\u00a0all have a similar organization, so any part of the code you look at will have a sense of familiarity.<\/li>\n<li><strong>Flexibility is also the key.<\/strong>\u00c2\u00a0 While we&#8217;ve gotten enormous gains from standardizing what we can standardize, there&#8217;s always going to be something that has to be different.\u00c2\u00a0 Don&#8217;t paint yourself into a corner by forcing everything you put in the system to act in exactly\u00c2\u00a0the same way.\u00c2\u00a0 However, don&#8217;t be too flexible.\u00c2\u00a0 If someone&#8217;s trying to take advantage of the flexibility in the system because they&#8217;re too lazy to be consistent, then, by all means, break out the Cane of Conformity and make them play by the rules.<\/li>\n<li><strong>Your build system is software, too.<\/strong>\u00c2\u00a0 Don&#8217;t think of it as just a throwaway collection of files to get the job done.\u00c2\u00a0 It isn&#8217;t.\u00c2\u00a0 It&#8217;s an integral part of your software.\u00c2\u00a0 It might not be deployed to production or shipped to\u00c2\u00a0the customer, but everything that is deployed to production or shipped to the\u00c2\u00a0customer will run through it.\u00c2\u00a0 Do things that make sense.\u00c2\u00a0 If you can share\u00c2\u00a0functionality, share it.\u00c2\u00a0 If you can parameterize something and reuse it, do that.\u00c2\u00a0 Check in your build scripts and CI configuration.\u00c2\u00a0 Remember, you&#8217;re going to have to update, extend\u00c2\u00a0and maintain your build system as you go, so it&#8217;s in your best interest to spend the time to make a system that&#8217;s simple to update, extend and maintain.<\/li>\n<li><strong>Be mindful of oversharing build scripts.<\/strong>\u00c2\u00a0 You want to try to make your build scripts and CI configs so that they&#8217;re modular and reusable, but be careful that you don&#8217;t share too many things or share\u00c2\u00a0the wrong things between unrelated projects.\u00c2\u00a0 At my company, we have a handful of teams, and most of them have one or more build servers.\u00c2\u00a0 At one point, one of the teams was reorganized into several smaller teams, and sent to work on wildly divergent projects.\u00c2\u00a0 However, they continued to share a single\u00c2\u00a0library\u00c2\u00a0of build scripts.\u00c2\u00a0 Some time later, someone made a change to the one of the scripts in the library that he needed for his project.\u00c2\u00a0 His project on his server worked just fine.\u00c2\u00a0 Then, two weeks later, every other server that used these shared scripts began to fail randomly.\u00c2\u00a0 No one had any idea what was going on, and it took several hours to trace the problem back to the original\u00c2\u00a0change.\u00c2\u00a0 This illustrates the danger of sharing too widely.\u00c2\u00a0 You want to try to structure the build script libraries in such a way that changes are isolated.\u00c2\u00a0 Perhaps you can version them, like any other library.\u00c2\u00a0 Or, like we&#8217;ve done for most of our projects, copy the build script libraries locally into the project&#8217;s directory, so everyone&#8217;s referencing their own copy.\u00c2\u00a0 ((However you choose to organize it, sticking project specific e-mail addresses in a shared file that everyone references is a dumb idea.\u00c2\u00a0 There&#8217;s nothing shared about it.\u00c2\u00a0 Don&#8217;t force me to rebuild everything just because you&#8217;ve hired a new intern.))<\/li>\n<li><strong>Check in your build scripts and CI configuration.<\/strong>\u00c2\u00a0 I know I just said that in\u00c2\u00a0a\u00c2\u00a0point above, but it&#8217;s important enough that it deserves its own bullet.\u00c2\u00a0 Your build scripts are an important piece of your software, so when they change, you need to rebuild your code, in order to make sure that the process still works.\u00c2\u00a0 You want them checked in for the same reasons the rest of your source code is checked in.\u00c2\u00a0 We even have our CCNet.config checked in, which means that our Continuous Integration systems themselves are in CI.\u00c2\u00a0 In other words, CCNet will detect a change to the CCNet.config that&#8217;s checked in, pull down the latest version, and restart itself to pick up the changes.\u00c2\u00a0 Under normal circumstances, we never have to touch the build server.\u00c2\u00a0 We just check in our changes and everything is automatically picked up.<\/li>\n<li><strong>Don&#8217;t touch the build server.<\/strong>\u00c2\u00a0 Obviously, you&#8217;ll have to set up the box, and once in a while, there&#8217;s some maintenance to be done, but for day to day operations, don&#8217;t log on to\u00c2\u00a0the box.\u00c2\u00a0 In fact, don&#8217;t even manually check anything out other than the CI configuration file.\u00c2\u00a0 Everything that needs to be checked out should be checked out automatically by the CI system.\u00c2\u00a0 This even extends to tools, where possible.\u00c2\u00a0 We&#8217;ve got versions of nant and other command line tools checked into SVN, and any build that\u00c2\u00a0uses them is responsible for checking them out.\u00c2\u00a0 One of the benefits of this is that it makes it easy to move the builds to a different server in an emergency if something goes wrong.\u00c2\u00a0 If any of our build servers dies, then we can probably get\u00c2\u00a0things back up and running on an alternate server in about half an hour.<\/li>\n<li><strong>PSExec is your friend.<\/strong>\u00c2\u00a0 If you&#8217;re doing stuff on Windows systems, there&#8217;s a tool from <a href=\"http:\/\/technet.microsoft.com\/en-us\/sysinternals\">Sysinternals<\/a> called &#8220;PSExec&#8221;, which makes it fairly straightforward to run a command on a remote machine.\u00c2\u00a0 Get it.\u00c2\u00a0 Use it.\u00c2\u00a0 Love it.<\/li>\n<li><strong>Every build should check out everything it needs.<\/strong>\u00c2\u00a0 Every build should be able to run from scratch.\u00c2\u00a0 ((Well, almost every build, at least.\u00c2\u00a0 Obviously a test\u00c2\u00a0run is going to need the\u00c2\u00a0project it&#8217;s testing to have been built first.))\u00c2\u00a0 Every build should check out all the code it needs, all the dependencies, all the tools, all the data files.\u00c2\u00a0 It should be possible for you to go on to the build box, wipe out the source tree, and have all of your builds succeed the next time they run.\u00c2\u00a0 In fact, I&#8217;d recommend intentionally deleting the source tree from time to time, just to may sure that everything is self sufficient.\u00c2\u00a0 The reason for this is that every build should be using the latest code.\u00c2\u00a0 If\u00c2\u00a0Build A depends on a library that only\u00c2\u00a0Build B checks out, then there&#8217;s a chance that Build A will run before Build B updates, leaving Build A\u00c2\u00a0potentially out of\u00c2\u00a0date and in an unknown state.\u00c2\u00a0 Yes, this requirement often means that there are certain libraries or files that are included by pretty much everything, so even a simple change to those libraries will cause a build storm where everything gets rebuilt.\u00c2\u00a0 People hate these, but think about it:\u00c2\u00a0 You&#8217;ve changed a core dependency, therefore everything has to rebuild because everything was changed.<\/li>\n<li><strong>Only check out what you need.<\/strong>\u00c2\u00a0 Target the checkouts to be only the directories that you need for that particular build.\u00c2\u00a0 Don&#8217;t have every project watching the root of your repository, because then you&#8217;ll get builds for every single check in, no matter how unrelated the check in was.<\/li>\n<li><strong>Don&#8217;t set up your CI system with a normal employee account.<\/strong>\u00c2\u00a0 You want your CI system to run under an account that won&#8217;t go on vacation or get fired.<\/li>\n<li><strong>Fail builds when unit tests fail.<\/strong>\u00c2\u00a0 If you&#8217;re doing unit tests, you want those tests to be running as part of the CI build.\u00c2\u00a0 You also want them to fail the build.\u00c2\u00a0 The philosophy of unit testing is that when a unit test breaks, your code is broken and you need to fix the issue immediately.\u00c2\u00a0 This will very strongly encourage developers to make sure that the unit tests are maintained and in good working order and that their new code doesn&#8217;t break anything.<\/li>\n<li><strong>Don&#8217;t fail builds when functional\/regression\/integration tests fail.<\/strong>\u00c2\u00a0 It&#8217;s generally expected that at least some of your functional or regression tests will fail with every build.\u00c2\u00a0 If you don&#8217;t have at least a handful of regression tests failing due to open bugs in the software, then you need more tests.\u00c2\u00a0 Where I work, the functional test builds only fail when there is a problem when building or executing\u00c2\u00a0the tests, not when one of those tests fail.<\/li>\n<li><strong>Don&#8217;t deploy if a build fails.<\/strong>\u00c2\u00a0 Deployment should be one of the last steps in a build and should only be performed after the build and unit tests (if you have them) are successful.\u00c2\u00a0 Don&#8217;t wipe out a previous build that was successfully deployed with something that you know is broken.<\/li>\n<li><strong>Don&#8217;t archive deployment packages if\u00c2\u00a0a build\u00c2\u00a0fails.<\/strong>\u00c2\u00a0 If the build breaks or the unit tests die or the deployment doesn&#8217;t work, don&#8217;t archive the deployment package.\u00c2\u00a0 This will ensure that any MSI that&#8217;s saved is of a build that passed all the unit tests and was installed successfully.<\/li>\n<li><strong>Split your builds into logical components.<\/strong>\u00c2\u00a0 If possible, avoid having a monolithic build that will build everything in your company.\u00c2\u00a0 The bigger a build is, the more stuff that&#8217;s included, the more chances for the build to go wrong and the longer a build will take.\u00c2\u00a0 You want to aim for quick and small builds for faster feedback.\u00c2\u00a0 It&#8217;s fine if you have cascading builds, where one build triggers the next, as long as the project that was actually\u00c2\u00a0changed is built first in order to give immediate feedback.<\/li>\n<li><strong>Don&#8217;t filter build failure mails.\u00c2\u00a0 EVER.<\/strong>\u00c2\u00a0 Build failure notices are vital to the health of a CI system.\u00c2\u00a0 When a build breaks, you need to know about it.\u00c2\u00a0 However, I&#8217;ve seen a lot of people who simply set up a mail rule that forwards all mail from the build server into a folder that they never look at.\u00c2\u00a0 DON&#8217;T DO THAT.\u00c2\u00a0 It&#8217;s fine if you filter the successes away, but the failures should be front and center.\u00c2\u00a0 If anything, you need to set up a rule that flags failures as important.\u00c2\u00a0 I have a mail rule that specifically filters mails from the build server only when the subject line contains &#8220;build successful&#8221; and does not contain &#8220;RE:&#8221; or &#8220;FW:&#8221;, etc.<\/li>\n<li><strong>Fix your broken builds.\u00c2\u00a0 NOW.<\/strong>\u00c2\u00a0 Don&#8217;t wait.\u00c2\u00a0 Fix it.\u00c2\u00a0 Now.\u00c2\u00a0 NOW!\u00c2\u00a0 When a build breaks because of you, it should leap to the front of your priority queue.\u00c2\u00a0 Do not go to lunch, do not go for coffee, do not pass go, do not collect $200.\u00c2\u00a0 FIX. IT. NOW.<\/li>\n<li><strong>Don&#8217;t go home without making sure your builds are successful.<\/strong>\u00c2\u00a0 And especially don&#8217;t go on vacation.<\/li>\n<li><strong>Sometimes, broken builds aren&#8217;t the end of the world.<\/strong>\u00c2\u00a0 Sometimes it&#8217;s okay to check in something you know will break the build.\u00c2\u00a0 If you&#8217;re doing a major refactor of part of the code, it&#8217;s fine to do it in stages.\u00c2\u00a0 Go ahead and check in bits and pieces as you go.\u00c2\u00a0 In fact, in some source control systems, it would be suicidal to attempt to pile up all of the renames, moves, and deletes into a single check in at the end.\u00c2\u00a0 In some cases, you might even want to disable a particular build while you make changes.\u00c2\u00a0 (Just make sure to turn it back on again.)<\/li>\n<\/ul>\n<p>Finally, and most importantly, stop your whining, stop your objections, stop reading this post, and go set up your Continuous Integration system immediately.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Continuous Integration is probably the most important development in team programming since the rise of source control systems.\u00c2\u00a0 It&#8217;s not a recent development, either, but while source control usage is universal, Continuous Integration tends to meet with calls of &#8220;It&#8217;s too hard&#8221; and &#8220;We don&#8217;t need it yet&#8221;.\u00c2\u00a0 Even in my company, where the benefits [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[7,34],"tags":[56,183,182,184,36],"_links":{"self":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts\/1982"}],"collection":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/comments?post=1982"}],"version-history":[{"count":18,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts\/1982\/revisions"}],"predecessor-version":[{"id":2001,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts\/1982\/revisions\/2001"}],"wp:attachment":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/media?parent=1982"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/categories?post=1982"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/tags?post=1982"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}