One year ago we set ourselves the task of doing Continuous Deployment on the SAAS environments running our application TOPdesk. Today we’ve improved our release rate from once every 6 months to once every week. These are the steps that brought us success.
Step 1) Set a clear and ambitious goal
In June 2015 we started out by setting a clear and ambitious company goal:
80% of our customers’ environments have to be updated weekly starting from May 2016.
Setting this goal really helped us to move forward in two ways. First, it changed the discussions from “Should we do this?” to “How can we do this?”. These discussions were much more fruitful and helped us resolve a lot of our problems. Secondly, it helped us align people throughout the organization. Doing Continuous Deployment is not a Development or Operations effort. It affects how you sell, implement and support your product. We’ve spent quite some time getting everybody on the same page.
Step 2) Automate, but not everything
To do Continuous Deployment you need automation. We’ve spent our efforts in a few directions. The first was quality control. With a 6 month release cycle we relied on manual regression tests to ensure a high quality product. We had to get rid of our dependency on manual labor at the time of the release. To achieve this we gathered the 100 most critical workflows in our product and created automated tests for those. To determine what to test we looked at where we had the biggest risks, what the fragile parts are and what potential mistakes have a high impact. This gave us much more certainty about the quality, at the places where it is needed. We also skipped some tests that we used to perform, which turned out not to be very useful. The automation effort also made sure that product owners and teams got used to the new situation where test automation is the way we guarantee quality on delivery. In January we performed our last manual regression test of the full product.
Another aspect that was automated was our deployment pipeline. We already had continuous integration in place that helped us by providing packaged versions of TOPdesk. These packages were used for on premises installations and for deployment on our SAAS environment. Our SAAS operations team had automated the deployment of our product on production machines. Between packaging and deployment a lot of manual steps had to be taken. An example is determining which customer should be updated when. Our efforts focused on removing these steps.
Finally, we worked on getting the right information available at the right places. Examples are tracking version numbers in use by our customer, so we can optimally support them, and automatically generating release notes from our issue tracker. Deploying more often forced us to rethink everything we did manually.
Step 3) Iterate, iterate, iterate
To get to the point where we are, deploying 900 of our 1300 environments automatically, we had to learn a lot. We chose to do this step by step to gradually reduce the manual work, and increase our confidence in the quality of the product and automated deployment.
We reduced risks by starting with the release of just small bug fixes, this helped us to smoothen out problems in our deployment pipeline. After that we deployed bigger functional changes in a staged fashion to gain confidence in the stability of our product. We start by updating our internal production environments, after that we update the test environments of our Beta customers, then that their production environments and finally all remaining environments. With every step we learned what to resolve next.
What’s next
In less than a year we’ve made a bigger leap forward than some people thought was possible. Still, we are on a journey. A small percentage of our customers cannot automatically be updated for various reasons. We need to continue taking these reasons away. Getting the deployment cycle even shorter is another challenge ahead. Inevitably we are going to make mistakes at some point. This is ok, as long as we resolve them quickly and learn from them.
What were the lessons that you had to learn when you started doing Continuous Deployment? What topics did I touch upon that you would like to know more about? Please comment to let me know.