Example of a Continuous delivery pipeline

Continuous delivery has become a juicy problem again with the steer towards Virtual Machines as an Artefact and Operating System containers as an artefact. To put it short, your Vagrant managed VirtualBox/Docker.io style applications. I would highly recommend to read about Vagrant with Virtual Box and Vagrant with Docker.io before we continue.

Firstly I am assuming you have a series of Environments, for example Development, Staging and Production. I believe in the concept of Genderless applications where environment specific configuration is provisioned at deployment, you will see why this will become important!




1.       Build
a.       Compile code
b.       Run unit tests
c.        Run component tests / Small integration tests
d.       Build WAR archive (Or similar)
2.       Package
a.       Destroy Virtual Machine
b.       Copy WAR from Build stage to working directory
c.        Initialize new Virtual Machine
d.       Start the Virtual Machine
e.       Provision binary dependencies (e.g. JDK 1.7, Jetty 9)
f.        Deploy WAR to Virtual Machine
g.       Package Virtual Machine
3.       Deploy – DEV
a.       Copy packaged Virtual Machine from Package stage
b.       Deploy constructive SQL Changes
c.        Deploy to inactive Pool (For N number of Boxes)
                                                               i.      Deploy packaged Virtual Machine
                                                             ii.      Provision with DEV Environment configuration 
                                                           iii.      Start application
                                                            iv.      Smoke test/Query Health check URL of Application
d.       Smoke test pool
e.       Route traffic to new pool
f.        Deploy destructive SQL Changes
4.       System Test – DEV
a.       Run Acceptance tests against deployed pool
5.       Publish
a.       Copy packaged Virtual Machine from Package stage or Deploy – DEV Stage
b.       Publish packaged Virtual Machine to artefact repository
6.       Deploy – Staging
a.       Download packaged Virtual Machine from repository
b.       Deploy constructive SQL Changes
c.        Deploy to inactive Pool (For N number of Boxes)
                                                               i.      Deploy packaged Virtual Machine
                                                             ii.      Provision with Staging Environment configuration
                                                           iii.      Start application
                                                            iv.      Smoke test/Query Health check URL of Application
d.       Smoke test pool
e.       Route traffic to new pool
f.        Deploy destructive SQL Changes
7.       Deploy – Production
a.       Download packaged Virtual Machine from repository
b.       Deploy constructive SQL Changes
c.        Deploy to inactive Pool (For N number of Boxes)
                                                               i.      Deploy packaged Virtual Machine
                                                             ii.      Provision with Production Environment configuration 
                                                           iii.      Start application
                                                            iv.      Smoke test/Query Health check URL of Application
d.       Smoke test pool
e.       Route traffic to new pool
f.        Deploy destructive SQL Changes

There is a lot of information to take in here, but most of it is rinse and repeat deployments! Isn’t that great!

Back to my statement of genderless applications, due to the size of Virtual Machines are considerably larger than a WAR file, you wouldn’t want to package and publish the same application per environment. This is a wasteful and slow process!

If you haven’t noticed, this process is quite messy. When we re-route traffic in the deployment stages, we have not cleaned up the old pool. I will leave this up to you when you wish to do this! Just be cautious on the speed and effectiveness of your pipeline. To me, a good continuous delivery pipeline should be reliable, simple, repeatable, and fast. Complex, unstable pipelines are a significant problem. How much time have you wasted with “Re-run it till it works”?
Another important point to make, Application deployments for each environment should be treated the same way as deploying an application to Production. Your application should never be unavailable due to deployment. Design your deployment solution to not break what is currently available. In our example we utilize Blue-Green application deployment and Constructive-Destructive SQL deployment.

Constructive-Destructive SQL deployment is a 2 phase deployment of any changes to your database. What we want to avoid is making a change to the Database which will make the currently running applications break as well as making a change to the Database so your new version will work. Examples of changes in the Constructive phase are; adding new Table, adding new Column, adding new Index. Destructive changes, as you might have guessed; deleting a Table, deleting a Column, adding a constraint.
You may be wondering why I placed the Publish stage so late on. If publishing a larger artefact takes a few minutes, it would probably be worth publishing only when you know it is a suitable release candidate (i.e. when all your automated tests are happy!). On that note, be aware of the disk spaced used in your repository. You may wish to have a purging strategy for old versions of your application.

My final parting words, this is simply an example. This solution will not fit all applications, but the principles will! Remember reliable, simple, repeatable, and fast!

Comments

  1. Unknown16 August 2014 at 17:42

    I have also found that making each step even more granular helps with developer understanding of the CI, and makes the scope of "where" the problems are smaller.

    ReplyDelete
  2. Rick Campbell25 November 2016 at 08:48

    Another technology may also be interesting for you or your followers - data room software . Any type of info can be exchanged via this tool from any part of the world.

    ReplyDelete

Post a Comment

Popular posts from this blog

Review of High-Performance Java Persistence by Vlad Mihalcea

Summary A structured and clear read into Hibernate, JDBC and JOOQ. Plenty of useful tips and common pitfalls that will help you improve the performance of your data access. Heavily focused on relational models, so don't expect much on other types of data stores. Lots of examples to clarify points of discussion. Why should you read it If you are a java developer and have to deal with persistence, you probably have come across Hibernate and JPA. Unfortunately, writing a highly performant data access layer is difficult to get right. Similarly, bad data access patterns are difficult to tease out from an application once they have been implemented. This book describes some intricate details of JDBC, Hibernate and lightly covers JOOQ with the aim of high performance. Useful topics What actually gets executed when you use @OneToMany, @OneToOne, @ManyToOne, @ManyToMany Which Identifier strategy should you use for your primary keys How you should fetch entities from the Databa
Read more

Tackling my fear of heights

I am afraid of heights. I am 6 foot 7 inches tall, and afraid of heights. I recently realized the extent of this fear, when I gave myself the task of cleaning the rain gutters. My girlfriend and I moved into our first home together, nearly 2 years ago, and it is the first time we have ever had to clean the gutters. Up until then, we lived separately, in middling floor apartments. Anyway, here I am, with my ladder propped against the side of the house, determined to clear the debris. I made it to 4 feet from the ground before fear took hold. It felt like an eternity, frozen part-way up a ladder, nowhere near where I need to be. I descend, shaking all the way down. It took me three attempts to climb up the ladder, to then give up. I was defeated. A few months have gone by, avoiding any thoughts of trying the ladder again. That was until I came across a podcast episode, Bravery by The Habit Coach. This inspired me to try again, and the perfect job was lined up, to wash the windows.
Read more

Forming a study habit - Pomodoro technique

Hello! I thought I would brain dump some of the things I have tried while self-studying. I have a (somewhat) ambitious goal to self-study mathematics to a BSc level in Mathematics, before the year 2025 (plucking that objective and timeframe out of thin air).  I am attempting this while working a full-time job as a leader of a software development team. Today, I won't be talking about goal formation, as clearly, I need to do better at it. However, I will be covering more day to day problems I have found when forming a study habit. Pomodoro technique For those who don't know, the Pomodoro technique is a focusing hack, whereby you give yourself 25 minutes of uninterrupted, 100% concentration-time, then a 5-minute break. The intuition of this is that Humans have a hard time concentrating after 20 minutes, their attention span breaks down. The importance of this technique lies in these 2 parts, You apply all your concentration on the task, during the 25 minutes. That mea
Read more