CI/CD automation for Agile projects - AWS CloudFormation (IaC), Jenkins, Git, Jira and Selenium

Introduction

In this article I plan to describe how to go about automating the entire software development process using Continuous Integration (CI)/Continuous Deployment (CD) tools sets. Following CI/CD toolset will be used by me to explain complete end to end automation

 

Architectural Overview

CI/CD Automation Flow

In the above diagram I am using numbering to explain a typical automation flow in a CI/CD type of application development.

Point 1: Source Code Management (SCM)
Developers will use some sort of source code management software (like Git) to check in code. Any source code repository allows multiple release branches to be maintained for parallel releases. A code “check-in” event can trigger an automatic built process and you can control when to build based on some predefined rules, for example:

• For git you can trigger automatic builds based on release “tags” – like rel1.0, rel2.0 etc.
• For GitHub you can trigger automatic builds based on Web hooks calling your endpoint URL to trigger a build. And within your endpoint URL code you can check the GitHub event to determine if a built is needed.

Point is, SCM allows you to associate “check in” events to trigger automatic builds based on pre-defined rules. This gives you control over which code “check in” event causes automatic builds.

Point 2: Build Server

Build servers are the main components that connect all the CI/CD components together by allowing orchestration of built process through numerous plugins. One such software is Jenkins. Jenkins is an open source software that provides plugins to integrate with

• SCM software’s like git, GitHub, CodeCommit etc.
• Cloud providers like AWS, GCP, Azure
• Java applications through maven plugin
• .NET applications through msbuild plugin
• ALM software’s like Jira

AWS also provides an end to end build software called – AWS CodePipeline. AWS CodePipeline is something you can consider if all the components you are using are from AWS cloud (for example: CodeCommit instead of git).

Point 3: Application Lifecycle Management (ALM) tools like Jira

With Jenkins you can use the Jira plugin to update any issues that are opened in Jira so Business Owners can look at Jira’s comments section to know what issues are fixed by the build. I have spoken about this type of integration in my blog: http://pmtechfusion.blogspot.com/2017/03/application-lifecycle-management-alm.html. I encourage you to read that blog in case if you are interested in knowing the details.

Point 4: Shared Infrastructure Stack

With Infrastructure as Code (IaC), it’s possible to define infrastructure in terms of code scripts that can be checked into SCM along with the application code thereby allowing versioning of infrastructure templates just like application code in SCM. AWS provides CloudFormation to define infrastructure templates in JSON or YAML format (Azure and Google - GCP also provide similar JSON style scripting capabilities). In a CI/CD type of development automation, you should define at least two infrastructure templates one for infrastructure that is shared across applications and another for application specific infrastructure and technology stack. The former is usually not torn down with each automatic deployment.

Shared infrastructure stack – Architectural Diagram

NOTE: The above shared infrastructure is defined with High Availability in mind.

A sample infrastructure layout is shown above. Some of the AWS components that you define in shared infrastructure CloudFormation stack are mentioned below

• VPC – Virtual Private Cloud (the virtual network)
• Internet Gateway (IGW) – this allows the VPC access to internet
• NAT Gateway – this is defined in the shared infrastructure so that private subnets in the application specific stacks can use the NAT gateway to access the internet.
• Elastic IP – NAT gateways need to be given a public IP (Elastic IP allows that to happen in AWS)
• Route Tables – typically you define three route tables (at a minimum)

1. One for public routes
2. Two for private routes (Since I have two subnets I will have two route tables). In each private route tables you define an outbound rule that sends all internet traffic to respective NAT Gateways. This is how private subnet EC2/RDS instances can access the internet for package/software updates.

• S3 – is the AWS storage used to store CloudFormation templates and Application specific Code.
• Two public subnets (A and B) – you need to define one each for the two NAT gateways for high availability.

Point 4: Application Infrastructure Stack

This is the CloudFormation stack that will be continuously build, destroyed and rebuilt as a part of the CI/CD process. By having a separate stack for application specific builds you isolate different applications from overlapping with each other’s build.

Application infrastructure stack – Architectural Diagram

The green shaded box shows the shared infrastructure stack defined earlier. The Blue shaded box shows the Application Specific CloudFormation stack.

Some of the AWS components that you define in Application specific infrastructure CloudFormation stack are mentioned below

• You typically will have 2 public subnets (C and D) – these subnets will be associated with the public “RouteTable” defined in shared infrastructure for routing packets.
• For the application layer you will define 2 private subnets (A and B) and for database layer (specifically RDS) you are required to have 2 subnets (C and D - defined private here for security) for high availability. The two private “RouteTable” defined in shared infrastructure are used by these subnets for packet routing. And since each “RouteTable” has a NAT Gateway defined for internet outbound traffic. Any EC2 instances and RDS instances can communicate with the outside world via these “RouteTable” only.
• You define 2 Application Load Balancers (ALB) in public subnets (C and D)
• You define at a minimum following security groups

1. Security Groups (SG) that allow Application layer instances access to database layer instances.
2. SG for ALB such that they can access the Application Layer.
3. SG to allow public traffic to flow through port 80/443 of the ALB (you typically terminate SSL at the ALB)
4. SG for a Bastion EC2 instance (not shown in the diagram) running in either public subnet A or B above that can be accessed via port 22 (ssh) from your corporate public IP’s. Bastion EC2 instances is how you get into the infrastructure to administer your VPC network and EC2 instances

• Auto scaling group and Launch Configuration for your servers
• RDS for your database layer
• CodeDeploy components in the Application Stack that use the S3 bucket for updating the application code using Deployment Group to deploy different built versions of application code. The newer versions of the application code will be built by Jenkins whenever automatic build triggers (say by a release tag commit in git). Jenkins will then put the application build code in AWS S3. Jenkins will then use AWS API to run the application specific stack with an “Update Stack” command thereby resulting in automatic deployments.

In the Application Layer you can have an application server running JEE/Spring/Hibernate technology stack or ASP.NET/WCF/ER Framework .NET stack or Python/Django framework or Node.js or PHP stacks.

In the Database layer the RDS can be – MySQL, SQL Server, Aurora etc. or you can even have a NoSQL like – MongoDB, DynamoDB or DocumentDB data layer.

CI/CD Automatic Flow:

The diagram below shows the complete logic that triggers the CI/CD process.

NOTE: It is technically possible and most certainly desirable to automate the build process based on testing results such that the success and failure of the automatic testing done via Selenium will determine if Production environment code deployment should occur. This flow logic is not shown above but is the next logical step to take to even automate production deployments
 

Conclusion
You can use the above toolsets mentioned in this article to do Continuous Integration and Continuous Deployment in your project. Infrastructure as Code (IaC) is changing the way software development is done these days and CI/CD automation is allowing application teams to be agile, productive and deliver bug free software products. The term DevOps is truly bridging the barrier between developers and operations and is merging the two roles into one cohesive role.