Application Lifecycle Management (ALM)/ Continuous Integration (CI)

Introduction

In this article I plan to describe how to go about doing Application Lifecycle Management (ALM) using open source or economically feasible software that you can afford to justify to your management. I will start with a 10,000 feet high level overview and then deep dive into installation and execution of the various elements that are used to make ALM and Continuous Integration a reality with minimal project cost.

Architectural Overview

In the following section I plan to show how to configure the various pieces of the software mentioned in the above Architectural Diagram to provide an ALM/CI experience. In a nutshell, I am using Jenkins to be the middle-man to connect all the dots here. Jenkins has plugins to download my “Sample Web App” (written in Java) from the Source Code Management repository – Git. Jenkins has plugins to integrate with Maven the source code built software, it has plugins to integrate with Jira the ALM software that is using Scrum methodology for project management. So basically the end-to-end user experience starting from Scrum User Stories to Scrum Releases to defects to code fixes to code builds is all controlled through Jenkins automatically and the various plugins within Jenkins automatically update the respective software’s proving the ALM/CI experience and most importantly the end-to-end visibility to all stakeholders.

Development Environment

I plan to use the following technology stack for ALM/CI

• Java/Spring Framework/Hibernate/MySQL
• IDE: Eclipse
• Web/Application Server: Tomcat
• Code built software: maven (you can also use Gradle)
• Operating System (OS): Ubuntu
• Virtualization software: VMWare workstation for Virtual Machines (VM)
• Built Software: Jenkins
• Source Control Management (SCM) software: Git/Apache server
• ALM software: Jira (I am using Jira but if you are looking for an open source ALM software that’s free try  - Tuleap)

Order of installation

1. We will start with “General Software Installation” section first

• Create a Ubuntu VM
• Install Java JDK
• Install eclipse
• Install Maven

2. Git/Apache
3. Jenkins
4. Sample Web App

• Add the sample web app to Git repository
• Install tomcat
• Import the sample web app into eclipse by pointing to local Git repository

5. MySQL
6. Jira

General Software Installation

Step1: Create a VM in VMWare workstation that uses Ubuntu OS

Download the latest ISO image for Ubuntu by going to following link: https://www.ubuntu.com/download

After downloading the same, use VMWare workstation to install Ubuntu as a VM. For the purpose of this blog, I am going to install the Ubuntu Desktop edition but will use terminal console commands most of the times to show how the server components can be installed in Ubuntu server OS where you do not have a graphical interface.

Step2: Install Java JDK

In console use the following command to install Java JDK

apt-get install default-jdk

This will install the latest Java JDK from the Ubuntu managed repository. If you want to use Oracle JDK then you will either have to install it manually or go for some PPA (Personal Package Archives). I suggest go with what Ubuntu gives you unless you want the latest greatest and if you do then I suggest going with manual installation versus relying on some third-party PPA. However remember manual installation means you need to ask the Unix/Linux administrator to document every step before he/she does it so that the documentation lives beyond the project dependency on that single person.

Step3: Install eclipse

I am going to do a manual installation for eclipse as the one that comes with apt-get repository is a very old version of eclipse IDE.

NOTE: You need at least 2GB RAM for running eclipse on an Ubuntu VM

Step3a

Go to - http://www.eclipse.org/ and download the tar for “Eclipse IDE for Java EE Developers”

Step3b

Untar the downloaded file using the following command in your temporary folder (my temporary folder is /mysoftwares)

tar -xvf eclipse-jee-neon-2-linux-gtk-x86_64.tar.gz

Step3c

Move the untar folder to /opt folder using the following command

mv /mysoftwares/eclipse /opt/

Step3d

That’s pretty much the installation, what I am doing next is to create shortcuts in desktop for launching eclipse as shown below

• First create a /mysoftwares/eclipse.desktop file with contents as shown below

[Desktop Entry]

Name=Eclipse

Type=Application

Exec=eclipse

Terminal=false

Icon=eclipse

Comment=Integrated Development Environment

NoDisplay=false

Categories=Development;IDE;

Name[en]=Eclipse

• Next use the following command to install the desktop shortcut for eclipse

sudo desktop-file-install /mysoftwares/eclipse.desktop

• Now copy the eclipse icon using the following command to allow the eclipse icon to be displayed from the start tray on left side within Ubuntu Desktop

sudo cp /opt/eclipse/icon.xpm /usr/share/pixmaps/eclipse.xpm   

• Now let’s create a symbolic link to “/opt/eclipse/eclipse” so the desktop shortcut can launch the eclipse IDE

sudo ln -s /opt/eclipse/eclipse /usr/local/bin/eclipse

The /usr/local/bin is in PATH of user account hence the symbolic link works.

Step4: Install maven

I am using maven to do source code built. If you want to use Gradle you can. Both have their pros and cons. I always say the same thing when it comes to technology shifts, “Flavor of the day” is different. I tend to focus on the ice cream.

Use the following command to install maven

sudo apt-get  install maven

Git/Apache

Step1 – Install Apache

I plan to use Apache web server to serve my private Git repository over http. I want to do this versus using github so I can control the code repository for our projects.

To install Apache just use the following command

sudo apt install apache2

I plan to use the default DocumentRoot folder (/var/www/html) of the default website to create the raw Git repository base

Step2 – Install Git

Step2a

Let’s install Git using the following command

sudo apt install git

Step2b

Let’s create a folder under DocumentRoot folder (/var/www/html)

mkdir myprojects.git

Step2c

Now let’s change the group to be www-data (the apache group) so that apache can read and write to folders and files inside this folder. The “myprojects.git” is the root folder of your local Git repository on the server-side

chgrp -R  www-data /var/www/html/myprojects.git

Step2d

Now also set the gid bit using the following command

chmod g+s /var/www/html/myprojects.git

This command will set the gid bit – special bit so any folders and files created in /var/www/html/myprojects.Git folder will have group “www-data” as the group owner

Step2e

Let’s create bare repository using the following command under the folder (/var/www/html/myprojects.git)

Git init --bare –shared

This command sets the folder (/var/www/html/myprojects.git) to be the root folder for Git on the server-side.

Step2f

Now we need to allow apache to render files and folders under folder (/var/www/html/myprojects.git) to be served to the client-side software of the git. Let’s start with enabling some modules that we will need for the same using the following command

a2enmod cgi alias env rewrite

Step2g

Now we need to enable push from Git server-side using the following command

Git config http.receivepack true

The above is needed to allow push; clone is enabled by Git by default but not push. Run this command from the Git root folder which is the folder (/var/www/html/myprojects.git)

Step2h

Now update the default site configuration file (/etc/apache2/sites-enabled/gedit 000-default.conf) to allow CGI script running from the DocumentRoot folder (“/var/www/html/”) as shown below

SetEnv GIT_PROJECT_ROOT /var/www/html/

SetEnv GIT_HTTP_EXPORT_ALL

ScriptAlias /myprojects.git/git/ /usr/lib/git-core/git-http-backend/

           Options ExecCGI MultiViews SymLinksIfOwnerMatch

           AllowOverride None

          Require all granted

        Options -Indexes

Step2i

That’s pretty much it now you can access the private local Git repository on the client side using the following commands on the developer’s workstation

Git config --global user.name "testuser1"

Git config --global user.email "testuser1@local.com"

Git clone http://localhost/myprojects.git/git/myprojects.git

cd myprojects

echo "sample file to create master branch" > mysamplefile.txt

Git add mysamplefile.txt

Git commit -m "Adding start text file - to create master branch"

Git push origin master

Jenkins

Installing Jenkins is pretty straight forward, go to following link to get the installation steps - https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu

Basically what we are doing above is adding Jenkins package repository to the Ubuntu apt “sources.list” file so we can then use apt-get to install and update Jenkins using Jenkins package repository (this is basically the PPA maintained by Jenkins)

If you want to change the port on which Jenkins web server listens on, you can edit the “/etc/default/ Jenkins” file to change the same.

I have installed Jenkins on a different Ubuntu VM that will serve as a built server for multiple projects. Following things need to be installed on that machine

Step1:

Java JDK/Maven

Refer to “General Software Installation” section for installation steps

Step2:

Git application (you need this to get the source code from Git repository on the remote “git” server

Refer to “General Software Installation” section for installation steps

Step3:

Git plugins needed for Jenkins

When you access Jenkins for the first time you can pick select recommended plugins option, that option will automatically install the Git plugin.

Step4:

Maven plugin

You can go to “Manage Jenkins” – “Manage plugins” to install the “Maven integration plugin” as shown below

Step5:

Now let’s create a Maven built job as shown below

Step6:

Let’s point to the Git repository that I have on the other Ubuntu VM so we can download the source code and built the war file using maven. The details are shown below

The URL: http://192.168.164.180/myprojects.git/git/myprojects.git is pointing to the other VM where I have installed git/apache as well as tomcat application server

Step7:

You also need to provide relative path to pom.xml if that file is not in the root which is the case for me as I have multiple projects in central Git repository with each folder representing separate projects. So provide a relative path to pom.xml as shown below

Step8:

Now let’s configure the path to maven on the built server by doing the global configuration in Jenkins as shown below (use mvn –v to know what the maven home folder is on your machine)

Step9:

Now all you need to do is click the “Built Now” link to build the sample Maven Built Job “My Sample Web App” as shown below

Step10:

The screen print below shows the console log of a successful built. By default the workspace for Jenkins in Ubuntu is in (“/var/lib/jenkins/workspace/”) folder and then there are sub folder (“/var/lib/jenkins/workspace/My Sample Web App”) for each job in it where the built occurs

Step11:

Now we need to upload the war file generated for the “My Sample Web App” Jenkins built job as a post step. There are three ways to do that

Option1a:

Use the Maven plug-ins provided by apache tomcat for Jenkins: http://tomcat.apache.org/maven-plugin-2.2/

Option1a-step1:

Let’s first start by changing the “pom.xml” file in your “Sample Web App” to use the Apache tomcat plugin as shown below

Option1a-step2:

We are referencing a lookup value “TomcatServer” in the above plugin section “Option1a-step1”, the actual details are stored in maven’s “/usr/share/maven/conf/settings.xml” file so we need to include the tomcat authentication details in that file to abstract the username and password from the project specific file (“pom.xml”).

For eclipse IDE, as I am using an embedded maven plugin provided by eclipse instead of the installed software; the settings.xml file is typically copied from “/usr/share/maven/conf/settings.xml” to the “~/.m2” folder and eclipse maven plugin looks at that location. So to compile my code I need to make changes to “~/.m2/settings.xml” file and then on the built Jenkins server where I have only installed maven software (no eclipse IDE) I need to do it at “/usr/share/maven/conf/settings.xml”.

NOTE: Instead of using the maven plugin in eclipse IDE, it’s possible to configure eclipse to use the installed maven software on the machine thereby making eclipse reference the same “settings.xml” file used by the maven software installed. Since you are copying the file from “/usr/share/maven/conf/settings.xml” to the “~/.m2” folder of the developers login profile make sure that the “~/.m2/settings.xml” file owner and group is that of the developers login ID (“testuser1” in my case), otherwise the eclipse IDE which runs under that users security credentials will not be able to read the “~/.m2/settings.xml” file

The changes to “settings.xml” file are the same and are shown below (password is masked by me for obvious reasons)

Option1a-step3:

Now we need to configure the “tomcat” user in “/opt/tomcat/conf/tomcat-users.xml” file to allow Jenkins to use that account to upload the generated war file to tomcat using the tomcat deploy URL: http://192.168.164.180:8080/manager/text (defined in pom.xml file in “Option1a-step1” section)

Option1a-step4:

Since my built server is going to be on a different Ubuntu VM than the tomcat server I will have to allow remote deployment to tomcat through the URL: http://192.168.164.180:8080/manager/text by editing the “manager” web applications “/opt/tomcat/webapps/manager/META-INF/context.xml” file as shown below

Option1a-step5:

Now all you need to do is change the maven goals within Jenkins to allow war files to be undeployed - tomcat7:undeploy  and deployed - tomcat7:deploy (you start with undeploy and then deploy so that you can repeat the builds multiple times, the first time the undeploy - tomcat7:undeploy goal will fail). The screen print below shows that change done to “My Sample Web App” Jenkins job.

NOTE: Even if the maven goals say “tomcat7:deploy” and “tomcat7:undeploy”, the same goals work for tomcat9 as well.

Option1b:

Write a post built script (shell script) that uses wget/curl command to copy the generated file from the built server workspace location “/var/lib/jenkins/workspace/My Sample Web App/AWSSampleWebApp/aws.webapp/target/aws.webapp.war” to tomcats hot deploy folder “/opt/tomcat/webapps”. This is pretty straight forward to configure and I do not plan to show the exact steps for this blog. The screen print below shows where within Jenkins you need to do the configuration and invocation of the bash script. This configuration is done at the Jenkins job configuration level “My Sample Web App” job in this case

Option1c:

Use third-party provided plugins – Example: Capitomcat: https://wiki.jenkins-ci.org/display/JENKINS/Capitomcat+Plugin

Using this plug-in is also pretty straight forward just install it in Jenkins and then follow the release notes on the above link to configure the plugin parameters and you are all set. It’s possible that other similar plug-ins are already available in the marketplace, that’s the very reason I am not elaborating these steps in this blog for ALM/CI.

Step12:

We also need to install the “Jira plugin” to allow Jenkins to update Jira comments when builds that fix defects are executed and the application with new code is deployed. This “Jira plugin” is what is going to give the end-to-end ALM experience. The installation of the plugin is pretty straight forward and details of the plugin are available on the following site link: https://wiki.jenkins-ci.org/display/JENKINS/JIRA+Plugin

Once the plugin is installed on Jenkins via the Manage Plugins screen as shown below

Note: the above screen print shows two Jira plugins, I am using “Jira plugin” in this blog. But you can also use the second plugin called “Jira Issue Updater”, the following site: https://wiki.jenkins-ci.org/display/JENKINS/Jira+Issue+Updater+Plugin has details for that plugin

The one thing you need to remember for the “Jira Issue Updater” plugin is the REST API URL for Jira is: http://localhost:8080/rest/api/2 (I am using localhost as the Jira and Jenkins software are installed on the same Ubuntu VM)

Back to the “Jira plugin” configuration, you need to perform two configuration steps as shown below for the same

Step12a:

The first configuration step is at the Global setting level within Jenkins as shown below

The above configuration needs to be done once for all Jenkins jobs. One thing to note here is the field “Issue Pattern” is empty so the default pattern will be used to match the Issue ID within Git commit messages which is -. This default “Issue Pattern” matches our two bug’s format “MYWEBAPP-7” and “MYWEBAPP-8” respectively. However if you have a different pattern than the default then this is where you configure the same at a global level.

Step12b:

Now at the individual Jenkins built level you do the following configuration

Basically I have a post built action for “My Sample Web App” Jenkins built that uses the globally configured “Issue Pattern” in “Step 12a” above to update the Comment section for the two Jira defects (“MYWEBAPP-7” and “MYWEBAPP-8”) thereby providing end-to-end visibility between code builds and defects that are being fixed by the code builds

Sample Web App

For the purpose of showing ALM, I have created a simple Java web stub. The stub is using Spring Framework to provide a MVC (Model View Controller) pattern. It’s using Hibernate as an ORM (Object Relational Mapping) to abstract the database SQL from application code. It’s using MySQL as the database server. Another reason for me to use this application architecture is so I can use the same stub to deploy code in AWS using features like AWS Beanstack, AWS Auto Scaling, AWS VPC.

Step1:

Step1a:

After creating the local developer Git repository by following the steps outlined in “Step2 – Install Git - Step 2i” section, let’s copy the sample Java Web app “AWSSampleWebApp” to the Git repository folder (/mysoftwares/dev_locrep_clientside/myprojects). After the transfer, the directory path for the “.git” folder should be: (/mysoftwares/dev_locrep_clientside/myprojects) and the project folder should be under that (/mysoftwares/dev_locrep_clientside/myprojects/AWSSampleWebApp)

Step1b:

Execute the following command to add the copied project under folder (/mysoftwares/dev_locrep_clientside/myprojects) to the staging area of your local Git repository

Git add .

Execute the above command from the (/mysoftwares/dev_locrep_clientside/myprojects) folder.

Step1c:

Next perform the first project commit to include the same to your staging local repository

Git commit -m "My project baseline commit"

Execute the above command from the (/mysoftwares/dev_locrep_clientside/myprojects) folder.

Step1d:

Now execute the following command to push and commit the local changes from staging to the central server

Git push

NOTE: If other developers have committed their code and you want to pull that code to your local repository then use the “Git pull” command for the same.

Now you have the base code in central Git repository, from this point forward you will use eclipse to create a Maven project in Java by pointing to a local Git repository and then any changes you do to the source code you will be able to use the “Git Staging” tab within eclipse to commit and push the code changes to local and central repository at the same time from eclipse IDE directly

Step2:

Let’s install tomcat to be used as an application server to deploy our sample Web application. Since we want to use the latest tomcat (version 9) with latest version of eclipse (neon), let’s manually download the file from: http://tomcat.apache.org/index.html

Step2a

Untar the file using the following command

tar -xvf apache-tomcat-9.0.0.M17.tar.gz

We need to now move all the untar files/folders under the folder (/opt/tomcat).

The folder (/opt/tomcat) has the following permissions

drwxrwxr-x  9 testuser1 testuser1 4096 Feb 21 16:21 tomcat

Step2b

Let’s create a tomcat user for the tomcat service using the following commands

sudo groupadd tomcat

sudo useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat

(a shell of /bin/false means nobody can log into the account, we do that for security reasons as user “tomcat” is running as a service on the Ubuntu box)

Step2c

Now let’s give permission to the tomcat user as shown below

Give the tomcat user write access to the conf directory, and read access to the files in that directory:

sudo chgrp -R tomcat conf

sudo chmod g+rwx conf

sudo chmod g+r conf/*

sudo chown -R tomcat bin

sudo chgrp -R tomcat lib

Next, make the tomcat user the owner of the webapps, work, temp, and logs directories:

sudo chown -R tomcat webapps/ work/ temp/ logs/

Step2d

Next let’s create the tomcat.service file for the “systemd” daemon to control start/stop of tomcat service. The contents of the tomcat.service file are shown below. Make changes to the environment variable as appropriate for your environment.

Next copy the file to tomcat.service location /etc/systemd/system (you need sudo privileges to do the same).

Next, reload the systemd daemon so that it knows about our new service file:

sudo systemctl daemon-reload

Start the Tomcat service by typing:

sudo systemctl start tomcat

Step3:

Now lets try to compile the file with eclipse by pointing the eclipse to the developers local repository folder (/mysoftwares/dev_locrep_clientside) and then writing the “.gitignore” file to exclude the “target” folder from being put into repository, we do not want to version control the “target” folder as the contents of that folder can be recreated with maven build’s and there is no point in versioning the compiled and packaged code.

Let’s start by importing the project from the local Git repository. Navigate to the folder (/mysoftwares/dev_locrep_clientside), select the local Git repository and hit Finish to import the sample web project into eclipse   

The rest of the steps in eclipse code compilation will change depending on what version of eclipse you use and what libraries you use within maven. I will not explain those details in this blog as this is not a Java/Spring/Hibernate/Maven/Git/MySQL blog.

MySQL

To install MySQL use the following command

sudo apt-get install MySQL-server

I am going to create a simple database and table within MySQL which will be used by the hibernate ORM as a persistent database. I am not going to spend much time explaining the database/table part of MySQL in this blog as I want this blog to focus more on ALM aspect and continuous application development/integration process.

Jira

I am using Jira here as an economically feasible ALM software. But if you want you can also use Tuleap (which is open source) and free.

Both have plug-ins to integrate with Jenkins. These plug-ins allow us to associate the artifacts within Jira (Jira calls it issues) with builds done within Jenkins. There by providing end-to-end visibility starting from Jira all the way to code built.

I am going to use the Scrum board within Jira for Agile development (Jira out of the box provides Scrum and Kanban as two Agile templates).

Step1:

In order to install Jira you can either use the installation binary provided for supported Windows/Linux versions or download the tar file and do the installation manually.

I plan to use the binary installer provided for Linux 64bit OS (Ubuntu in my case). The steps to do the installation are pretty straight forward and are mentioned in the following site: https://confluence.atlassian.com/adminjiraserver073/installing-jira-applications-on-linux-861253030.html#InstallingJIRAapplicationsonLinux-2.Runtheinstaller

If you want to do manual installation with the tar file, use the following site link that provides installation details:

https://confluence.atlassian.com/adminjiraserver073/installing-jira-applications-on-linux-from-archive-file-861253033.html

Step2:

Jira comes with a Jenkins plugin to integrate with Jenkins and Jenkins also has a Jira-plugin to do the same. I am using the later. The installation steps to do that are pretty straight forward and are mentioned in “Jenkins – Step12” section

Step3:

I used a sample Scrum template available within Jira to manage the Sample Web Application. Setting up of Scrum project is pretty straight forward. The screen print below shows the Scrum Dashboard for the “MYWEBAPP” project.  Within Jira everything is considered an “Issue Type”, to some extend that is due to the fact that Jira is more widely used as an Issue/Defect tracking software and has grown into an ALM software by providing Agile templates for Scrum/Kanban. The Scrum board shows User Stories that need to be done, Tasks associated with those User stories for the “Sample Web App” project. My current Scrum dashboard is displaying only User Stories and Task based on the filter, but it’s completely possible to customize this out of the box Scrum Dashboard to suit the needs of individual projects.

Step4:

For showing how end-to-end visibility is achieved I created two sample “bugs” in Jira for the “Sample Web App“project as shown below (MYWEBAPP-7, MYWEBAPP-8). Each “Issue Type” within Jira has a unique ID that uses the following format: -

Step5:

Let’s say the developers in the team get to work on the two defects opened in “Step4” above and fix the same and check in the code with comments like – “Developer 1: Fixed defects MYWEBAPP-7, MYWEBAPP-8” in Git. The screen print below is from eclipse IDE where the developers are entering their comments in the “Git Staging” tab within eclipse IDE

The Jira plugin within Jenkins out of the box parses the text used in Git commit and is smart enough to associate the Jenkins built with the two bugs by updating the Comments section accordingly as shown below. Both bugs comment section are automatically updated

Bug MYWEBAPP-7 comments section

Bug MYWEBAPP-8 comments section

The screen print below shows the Jenkins built log showing the two Jira defects being updated.

Conclusion

Most of the software I have explained at 10,000 feet level here is open source and free but free does not mean cheap, sometimes investing in commercially expensive alternatives like Visual Team Suites, IBM Rational products (DOORS Next Generation) is a better approach. In the end, the saying one shoe fit all strategy does not work. With that said, in my experience these products do get the job done especially for a team that is comprising of 10-15 members; the larger the team size gets its harder to use these products as one cohesive unit. For large projects if your management has the budget using commercially expensive products like Visual Team Suites, IBM Rational products (DOORS Next Generation) is a better choice as the software cost pays out in the end for time wasted in using bandages with cheaper alternatives. Hopefully I will find time to write article on IBM DOORS next generation and Visual Team Suites in the coming months. If you are in the Java band wagon using IBM DOORS next generation will be a natural platform to use than Visual Team Suites which is very much meant for Microsoft enthusiasts. For all others who have a small budget use the software’s mentioned in my blog and you should be equally as productive with your ALM/CI experience.