Kubernetes (K8s), ELK, Kafka, Fluent and logging data analytics

Introduction

In this article I will provide an overview of how you can send logging information from Kubernetes (K8s) cluster to a central place for data analysis. Even though I am providing explanation for a K8s cluster the same approach applies for non-containerized applications.

Architectural Overview

The above logical diagram represents a high level overview of the technology stack involved.
A typical log data generation flow is described below

• User will access different applications that are hosted on K8s cluster. In the above example I am showing two K8s services each with three replica sets. The K8s cluster provides load-balancing and all the capabilities needed for High Availability (HA) and auto-scaling. These user interactions will create application specific logs, for example: an invalid login attempt, runtime application exception etc.
• The K8s Cluster can be a 3(or N) Node cluster for small scale applications. On each of the nodes you can run K8s DaemonSet that will stream log data from Docker containers hosted on that Node to Kafka cluster or ElasticSearch (ELK – ElasticSearch, Logstash and Kibana) Cluster.
• The DaemonSet will run light weight version of Fluent (Fluent Bit), it’s possible to use Filebeat instead of Fluent Bit if you want to stay within the ELK ecosystem.
• I am using Kafka cluster to provide application specific filtering and routing.
• The ELK cluster is used to store all logs and acts as a central repository for log storage.

Component description:

Kubernetes (K8s) Cluster

This is the typical Kubernetes cluster that will host containerized applications.

Web: service – 3 replicas/App: service – 3 replicas

This represents a typical distributed application where your web application is hosted as a Service within K8s cluster and your application layer is also hosted in the same cluster.

DB

I have shown the DB component outside the K8s cluster as that helps having specialized teams optimize that layer outside of K8s. Also when I say DB, it’s not necessarily relational DB, it can be a NoSQL database like MongoDB or a combination of both. Also it’s not a hard and fast rule to have data layer outside the K8s cluster, I am just providing what is a typically preferred approach.

DaemonSet (Fluent Bit or Filebeat)

Either a Fluent Bit container or a Filebeat container can be deployed as a DaemonSet on each K8s cluster node, the container can be configured to stream log information from Docker location (“/var/lib/docker/containers”).

NOTE: You can also use the same container to stream logs from other locations like “syslog”.Assuming you are using Fluent Bit, within the K8s cluster you will have to define ConfigMap that will be referenced by the DaemonSet Fluent Bit container that will then use the mounted configuration files and stream log contents to either ELK or Kafka depending on how the Fluent Bit configuration file is defined. Below is a high level overview of how the Fluent Bit configuration setting is defined

1. You start by defining a starting configuration file which will load different sections of Fluent Bit configuration by referencing other files (via includes)
2. You will have one file defining the different inputs – which in this case will be the location of Docker file container logs on each node, you will use the standard “tail” plugin that comes with Fluent Bit for streaming log contents.
3. You will have one file defining the different filters – in this case you will use the standard “kubernetes” filter that comes with Fluent Bit to annotate each record with details that include information like – pods namespaces, node name, container name etc. This additional metadata can be used later on to do data analytics via Kibana in the ELK cluster.
4. You will have one file for different outputs – in this case you will define one output stream for Kafka using the standard “kafka” output plugin and another for ElasticSearch using the standard “es” plugin

The diagram below gives an overview of different configuration elements involved in the Fluent Bit

Node 1 ………..Node N

This component represents the actual nodes that are hosting the K8s cluster. For smaller applications start with a 3-node cluster and then scale from there.

Kafka Cluster

The Kafka cluster is used to filter log records that are getting streamed to it in order for you to create application specific Tasks (through the Kafka Consumers) or send alert notifications, SMS etc.

ElasticSearch Cluster

The ElasticSearch cluster is used to store all log records so you can use Kibana to create visual dashboards and use the ELK console client to perform data analysis.

A few implementation details to consider

• Even though I have mentioned Fluent Bit as the log forwarder in this article, you can certainly use Filebeat as a viable alternative if you want to stay within the ELK ecosystem.
• You should consider defining buffer limits on your Fluent Bit input plugin to avoid back pressure
• By default Fluent Bit does an in memory buffering of records while data is routed to different outputs, consider using file system buffering option provided by Fluent Bit to avoid data loss due to system failure.
• I have used “Kafka” as a central filtering option for log records. It’s possible to use filters at the Fluent Bit level to remove records before they make their way to the Kafka topic. I prefer the former as that way I have all the log records in the Kafka Topic, in case if I need to do detailed analysis later. It also eliminates the need to utilize CPU/RAM processing power at the container level if the filtering of log records is not done there. The benefit of doing filtering at the Fluent Bit level is that less data transmits over the network(it’s a trade-off)

Conclusion

I hope you liked this article.

Microservice and Distributed System infrastructure with Spring Cloud

Introduction

In this article I will provide an overview of the building blocks for creating a Microservice infrastructure and in general a distributed system infrastructure. I will be using the Spring Cloud project an umbrella project under which various sub-projects are available to provide the building blocks for creating a distributed system (Microservice is one such example). If you are looking for a technology that provides all the components out of the box then I suggest you go with Kubernetes (K8s) the most widely used open source containerized platform for Microservices. This article is for folks who want to build these components from ground up.

Building Blocks

The above diagram provides some of the build blocks that go into providing components that are required for a Microservice or distributed system implementation. The individual block description follows

External Configuration Server

In a distributed system there needs to be a way for configuration related details to be centrally available and managed for all the services so that changes can be done from one place. External Configuration server component serves that purpose. Within the Spring Cloud ecosystem, Spring Cloud Config server provides that capability. Consul and etcd are the other two widely used alternatives.
NOTE: If you create a multi-node Kubernetes cluster it uses “etcd” as its central configuration server (runs three pods typically for high availability)

API Gateway

When you are exposing the Services via API (typically REST API) so that they can be consumed by either other services or external systems you need a gatekeeper to control the access of these services. API Gateway is the component that provides you with that capability. With the Spring Cloud ecosystem, Spring provides framework that allows you to use Netflix Zuul as one such API gateway server. Just recently Spring Cloud announced another project initiative called Spring Cloud gateway that is worth checking. There are many other alternatives listed in the above diagram and that list is not even complete. At a minimum some of the capabilities that this component provides include

• Dynamic routing
• Security around access to services
• Rate limiting
• API management
• Acts as a proxy to external clients accessing the internal Services.

Service Discovery

With services build around Domain Driven Design (DDD) or Business capabilities, the need to allow these services to register and discover each other dynamically is there. The Service Discovery component provides that capability. With the Spring Cloud ecosystem, the Netflix Eureka Server provides that capability. Apache Zookeeper and Consul are popular alternatives.

Logging & Monitor Tools

With any distributed architecture the need to have logging, monitoring and tracing capabilities is essential. There are multiple projects/libraries that are providing such functionality a few of which are listed below


Sleuth
Sleuth provides capabilities to capture identifiable logs which can assist in distributed tracing. Spring Cloud provides integration libraries for Sleuth.

ZipKin

Zipkin provides visual tracing of calls made across multiple services. It relies on Sleuth to provide the identifiable logging capabilities upon which it relies to provide a visual representation via a dashboard. Spring Cloud provides libraries that directly integrate with Zipkin.
With Sleuth/Zipkin you can identify Services in your workflow path that are taking longer time to execute and can identify the root cause visually very quickly.


Hystrix
In a distributed system where inter service communication can be complex and where one slow or non-functioning service might bring the entire service flow orchestration or choreography to a grinding halt. It’s essential to have fail-safe mechanism in place. Circuit Breakers provide that fail-safe capability. Netflix Hystrix is a library that allows a fail-safe service to be called if the primary service is not functioning based on expected response times. Spring Cloud provides libraries that integrate with Hystrix and also provides a dashboard that gives you visual insight into a Service health. Hystrix relies on Spring Actuator a capability provided by Spring to monitor and manage any Spring Java based application. Spring Actuator exposes the endpoints via JMX and/or HTTP and Hystrix relies on those exposed endpoints for its functionality.


Turbine
Hystrix provides metrics on a per application basis. We need something that aggregates metric across applications and provides a visual dashboard for the same. Turbine is the library that provides that aggregate capability across applications.

Logical Architecture:

The above diagram provides details regarding the various components that can be utilized through Spring Cloud to provide a distributed system infrastructure that can be used for Microservices architecture implementation. All of the individual components shown above are typically implemented via Spring Boot style applications with Maven controlling which component specific dependencies provided with Spring Cloud can be used

In the following sections I will elaborate on some of these components to explain how they interact with each other in the overall distributed architecture.

Zuul Proxy
A typical user flow will start with user accessing a website URL via the Eureka Zuul proxy. The Zuul proxy server registers itself as a Eureka client with the Eureka Server so that it can discover all the application specific services that it needs to interact with on behalf of the user. Zuul proxy allows you to control which request headers can be passed on to the services invoked by the client. Typically you will pass JWT tokens for single page applications via http headers to the services so they can authorize the client’s requests. The Zuul proxy runs as any other spring boot application and therefore can utilize the Spring Security framework to authenticate the calling user or application. In the diagram I am showing Redis cluster being utilized to externalize the session management capability. Spring Session another Spring sub-project allows sessions to be externalized into Redis cluster which is what you will require to do in a distributed system versus the traditional application server specific session management. If you are using Angular/React or any other Single Page Application (SPA) you can use JWT tokens for authorization. Hystrix (Circuit Break pattern) support is built into Zuul and can be configured to provide fail-safe service implementations.


Spring Cloud Config Server and Eureka Server (Discovery)
All components that need to externalize their properties to a central place will register with Spring Cloud Config Server. The Eureka Server will also via the bootstrap process register as a client with Spring Cloud Config server and the Spring Cloud Config server will register itself as a Eureka Client with the Eureka Server which allows other services to use the Spring Cloud Config server to store and retrieve their service specific configuration details. Basically both the Eureka server and the Config server will act as clients for each other allowing dynamic discovery to occur.

Redis Cluster

Spring Session provides capabilities to externalize any session management to Redis cluster which is needed for a distributed system so session state is not restricted to specific application server instance. You can also use Redis cluster as a cache server for your Spring Services that are encapsulating your business functionality. Also if you plan to use RDBMS (specifically MySQL) as one of your data layer you can consider using Redis as your second level cache to improve the performance of your database calls.

NOTE: If you are building a Single Page Application, you can and should use JWT tokens to invoke Services via the API gateway in a stateless manner versus the traditional way of managing state via session cookies in a typical server-side web application.
Services

Depending on your design approach, your services can be built around business capabilities or DDD. In the diagram I am showing a sample application that is dividing services based on business capabilities. Each of these services in turn are simple Spring Boot applications that register themselves as discoverable Services with Eureka Server. It is through the Eureka server that other services discover each other so does the API Gateway (Zuul). This decoupling allows each of the services to be build and deployed independent of each other and allows us to horizontally scale them for High Availability and increased work load. For example, in order to include multiple instances of Shopping Cart Service, all you need to do is run multiple instances of Spring Boot application and have each of those instance use the same Service-ID that is used by Eureka server to identify those instances to any clients (Discovery Client) that needs to consume them. The client then gets to load balance request between those service instances which typically are stateless because any state that they have is either saved in Redis Cluster (for sessions/cache) or in Data layer (relational like MySQL or NoSQL like MongoDB).

Two project libraries worth mentioning in the context of Services are

Feign

It creates a declarative style REST client that allows services and clients to invoke REST API services quickly with annotation based approach and with very little boilerplate coding. You should use this client library in all places where you are consuming REST API exposed via Services. The other benefit that you get with using Feign is that you can make direct reference to Service names that are registered with Eureka Server as Feign can directly integrate with Eureka Server and can discover services automatically for you based on service name only.


Ribbon
Ribbon provides client side load balancing capabilities with configurable load balancing rules like round robin, availability filtering and weighted response. It also works with Feign to allow service consumers to access services that are load balanced on the client side using one of the pre-defined rules as well as customizable rules if the predefined ones do not fit your needs. Zuul Proxy also uses Ribbon to invoke Services.


Spring Security

As mentioned before, each of these components are simple spring boot applications that can easily take advantage of the Spring Security framework to perform authentication and authorization of the users.
Spring Security provides Authentication Providers for – custom databases, LDAP, SAML 2.0 (as SSO), OAuth2, CAS and a few more and if none meets your need you can always extent the security framework to provide your own authentication/authorization layer.


Data Layer
The choices to pick here are plenty and hence I am not going to elaborate on this component as your business needs will help you determine whether you need a relational database versus a NoSQL type of database like (MongoDB) or an in memory data grid like Redis cluster. One thing worth mentioning here is that, if you plan to use a Microservice architecture and do not want your data layer to be a single point of monolith communication. You can consider using a Database per Service approach which allows you to build individual services in a “silo” manner. One thing that you may then need to consider is how you achieve eventual consistency post that architectural decision. Some of the things to consider for doing that is mentioned below

• Event sourcing – you can use Apache Kafka as an event source. There are other alternatives like using an Event Store database. Conceptually what you are doing here is you are storing events in your database (either Kafka or EventStore) along with Payload of the entity whose state is going to change as a result of that event (or action). The way event sourcing works is you store these events and arrive at the final state of your entity by replaying these events from the start until you arrive at the final state (basically you are doing a replay or simply put as an analogy – you are seeing a cartoon character drawn at different static positions run when you flip through pages. This analogy brings back childhood memories for me). You may have to use techniques like snapshot to improve performance to ensure that overtime you do not start replaying the events from the beginning but start from a snapshot of the data at a more recent point in time.
• Saga patterns to achieve eventual consistency– Two of the saga patterns that are basically built on an Event Driven Architecture needs to be considered - Choreography-based saga and Orchestration-based saga. I am not elaborating on these as these are well defined patterns in an Event Driven Architecture
• Be prepared to design systems for eventual consistency

In a nutshell, your choice of Database per Service will make your implementation of a distributed system a little bit more complex given that you have to design for eventual consistency. If your business need does not require a Database per service approach I recommend using one database to achieve your ACID requirements.

Conclusion

I hope you liked this article as always send me constructive feedbacks.

Shibboleth IdP/SP, Spring Boot with Redis and MySQL – SSO flow

Introduction
In this article I will provide an overview of following technology stack

• Single Sign On (SSO) identity/access management using Shobboleth IdP, CAS server and Shobboleth Service Provider (SP)
• Using Spring Boot for Rapid Application Development
• Redis to cache Spring data as well as its use as a second level cache for ORM (Hibernate/JPA)
• I will also provide an overview of JHipster – a Swiss army knife for code generation

There is a lot to cover from the above technology stack, the intent of this article is to provide pointers to create enterprise grade scalable system and assist Software Architects in taking decisions that will help in scaling their systems using the mentioned technology stack. I know this is heavily tilted towards Java audience but a similar approach can be used for .NET using ER framework.

Architectural Overview

The above logical diagram represents a high level overview of the technology stack involved.

A typical user activity flow is described below

• User will start the flow by accessing the protected resource (web application) via the browser
• The Service Provider (SP) will intercept that request and will redirect the user to the identity Provider (IdP)
• After successful authentication and exchanging of SAML token the Service provider will redirect the user request to the web application and will set Authorization Headers that the web application can read and determine what the user is authorized to do. These authorization headers are not sent back to the user browser for security reasons. Typically you will have an Apache server hosting your Service provider acting as a reverse proxy for your web application which is hosted on Tomcat.
• Depending on what the user is authorized to do different User interface functions will be enabled for subsequent actions.

In this flow, I am assuming the authorization piece (roles, access policies etc.) is application dependent and in the above diagram is stored in RDBMS (MySQL) database. It’s recommended to externalize this part for all applications into a Central Access Management System (similar to Identity Authentication - IdP) thereby standardizing access management across all applications.


Component description:

LDAP server

In the overall Single Sign On (SSO) flow, you want to ensure your user passwords are centrally managed. LDAP server represents one such option

Identity Provider (IdP) – like Shibboleth IdP or CAS Server

In the SSO flow, this component will abstract the authentication piece from multiple applications and will provide a standard interface for authentication to multiple applications. You can either use Shibboleth IdP which uses SAML protocol for authentication or CAS server if you are using the CAS protocol. (NOTE: CAS server also supports SAML).

Service Provider (SP) – like Shibboleth Service Provider

In a SSO flow you can protect your web application behind a Service Provider which can co-ordinate the SAML SSO flow with your IdP and pass the authentication details via HTTP headers to your application. Shibboleth SP can be used as a service provider to perform such a function. If you happen to use the Docker image for this component then it will be running Apache server and can be configured to act as a reverse proxy for your Web application. Alternatively you can co-locate the SP with your web application. If you are using Spring then you can use Spring Security SAML that provides SAML support within your application and can act as a Service Provider for your Identity Provider. I still prefer a separate Service Provider configuration via Apache reverse proxy approach as that is abstracting the cross cutting responsibility from individual applications and also provides an extra layer of security through proxy function.

Web application

This is the protected resource (the web application) that the users are accessing in the SSO flow. In the following section I will provide further details on this component.

Redis Cluster (or AWS Elasticache or Azure Redis cache)

You can use Redis as a distributed Data Grid to reduce read load on your relational database. The Redis cluster can be build using virtual machines in your local data center or you can use cloud providers like AWS/Azure that provide Redis cluster as a service. You can use client drivers (like Redisson) that can then communicate with your Redis cluster. Redisson also provides capabilities to integrate with Hibernate to provide second-level caching via Redis to JPA queries thereby reducing database queries and speeding up performance of your application. Redisson comes in two flavors – professional edition and open source edition. I recommend going with professional edition as that gives you features like local caching, data partitioning, XA transaction to name a few. These features will come in handy to provide enterprise level scalability (and the cost is not that bad)

RDBMS (like MySQL)

In the diagram this component is where your data storage will occur. If you are using MySQL you can create read replicas on top of your Redis Cluster to further improve read performance. You can also use a NoSQL database like MongoDB instead of a RDBMS like MySQL to store data.

A few implementation details for the components to consider

There are Docker images available in Docker Hub for

• Shibboleth IdP
• Shibboleth SP
• CAS Server
• Tomcat
• OpenLDAP (if you want to use this as your LDAP server)
• Redis
• MySQL

You can use these images to containerize your application and deploy it in container aware environments like Kubernetes (K8s), EKS, AKS, GKS or ECS. Containerizing your components will help you design and build a scalable, highly available and resilient system.

Web Application Architecture:

The above diagram provides further decomposition of the web application component.

Spring MVC/Spring Boot

On the left hand side of the dotted line, you will see decomposition of a typical Spring MVC framework. The right hand side shows the same using Spring Boot. If you are heading towards Micro service style architecture and wish to use Java platform then using Spring Boot is the way to go. It allows you to build applications rapidly. If you are building even monolithic applications I suggest you start with Spring Boot since it utilizes under the hood the same concept that Spring MVC does – Dependency Injection (DI) and Inversion of Control (IoC). It also reduces a lot of boiler-plate code which encourages rapid application development. In my opinion, in the Java world Spring Boot is the answer to other competing technologies like NodeJS, Ruby on Rails and/or Django framework (python) in order to do rapid application development.

For the rest of the topics in this article I am going to assume the use of Spring Boot

Redis Cluster (or AWS Elasticache or Azure Redis cache)

This component is used to provide caching at two levels

• At the application level (Spring) and
• As a second level cache for Hibernate

NOTE: Hibernate provides first level caching at the session level by default.

As mentioned before if you are using Redisson client (open source or professional), it supports connecting to local data center Redis cluster as well as AWS/Azure hosted Redis clusters.

View

You have multiple options to pick from

• If you are looking for server side rendering of user interface (UI) then you can pick either JSP or Thymeleaf or any other templating engine.
• If you are building a Single Page Application (SPA) you can select Angular, React or Vue (there are many more choices to pick from in the SPA world but these three cover 80%+ of the SPA market interest)

In the next section I am going to talk about a code generation tool called JHipster that will allow you to build production grade code very quickly.

JHipster:

The above diagram provides a high level overview of how you can integrated SSO with SPA application. I thought of including this section so I can talk about one of my favorite code generation tool – JHipster. I call it a Swiss army knife for code generation. In the above diagram the dotted line section is generated automatically for you by JHipster.

The user interaction flow described earlier is still valid which is the user starts the flow by trying to access the protected resource (the SPA application in this case), the only delta here is as follows:

Once the service provider redirects the user internally (via reverse proxy) to the location where the SPA application is hosted. The rest of the interaction from that point forward in terms of authorization between the SPA application and the server side component (which typically is REST API exposed via Spring Boot) is done by exchanging JSON Web Tokens (JWT).

In order to code generate with JHipster, you typically start with creating an Entity or Domain model using JHipster online graphical editor. You can then save the model to a file which contains the textual representation of the graphical model in what’s called JHiphster Domain Language (JDL). You can feed that JDL file to JHipster’s code generation engine and voila you have a working application with basic CRUD operations ready for all your Domain entities. JHipster has a lot of options to generate code for different technology stacks but at a high level you get to decide if you want to create

• A Monolithic application or a
• Micro Service application

When you select a monolithic application, JHipster will auto generate code spread across following layers using technology stack that you select during initialization step. Some of the layer choices are mentioned below

• It creates a SPA application using either Angular or React at the client side.
• On the server side it creates a Spring Boot REST API driven server layer
• It also through Hibernate/JPA layer give you option to talk with RDBMS layer (MySQL, Oracle, SQL Server etc.)
• It allows you to pick caching option for Hibernate and Spring Boot. Unfortunately not Redis at the time of writing this article. However, given that Spring Boot and Hibernate provide annotation based caching capability that is independent of which Cache provider you are using. The amount of time to change the code to use Redis is very much restricted to touching one or two files at the maximum. The rest of the Spring Boot/Hibernate code is unaware of the underlying caching provider. With that said JHipster does generate code automatically for Ehcache, Hazelcast, Infinispan and Memcached. With JCache API I have a feeling that JHipster code generation will cover most of the other caching providers including Redis in near future.
• It integrates the client side calls with the server side call using JWT token authentication flow built into the Spring Security. If you want to authenticate users using the Identity provider approach mentioned earlier. All you have to do is use the Pre-authentication flow that Spring Security provides out of the box thereby delegating the authentication piece to your IdP and then you can use JWT tokens for subsequent authorization. NOTE: It is also possible to use OpenID Connect for SSO with Single Page application as an alternative to IdP.

A few things at the time of writing that you should be aware of in terms of Jhipsters cons

• JHipster does not auto generate code for versioning or Optimistic locking for hibernate entities
• JHipster requires JDK 1.8 for code generation. However, you can run the code generated by JHipster on Java 11 but you will receive warnings that some of the class libraries (especially Hibernate) are using native calls which is discouraged. Anyhow I do not think this is necessarily going to be a big issue once JHipster catches up with using the right version of Hibernate in its next release that does not use these java native calls. Just so you know your code will still work with Java 11, you will get some nasty warning’s in your application server logs that for the time being you can ignore.
• Given the choice of technology stack and more specifically the component versions within the individual technology that JHipster uses to generate code, one thing you need to be aware of is being locked down with specific version of that technology. For example, if JHipster code generation is using the most recent version of Angular today, six months down the road that version may become obsolete and you will have to do the upgrade manually or wait for JHipster to do catch-up.
• It only generates code for CRUD operations and does not generate code for business specific workflows which you still need to design and code using your developers but in my opinion it generates a very good code base for you to build from

Conclusion

I hope you liked this article. I plan to write an article on Event Driven architecture as well as Microservices when to use them and most importantly when not to use them next. As always send me constructive feedbacks.

Scaled Agile, SAFe framework and TFS (VSTS) use for the same

Introduction

In this article I plan to describe how you can use Team Foundation Server (TFS) or Visual Studio Team Service (VSTS) to be Agile in a large scale project with multiple teams working in parallel. I plan to use Scrum terminology a lot in this article and if you are unfamiliar with Scrum then I suggest you read my article on Scrum at: https://www.linkedin.com/post/edit/project-management-using-jira-agile-scrum-jayesh-nazre . In that article I start with describing the terms used in Scrum and then use Jira’s Scrum template to explain how you can be Agile using a “fictitious web based application” project as an example. In this article I will use SAFe (scaled agile) framework and explain how TFS can be configured to use SAFe framework for a large team project. I am going to assume that you are familiar with Scrum terms for the remainder of this article.

I am also not going to spend much time providing an overview of SAFe framework in this article as I believe the following website provides a very useful interactive user interface to understand what SAFe framework is https://www.scaledagileframework.com/

Architectural Overview

A few observation points about the above diagram

The above diagram is providing a simplified overview of SAFe Framework. The color coding is used by me to help visualize how work gets decomposed from top to bottom. Typically you start with “Epics” in the Portfolio management layer, which is then decomposed into “Features” at the Program management layer which then gets decomposed into “Product Backlog Items (PBI)/User Stories (US)” at the Team level. In TFS/VSTS, there are different templates available when you set up a Team Project. Depending on which template you pick you will get a set of predefined work items. For example in Agile template you get User Stories and in Scrum template you get Product Backlog Items. I am going to use the PBI/US interchangeably within this article. Also unless mentioned otherwise whatever I say for TFS is also applicable for VSTS. Within TFS, the term “Work Item Type (WIT)” is used as a generic term to mean any of the following – Epics, Features, User Stories, Product Backlog item, Tasks, Bugs etc. Depending on which template you select some of the WIT will be available automatically to you within the project. For Example, if you use “Scrum” template following WIT will be made available automatically – Epic, Feature, Product Backlog Item, Task, Impediment and Bug. Instead of Scrum, if you select “Agile” template then following WIT will be made available automatically to you – Epic, Feature, User Story, Task, Issue and Bug.

At the Team level:

In Scrum, you start work by picking work items from the “Team backlog”, into the next sprint during “Sprint Planning”. A Sprint typically runs for 2 weeks (it can run for 3-4 weeks).

At the Program level:

You start with Program Increments (PI) which typically run for 10 weeks. Each iteration within PI is called a Cadence which typically lines up with a single Sprint duration of 2 weeks. The First Cadence is when you do PI planning and the last Cadence is when you do Innovation and Planning. As you can see in the diagram, the framework at the program level aligns with the framework at the Project team level to provide scaling, For Example:

• You do Sprint Planning at the beginning and you do PI Planning at the beginning of PI
• You do Sprint Retrospection at the end of the Sprint and you do Innovation and Planning at the end of PI.

At the Portfolio level:

This is where strategy and investment funding are defined for value streams and their Solutions. The portfolio level aligns enterprise strategy to portfolio execution by organizing the lean agile enterprise around the flow of value through one or more value streams. TFS has limited capability to address all aspects of Enterprise Agile Management and Value Stream Management. VersionOne (now Collabnet) is a better software to provide you with the SAFe framework experience than TFS at this level. Within TFS, at this level you have the capability of using Kanban Board just like you do at other levels to define, prioritize, execute and monitor work progress (The Kanban board uses Epics at Portfolio level, Features at Program Level and PBI/US at Team level as show in the diagram). TFS also allows you to use “tags” as a quick and easy way to map Epics to their Value Streams, strategic themes, and associated budgets. But the capability TFS provides is by no means close to what Versionone provides.

Some of the things that I have not shown in the above diagram and is worth mentioning is that Continuous Integration/Continuous Deployment (or Delivery) (CI/CD) and Release Management is also something that needs to occur at the Program/Team level and automating that is what provides agility. TFS provides CI/CD and Release Management capability. TFS allows source code versioning through Team Foundation Version Control (TFVC) or Git. Within SAFe Framework, Agile Release Train (ART) is a team of individuals whose goal is to work towards Feature release automation at the program level. Similar automation is also done at the Project Team level but is restricted to Sprint release automation. Typically System demos are events that are organized throughout the PI duration where multiple teams demo what Features are “release ready” and allow teams to collaborate and receive feedback from sponsors, stakeholders, and customers. A similar demo event is done at the Project Team Level at the end of each Sprint.

TFS/VSTS configuration

The above picture provides an example of how a fictitious project – “CRMProject” is decomposed into different levels within TFS/VSTS.

A few observation points about the above diagram

The above backlog view is visible to all Portfolio level users and is providing a drill down visibility to Portfolio users.

Points 1 through 4:  

These points provide a drilldown detail view of which Epics are associated with Features which in turn are associated with PBI which in turn are associated with Tasks providing a top down visibility. It’s possible to control what is accessible to users at what level depending on which TFS groups the TFS teams are assigned to. Within TFS access security controls are enforced through TFS groups, TFS teams and associating users to TFS teams.

Point 5:

Within TFS, Area paths is how you control which TFS team can add which WIT and which TFS teams/TFS groups have access to each Area Path to do the same. This is how TFS allows you to create the top-down hierarchy (Epic to Feature to PBI to Task).

Point 6 and 7:Value Area's along with "Tags" can be used to map Value Streams, strategic themes, and associated budgets to "Epic" - "Feature" - PBI/US. Within TFS, you can define search queries that filter WIT based on "Tags" and share the Query results across teams.

In order to do “PI” at the program level and Sprints at the Team level within TFS you configure the iterations for the project as shown below

Once the iterations are configured as shown in the above hierarchy. You can assign Sprint iterations to multiple teams as shown below

TFS/VSTS Build and Release Management

The above diagram provides a high level overview of Continuous Integration/Continuous Deploy/Delivery (CI/CD) pipeline capability of TFS/VSTS.

A few observation points about the above diagram

TFS provides complete end to end visibility where in you can associate WIT (like Epics, Features, PBI/US, Bugs) to code checked into the TFVC/GIt at the time of checking in the code. You can also trigger automatic build process based on code check-in event (For example checking in code in “Release” branch in Git). You can also use Gated check-in build with TFVC “only” where in successful build of code before checking it in is mandated. The above diagram shows how Agile Release Train (ART) can be realized through CI/CD pipelines to deliver Features to the customers in continuous manner there by providing value. It’s possible during Release management for an approval workflow to be enabled so releases into production environments can be controlled via a human interaction (an approver). Releases can also be initiated based on triggers like checking in code into a specific Release branch, successful release of code in one environment triggering another environment’s release or doing manual releases.

One of the things that I have not shown in the above diagram is that it’s possible to run automatic tests from Test plans as well as run Unit test on application code that’s being built.

A few take away

• SAFe framework and Scrum of Scrums (SoS) are two different things. Scrum of Scrums (also called Meta Scrum) is defined by the Agile Alliance as “A technique to scale Scrum up to large groups”, SAFe, on the other hand, is a complete framework. Put differently, SAFe is prescriptive via its framework, SoS is more of using best practices from Scrum and scaling it to a larger team but is not a very prescriptive approach and assumes that teams are mature enough to scale and be agile.
• Do not try to use SAFe to manage a project with waterfall mind set. That will defeat the purpose of why Agile frameworks exist. The idea is to change the mindset not replace waterfall with another methodology which will end in having the same results as a waterfall model does.
• Although I am showing how to use TFS/VSTS to scale Scrum/Agile projects using SAFe framework. You can also use other software’s like – VersionOne, Jira to do similar scaling. With Jira you can use the BigPicture plug-in to incorporate the SAFe framework within Jira as out of the box Jira does not provide SAFe framework configuration. VersionOne and Jira (with BigPicture) is also a good alternative if you are a non-Microsoft shop.

Conclusion

I hope this helps you understand how to scale Agile/Scrum methodology for large projects using TFS/VSTS and SAFe framework.

Salesforce CRM

Introduction

In this article I plan to provide a high-level overview of the three main modules (Salesforce calls them clouds) of Salesforce CRM and then proceed to explain the key entities (Salesforce calls them Objects) that play a crucial role in each of the three clouds. I will then end the article by providing analysis in assisting you with your make-or-buy decision.

Architectural Overview

Salesforce has three main modules - Sales cloud, Marketing Cloud and Service Cloud. There are many other modules (clouds) available from Salesforce but the most widely used one is “Sales” cloud. Salesforce also provides Platform as a Service (PaaS) thought Force.com and Heroku, the latter is a company it acquired. Force.com is the platform on which all clouds (modules) of Salesforce are build. Heroku is a true PaaS that allows developers to deploy their applications written in languages like Node, Ruby, Java, PHP, Python, Go, Scala, or Clojure into its managed containers without worrying about the underlying infrastructure.

NOTE: The reason I call Heroku a “true” PaaS is because Force.com is useful for Salesforce cloud customization and everything Salesforce related. Heroku is not.
The Force.com platform has gone through some architectural changes from its “Classic” version to its “Lightning” version. As shown in the diagram, in the classic model, Salesforce used a Model View Controller (MVC) pattern to allow custom web pages to be built utilizing Visualforce, Apex classes and Salesforce objects. The design was more page centric, later they came up with a component centric design approach and gave a new fancy word to it called “Lightning”. This approach is basically based on Model Controller-Controller View (MCCV) pattern. The component centric approach allowed the same component to be reused across multiple pages. In Lightning, as shown in the diagram there are two controllers one on the client-side (written in JavaScript) and another on the server side (written in Java like language - apex classes). The latter is your classic style controller. Having two controllers allows Salesforce to change the client-side framework independently of the server-side framework and utilize SOAP/REST API for exchanging data, thereby providing architectural decoupling. Since the data exchange occurs behind the scene the user interface is not rendered on browser or phone “apps” every time an action is performed by the user on the screen there by providing a more responsive UI experience (hence the term Lightning, which by the way is what any Single Page Application framework does)

Key Objects in Salesforce CRM

A few observation points about the above diagram

The CRM process flow shown in the diagram provides you with key Objects that are heavily used within Sales and Marketing cloud. There is a lot of functionality overlap between the two and as such Marketing and Sales go hand in glove in any organization so its obvious Salesforce ended up with overlaps as well. The blue arrow flow in the diagram is my way of showing how the process flows in a CRM application, which is

• We start with Campaigns – like trade shows, email reach out, Webinars etc.
• These Campaigns may result in potential Leads
• Some Leads will get converted to Opportunities and when that conversion occurs you can create other entities like Contacts, Accounts etc.
• When Opportunities are won, you can create Contracts

The above bullet points is pretty much the meat of the two clouds (Sales and Marketing)

The Service cloud is typically used for IT service management (ITSM) and provides capabilities like

• Opening a Case
• Tracking the various stages a case goes through until it’s resolved.
• Using Solutions (with Classic) or Knowledge Objects to categorize resolutions and Issue/Article types which can then be used by your help desk staff personnel’s for resolving similar Incidents in future, basically to provide a Knowledge base. You can then use the Knowledge base like a “google” Help Desk search engine.

NOTE: Coming from object oriented and database designing world, I am uncomfortable calling Entities and Classes as “Objects”. In my mind, objects are class instances but Salesforce uses the term Objects to mean Entities and Classes.

Make-or-Buy Analysis

The above flowchart will help you determine when you should use any CRM system like Salesforce as a SaaS versus building it from ground up.

A few observation points about the above flowchart while you decide which way to go

If you are a small size company that does not plan to use developers in the long run beyond the initial setup phase then you should go for Salesforce CRM or any other competing SaaS CRM software. When you do go in that direction make sure that you do not customize the CRM software using programming capabilities of the software, instead try to customize through configuration (declarative) changes provided by the Salesforce Admin pages (Setup), so basically what I am saying is use things like Validation Rules, Page Layouts, Profiles, Workflow Rules (for classic), Process Builder (for Lightning), Approval process etc. versus custom coding with Visualforce, Apex Classes, Triggers, Controllers etc. In a nutshell, try to change the business processes in your company to adapt to what Salesforce provides out of the box without having a need to custom code. Do not do custom code - just speak with other people about their painful experiences (cost wise) in converting Classic-style custom code to Lightning-style equivalent. By avoiding custom code to begin with, such drastic changes done by Salesforce will have minimal to no impact on your migration path thereby avoiding the need to have developers.

NOTE: With classic you will typically need developers who come from Java/JEE background. For Lightning you will need developers with Java/JEE background as well as developers who are very good with JavaScript. So you see how this defeats the purpose of SaaS. Especially for a small sized company. Hence I suggest no custom coding.

If you find yourself using developers to constantly tweak the salesforce platform using custom coding options (like Apex classes, Controllers, Visualforce, Lightning App Builder etc.) provided by Salesforce then I truly believe you are better off custom building your own CRM application that will save you money in the long run and here is why –

• Physical datacenters are disappearing and are getting substituted with  Cloud providers like AWS, Azure, GCP and each of these cloud providers provide Infrastructure As Code (IaC) capabilities that allow you to build data centers in a matter of minutes using templates. Point is – it’s much easier for companies to manage data centers these days with IaaS than it was in the past
• Salesforce is heavily Java like in terms of its custom coding capabilities and you will end up with Java developers working for your company if you do too much customization to the Salesforce platform. So why not ask the Java developers to build your CRM, you saw how simple the CRM process flow is in my earlier “Key Objects in Salesforce CRM” section, I believe you can built your own CRM with a team of 3-5 developers within 6-8 months’ time period (or at least get most of the functionality built). 
• Salesforce uses MVC pattern for classic version and MCCV pattern in Lightning. Both these patterns are widely used in software designing. They are primarily used to avoid ripple effects from changes in any of the three layers - the Presentation (View) layer, Business (Application) Layer and Data (Database) layer. These patterns are also widely used in modern day application programming languages/frameworks like ASP.NET MVC (C#), Java (JEE), Django (Python), Ruby on Rails, NodeJS etc. 
• There are many UX/UI application frameworks (React/Redux, AngularJS, Ember, Meteor, Vue etc.) that allow you to build Single Page web applications in shorter time period than before. My point is – application development is not how it used to be in the “dotcom” era, a lot has changed over the years and rapid application development is truly possible these days with advancements made in software development and more specifically these application frameworks are becoming more stable and mature as time goes by.
• With cloud providers giving NoSQL services (like DynamoDB, Azure Cosmos DB, Cloud Bigtable, Cloud Datastore), managed relational services (like RDS, Redshift, SQL Database, SQL Data Warehouse, Cloud SQL, BigQuery) and Big data services (like EMR, HDInsight, Cloud Dataproc) it’s easier than before to manage and scale data.
• The technology you pick behind your custom built CRM application framework changes at a faster pace than Salesforce and you will be able to perform upgrades much faster than Salesforce giving you the competitive edge over others who are using Salesforce for their CRM platform – very key differentiating factor to win new customers. 

Conclusion

I hope this article was helpful. My intent in the Make-or-Buy analysis section was to convey the point that if you use a CRM software “as is” with no custom coding then you get the biggest return for your investment otherwise you are better off with home grown equivalent.

Internet of Things (IoT), AWS & Raspberry Pi3

Introduction

In this article I plan to describe how you can use Raspberry Pi3 as a Single Board Computer (SBC) to read temperature and humidity through DHT11 sensor and use AWS IoT as the MQTT message broker to send the data to AWS for further analysis.

Architectural Overview

In the architectural diagram above the arrows are showing a flow of events with numbers used to explain the order in which these events occur.

Point1:

This is the Raspberry Pi3 circuit that is generating sensor data. I did not use a microcontroller here to make things easy for me.

Point2:

AWS IoT is being used as a Message Broker to collect device data. Although I am using AWS IoT here, other cloud providers like Azure’s IoT Hub or Google’s Cloud IoT will work as well. If you do not want to use a cloud provider and want to have a self-hosted MQTT message broker you can use Eclipse Mosquitto. All these providers support MQTT protocol which is the protocol I plan to use with Raspberry Pi3. Other protocols are also supported by cloud providers but MQTT is widely accepted standard. MQTT is a publisher/subscriber messaging protocol which fits well in a disconnected architecture like IoT. Its bidirectional so you can send information/configuration instructions to devices as well.

Point3:

AWS IoT allows devices to publish to an AWS Topic. In this diagram, the device (Pi3 with DHT11 sensor) uses AWS IoT MQTT broker to send data to the topic.

Point4:

AWS allows you to define rules that can trigger actions based on data that is published by multiple devices to a specific Topic. You can define multiple rules for the same Topic.

Point5:

AWS allows multiple AWS services to be integrated with AWS IoT through the execution of actions that are run by the AWS IoT rules. In the diagram a few of the AWS services are listed, most notably - AWS Lambda, AWS DynamoDB, AWS S3, AWS SNS/SQS, AWS Kinesis (for real time streaming) and ElasticSearch. NOTE: The list is not complete

Point6:

Once the IoT device data is received at the AWS end and the appropriate AWS service is used to store the data, it’s possible to do data analysis on the same. For example: If you save the data in DynamoDB you can do data analytics using AWS QuickSight.

Raspberry Pi3 Circuit Diagram

The above diagram shows a logical circuit diagram of how Raspberry Pi3 is connected to the DHT11 temperature and humidity sensor. I am not showing the actual pin connection to keep this article simple but there is a bread board missing here which is connecting all of these components. A few observation points about the above diagram

• On the MicroSD card of Raspberry Pi3 I have installed Raspbian Stretch Desktop OS (Debian based Linux OS)
• I am using python libraries to communicate with GPIO pins instead of “C” program, just to make life easy for me. Basically I am using the “RPi.GPIO” libraries that come with Raspbian Stretch OS 
• I have installed AWS CLI client on the Raspberry Pi3 to allow it to communicate with AWS IoT
• I have installed the “AWSIoTPythonSDK.MQTTLib” python library to allow Pi3 to communicate with AWS IoT MQTT broker as a MQTT client
• The resistors in the diagram are used to control the current flow.

The logical flow of events is as follows

Point 1:

The pressing of button controls when Pi3 will read sensor data. I am using a button just to control the event flow through a user driven action. If you want you can remove the button and the LED from the above circuit and let the Pi3 read temperature and humidity at periodic intervals directly from DHT11 sensor

Point 2:

Once the button is pressed, I am using a LED as a visual indicator to tell me that Pi3 will be reading sensor data

Point 3:

The button also sends a signal to Pi3 via the GPIO pins as a trigger to read sensor data

Point 4:

Pi3 reads the temperature and humidity data from the DHT11 sensor once it receives the trigger from the button press event.

Point 5:

Pi3 then uses the “AWSIoTPythonSDK.MQTTLib” python library to send data to AWS IoT. Once the data reaches AWS IoT, then data analysis can be done from that point forward using AWS services shown in previous architectural diagram.

A few take away

Things you should consider exploring but is not elaborated in this article are listed below

At the AWS end

• It’s possible to use AWS Athena to do data analysis directly from S3 bucket
• You can do real-time data analytics using Spark and EMR cluster of AWS and use AWS Kinesis as a streaming source which streams data from multiple IoT devices.
• If you want to do any data transformation on the IoT data received from devices, you can consider using AWS Glue/Data pipeline.

At the IoT end

• You can use a Passive infrared sensor (PIR motion sensor) and connect it to Raspberry Pi3 and built your own home security system that sends you email alerts when someone enters your house or you can connect a Piezo buzzer to your Pi3 along with the motion sensor to sound an alarm
• You can also connect a Relay to Pi3 and then use it to control pretty much anything over the web, I suggest using a smart Relay like Adafruit's “Controllable Four Outlet Power Relay Module version 2” that has an inbuilt relay so you can connect low voltage circuit like Pi3 at one end and have the power strips for high voltage at the other end to which you can connect any electrical appliance like (air conditioner, table lamp etc.). This setup will allow you to control pretty much anything in your house from anywhere over the cloud. Using a smart relay will avoid the need to work with high voltage wires and the need to connect them directly to a regular relay, safety first.
• You can measure pressure, temperature and altitude using BMP180 digital pressure sensor that can be connected to Pi3.

Conclusion

I hope this gets you excited with IoT and encourages you to do some DIY projects. I have given some DIY project pointers in the “A few take away” sections in my article for you to get started on. Good luck.