In today’s fast evolving digital world, speed-to-market is what matters for businesses across industries. To achieve this the entire software development lifecycle needs to become faster, agile and scalable. DevOps has pretty much become the new normal to accelerate the software development and delivery processes. According to a recent IDC study, the worldwide DevOps software market is forecasted to reach $6.6 billion in 2022. Continuous Integration, Continuous Delivery and Continuous Deployment (CI/CD) are considered to be the cornerstone of DevOps. Following Agile development methodology, in the past companies could achieve monthly, quarterly, bi-annual or annual releases. However, in the current DevOps era, we see multiple releases taking place in a single day. With over 50% of organizations across multiple industries embracing DevOps last year, continuous integration and continuous delivery has become an integral part of the software development process.
In this whitepaper, we will deep dive into the concept of continuous integration, continuous delivery and continuous deployment and explain how businesses can benefit from this. We will also elucidate on how to build an effective CI/CD pipeline and some of the best practices for your enterprise DevOps journey.
The “shift-left” approach is getting increasingly adopted in the Agile and DevOps world for detecting issues early in project lifecycle and prevent them from getting discovered at a later stage. CI/CD has further enabled the implementation of the shift-left approach. Shifting-left requires two key DevOps practices: Continuous Testing and Continuous Deployment. Continuous deployment automates the provisioning and deployment of new builds, enabling continuous testing to happen quickly and efficiently.
Shift-left includes a test driven development approach which is geared towards reducing errors and improving flexibility while designing a software. It reduces the risks and consequences of things going wrong and increases the speed to market. It is perfect for the successful implementation of fearless software development. Its main aim is to build smaller testing cycles into the process and bring more agility.
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. By integrating regularly, one can detect errors quickly, and locate them more easily. Earlier new features were built in isolation and each of them were submitted at the end of the cycle.
Often bugs were discovered later in the product development lifecycle which would result into increasing build costs. To better understand the benefits of CI, let’s imagine that you have a project which you have broken down into 10 modules. Assume that you’ve given one module to each developer. So the developers will write the code and then they are also responsible to test it which we call unit testing. So, say the unit test is done and everything is working perfectly but then all of your developers have been working separately. So now, all of their code have to be integrated and integration testing has to be done so that your operations team can deploy the project on a production server. Let’s say it was decided that your developers will work throughout the week and will integrate the project on the last day of the week. When they integrate there will be an automated testing done to the build. What happens when this testing throws up bugs? We now have a problem which is absolutely necessary to solve and you don’t want to wait another week to release your project.
To solve this problem the concept of Continuous Integration was contrived. In CI, your developers will have multiple commits a day. So every time they make a change and it is working for them in their unit test, they can push the code to a shared codebase like Github. So, every developer in the team will push code frequently to the shared codebase. Every time they commit, there will be an automated test done immediately. Now after every commit you have a ready build which is tested and working and you don’t have to wait for a week to release, you can keep releasing continuously.
Continuous Integration is backed by several important principles and practices.
Continuous delivery is a series of practices designed to ensure that code can be rapidly and safely deployed to production by delivering every change to a production-like environment and ensuring business applications and services function as expected through rigorous automated testing. This is an extension to CI, wherein the delivery process is automated enabling anytime deployment into production.
Continuous Delivery at Work
At the core of the continuous delivery, lies a deployment pipeline based on which the team automates the testing and deployment processes. This pipeline is an automated system that executes a set of test suites for each of the builds. In each segment in the pipeline, the build may fail a critical test and alert the team. Otherwise, it continues on to the next test suite, and successive test passes will result in automatic promotion to the next segment in the pipeline. The last segment in the pipeline will deploy the build to a production-equivalent environment. This is a comprehensive activity, since the build, deployment, and the environment are all exercised and tested together. The result is a build that is deployable and verifiable in an actual production environment.
Continuous Delivery Tools
Continuous deployment is a strategy for software releases wherein any code commit that passes the automated testing phase is automatically released into the production environment, making changes that are visible to the software’s users.
A major advantage of continuous deployment is the small change sets. A small change is easy to code -> build -> test -> deploy -> monitor. You would not have to wait for once-a-week, once-a-month, once-a-quarter, or even once-a-year to release. The change set in such cases is usually large to massive depending on the release cycle, requiring a long and painful testing process. So, with continuous deployment, features go live within hours or sometimes within minutes instead of a month.
Since deployment is fully automated, continuous deployment also speeds up your time to market. Thus, continuous deployment is a safe, reliable and faster option to release new products, updates, new features etc.
However, continuous deployment can also get risky as we are making changes directly in the production server. Any unprompted build can quickly end up in a big mess with delayed problematic releases again. Therefore, most enterprises dealing with large software products prefer continuous delivery to continuous deployment.
Continuous Delivery and Continuous Deployment are two terms which are highly conflicted in the industry. The confusion occurs because continuous delivery and continuous deployment are very similar in nature but there is one particular difference that splits them apart.
In order to understand this difference you need to know that when you deliver a software there is an entire software delivery lifecycle and multiple environments in the middle. We call this the delivery pipeline. You start off with the version control system then you code in the development environment and then you build the software. You have your QA environments, which is usually more than one, as you’ll need QA environment to support each kind of environment and you are going to be eventually supporting your software. Then we have several other pre-production or non-production environments, for example system integration tests, user acceptance tests etc., depending on how your organization is structured or how your delivery pipeline is structured and what kind of different environments you have. Finally, you have production where your software actually runs and customers interact with it. So, what you need is to deploy software from one environment to the next. For that in DevOps we use automation. There are delivery automation tools like AWS CodePipeline to help you automate your delivery pipeline. However, you do not directly deploy to the production environment. You first deploy on a production like environment to check the performance of the system and conduct other tests. Only when you are fully satisfied with the performance and functioning of your software, will you deploy it to the real production environment.
Thus, continuous delivery is the capability to deploy software to any particular environment at any given time including the code changes, configuration changes and also changes to the environment if the environment has changed since your last deployment. This is a must-have capability for DevOps.
Continuous deployment, on the other hand, is the capability where any change the developer makes gets automatically deployed all the way up to production. You might have heard about born on the web companies. These companies will deploy multiple changes every day. These companies need to constantly keep their systems updated and thus they need to continuously deploy to production.
However, continuous deployment is risky and if you are building some large enterprise software you should not do continuous deployment but you need to do continuous delivery. So that you are absolutely sure about the performance of your software before you deploy. Thus, continuous delivery is a capability you must have and continuous deployment is an option which you might choose to exercise based on the kind of product you have.
Your organization will grow and application and infrastructure complexity is likely to increase. Good DevOps processes become very essential in order to effectively deliver software. Continuous Integration and Continuous Delivery are a vital part of your DevOps processes. CI/CD enables your organization to grow your team, codebase, and infrastructure.
Having an automated delivery pipeline is crucial for serverless applications since they tend to be very distributed systems. Hence, if you don’t have a well-defined automated pipeline, deploying serverless applications can become very difficult.
In order to deploy your applications several times a day and at the same time feel secure about it, one needs to have an effective automated delivery pipeline. Keeping that in mind, here are some best practices to follow when designing your pipeline.
1. Adapting your CI/CD Pipeline to your Development Process – The CI/CD pipeline is the automatization of your development process. Adaptation to fit your development process strategy is an integral part. The first step is formalization of the development process which helps you to determine which branches you have and what their functions are. Only then you are able to create the correct pipeline.
2. Identifying the Tests that can be Automated – It is unlikely that 100% of the tests can be automated. Since test automation is core to CI/CD pipeline, identification of those test cases which can be automated is one of the crucial and vital best practices for CI/CD.
3. Testing in a Clone of Production Environment – A staging environment or a clone of the production environment is required to be built. This should be capable of maintaining the technology stack and composition which should allow developers to perform testing.
4. Documentation is an Absolute Necessity – Project requirements change/evolve along with the course of project execution. The test automation strategy should be devised keeping in mind the short-term goals and long-term goals into account. There may be changes to the test plan or test strategy over a period of time. It is important to have everything documented. It should be a free-flowing document, updated with important timelines and should be version-controlled. One should be able to revisit the document to track the progress.
5. Maintaining a Central Repository for Code Development – To maintain the source code on central repository is of utmost importance and one of the main best practices for CI/CD pipeline. This way the developers can keep their changes up-to-date with the latest source code available on the production server. A revision/version control system is also important to track changes and identify differences.
6. Choosing the Right Tool for CI/CD Process – There are number of tools available for CI/CD. One should choose the right tool based on the budget, and requirements. Some of the commonly used CI/CD tools are Jenkins, Travis CI, Gitlab, TeamCity, Codeship, Circle CI, etc. Before deciding on any particular tool, it is recommended that you research on the pros and cons of that particular tool. Any change in the tool during the course of development could affect the deliverables and deadlines.
7. Making the Pipelines Fast – Making your pipeline fast is a very important requirement for the team. This facilitates productivity and enables fast turnaround. If a team needs to wait an hour long for the pipeline to complete in order to be able to deploy a one line change, then the task becomes arduous.
8. Writing an Extensive Test Suite – To be successful with CI/CD, the team needs to develop a culture of writing tests for each code change they integrate. This should include writing unit, integration, regression and smoke tests (to name a few).
9. Provide an Easy Way to Rollback – One of the most important parts of a successful CI/CD pipeline is an easy way to rollback your changes in case something seems to go wrong. Idempotent deploys can prove to be useful where doing a rollback just means re-deploying a previous release.
10. Securing your CI / CD Environment – The CI/CD system has complete access to your codebase and credentials that need to be deployed in various environments. Hence, it is essential to secure it so that you are able to safeguard internal data. CI/CD systems should be deployed to internal, protected networks which are unexposed to outside parties.
Following some of the basic principles will help you to avoid some common pitfalls thereby, strengthening your testing and development practices. Adopting CI/ CD best practices allow greater collaboration among the team members and increases overall efficiency and agility. It helps you to work together seamlessly eliminating bottlenecks and achieve faster time to market.
Enabling Continuous Product Innovation for a Leading Life Insurer Leveraging DevOps
About the Client : The client is one of the leading life insurance providers for individuals and groups. It offers a range of individual and group insurance solutions that meet various customer needs such as Protection, Pension, Savings, Investment and Health. It has over 30 individual and 10 group products in its portfolio, catering to a diverse range of customer needs.
The Client’s Requirement : The client wanted technology to be the differentiating factor and build solution for its future growth. It needed innovation as maintenance costs of its legacy systems were increasing exponentially. They wanted to break away from legacy systems in areas like Underwriting, Claims Management, Customer Onboarding. The client faced challenges with the current vendor development team and as a consequence incurred many costs. Though the legacy system could be maintained, they couldn’t innovate.
Business Solution Highlights : RapidValue analyzed organizational DevOps readiness and did a maturity assessment. It provided end-to-end automated solutions for continuous deployment, standardized processes, templates, guidelines and checklists for DevOps. The certified DevOps test architects at RapidValue with deep domain expertise provided seamless Agile delivery and deployment. Applications are built in Agile & DevOps methodology with microservices-based architecture. There was a complex product development by implementing strategies while touching the core of the company. RapidValue built the “Underwriting automation tool” as well as “Insurance as an API tool” for channel Integration and implemented DevOps for enterprise thereby, introducing the client to AWS Cloud and DevOps / Automation testing processes.
Benefits : RapidValue partnered with the client to modernize their legacy systems for boosting efficiency.
CI/CD, when combined effectively within a DevOps setup accelerates delivery of production-ready code resulting into faster time-to-market and reduced costs. Integration of CI and CD result in faster builds and deliver quicker results. With deployments running in continuous cycles, this enables you to track the project and provide feedback in real-time. It also lets you fix the shortcomings with your team whenever necessary.
Hence, the CI/CD methodology facilitates building and improving applications with faster time to market. In addition, having a good automation allows you to gain a more streamlined app development cycle thus, enabling a much quicker feedback cycle and build better, more consistent apps.