Enterprises across industries are adopting Agile and DevOps driven processes and methodologies to accelerate project execution and delivery and are reaping a number of benefits by switching from conventional software development and infrastructure management processes to DevOps. In DevOps, both the development and operations team work together, to develop, test and deploy high-quality software. Software testing has become as integral part of the entire DevOps process and to achieve speed and agility, it is important to automate all the testing processes and configure them to run automatically when the deployment is completed in the QA environment. Specialized automation testing tools and continuous integration tools are used to achieve this integration. This also necessitates the building of a mature automation testing framework through which one can quickly script new test cases.
DevOps is a transformation of culture that includes testing at each and every stage of the Agile project lifecycle. Continuous testing is very important in this transformation because it helps everyone communicate and work together, interact better, and provides a quality product.
Continuous testing is the process of executing the automated scripts once the builds are available in the server to receive immediate test results in the areas where the software changes happen. It means an ongoing testing process done on a continuous basis. In a Continuous DevOps process, change is continuous from Development to Testing to Deployment.
The code is continuously tested, developed, delivered and deployed. Every issue that you have identified during the testing stage will again go through the process of development, integration, testing before ultimately closing the issue. If this process is done manually, it will take a considerable amount of time and testing effort. Therefore, ‘test automation’ is considered to be the best choice for CI/CD.
Deployment Pipeline is an automated system in CI/CD which automates the testing of incremental builds that are available on the server. Here, you are automating the entire process therefore the overall turn-around time is considerably lesser when compared to manual testing. In most scenarios, you cannot automate the testing completely. You require manual testing in some type of scenarios to check whether the test has passed or not. Therefore, identifying the test scenarios that would give you better results while automating is a very essential aspect of CI/CD.
Some of the best advantages of using automation testing in CI/CD pipeline are listed below :
• Faster bug closing (Issue Finding -> Issue Fixing-> Issue Closing).
• Efficient utilization of overall resources in hand i.e. testers, testing infrastructure, etc.
• Ability to execute tests in parallel.
• Consistency in test planning and execution.
• Minimum requirement of technical skills required for automated test-case execution.
In this whitepaper, we will explain how QA automation can be implemented in DevOps and share some best practices and tips based on our experiences on working on various DevOps projects. We will deep dive into some of the key areas in DevOps such as CI/CD Pipeline, Code Management, Team Collaboration and Communication and CI/CD tools. We have also shared real-world example of how implementing QA automation in DevOps helped a financial services company achieve faster time-to-market.
Automating 100% of test cases in QA automation may not be possible. Some scenarios require manual testing effort. Since QA automation is one of the major areas in DevOps, you need to identify the test cases which are going to automate in CI/CD pipeline.
Two categories of test that can be automated are :
• Tests that are executed on a frequent basis : In this case, the testers manually test the scenarios and the testing has to be done several times in a day. The tested application may be prone to errors whereas the productivity is likely to reduce as the testers are executing the test cases multiple times in a day. For e.g., you have a test case from which you need to generate a report in web application and take a screenshot of that report in different browsers and devices. It is a tedious job for testers to generate the report and take the screenshot in different browsers multiple times. In this case, automated cross-browser testing helps the testers for writing effective test cases.
• Tests that require knowledge and have the dependency on specific set of testers : In some cases, the project team will be too much dependent on the testers/developers who have good domain knowledge of the application. If that resource is not available at a crucial time of the project, his absence can influence the overall project deliverables, since only that tester is aware of the prerequisites and procedure for testing. You can avoid these types of situations by integrating QA automation in CI/CD pipeline, so that the project deadlines are not affected negatively.
Make it Simple & Organized by Executing the Smaller Test Cases First
When it comes to test automation, you forget to prioritize the test cases in a systematic approach. It is always better to execute smaller and simple test cases first before going with the complex ones. In this way you can ensure the individual test coverage and performance with respect to functionality testing separately. Execution of complex test cases to check the interaction between the multiple modules will come after that.
Parallel testing is another best practice while implementing QA automation in CI/CD pipeline. After you have identified the tests that need to be automated, next step is to include the concept of ‘Parallel Execution’ in QA automation. Better results will be obtained when you use parallel testing in QA automation as it accelerates the entire process.
You cannot get the best output of parallel testing if the tests are executed on one single machine. The tested machine resources like CPU, GPU etc. can slow down the execution of other tests. The base on which the tests are executed is very important. To test an application, having an in-house infrastructure for test deployment and execution is not a perfect solution.
You can use Lambda test here to perform parallel testing with test automation frameworks such as TestNG, PyTest etc. over an on-cloud online Selenium Grid consisting of more than 2000 browsers and browser versions.
Test strategy document describes a plan on how the automation and other test-related activities would be planned and executed. While updating the test strategy, you have to use the feedback loop for timely updating the code for test automation in CI/CD pipeline.
This feedback can be used to realize the user’s challenges and get better details about the overall usage of the infrastructure that was given for automation testing. For e.g., the QA automation engineers and the DevOps team might find some remarks about the infrastructure over a short period of testing. And these remarks could be related to the approaches used for either capturing logs/carrying automation test code implemented in Python and Selenium to the cloud infrastructure used for testing/performance or related to the execution of tests, etc. The feedback needs to be captured in a document and the important ones should be integrated as a best practice to CI/CD automation pipeline. Therefore, the test code is in-sync with the current requirements.
Developers start their development based on the requirements mentioned in the specifications. Once the development is complete, they will perform a unit test on the code and fix the issues faced during the round of testing. All the issues are independent of integration and the developer should be able to fix all of them as the tests are stand-alone ones. Once the local testing is complete, they have to push the code to a code repository. In most of the development environments, a ‘development branch’ is available and the developer will have to push the code to that ‘development branch’. Once it is approved and tested, then it can be pushed to the ‘production branch’.
By doing this, the developers will be encouraged and motivated to push their codes more frequently to easily keep a track of the changes. This is a very important aspect for CI/ CD and the developers should follow this carefully. Thus, you can achieve micro-agility for activities related to development and test automation in CI/CD pipeline.
In DevOps, there could be number of developers in the project and those developers will push their code as a result of any feature implementation or bug fix. At the same time, developers will execute a pull request to get the latest code changes from the server. Therefore, maintaining the source code on central repository is very important and considered as one of the best practices for CI/CD pipeline. Thus, the developers can maintain their changes and stay up to date with the latest source code which is available in the production server. Also, it is very important to maintain a revision/version control system to track the changes, identify the differences and promote an environment that helps you to maintain a track of the application builds.
In many cases, the project requirements change along with the series of project execution. Similarly, the test automation strategy will also change depending upon the long term and short term projects. As the test plan and strategy will change with time, the team should standardize some of the processes. Standardizing means listing the frameworks, identifying the cloud structure that matches your budget and requirements, preparing an automation team that works on the test scripts etc. Also, you need a backup plan in case any changes occur in the overall business strategy within the organization.
Documentation is very important and helpful in these cases as this document can be referred to at any point of time. Also, you can add a section called ‘Risks & Assumptions’ associated with the test strategy. You can reuse this test strategy document and update the timelines and milestones based on the requirements of the projects. It should be version-controlled so that you can review the document to track the progress.
The strain of moving the code to production environment can be decreased if there is a one-click feature for code migration, from one application environment to another. A well-architected CI/CD pipeline should have one-click migration because it reduces the conflict between different operations. Selecting good cloud infrastructure that provides this feature and effective use of test automation could improve the development and operation process. Committing your development with a single click is great as a best practice for CI/CD.
Documentation for Improving Reusability
It is very helpful to make a note about the QA automation techniques that you have used in the process as a best practice for CI/CD. Here, you can involve your team members i.e., your developers, testers to check and understand where the best practices of test automation can be reused. You can increase your productivity by doing this as your team can recreate the test steps again.
As you know CI & CD are two separate processes, however they cannot be termed as inseparable. If any delay occurs while executing the CI process, it will affect the output of CD process. Even though automation can be used in improving the efficiency of the CI/CD process, there are many areas where the automation may not be effective. Communication and Collaboration are the pillars of CI/CD process. In DevOps, different teams such as Product Planning team, Development team, Verification team, DevOps team are available so that they have to work together for the success of the project. Therefore, benchmarking the CI/CD process with the correct manager who is best in those areas is essential. Also, it is considered as an important best practice for CI/CD pipeline to provide accurate inputs and suggestions for optimization.
In DevOps, no activity can happen in isolation. All activities are dependent here which means there would always be somebody who is dependent on the results of your activity. There should be proper communication between the stakeholders of the project. For e.g., the QA team might be on some test case scenarios and its implementation which would not cover all the requirements of the project. In this case, if they execute the test cases, they will not achieve good test coverage as the important scenarios are getting missed out in the test case/test suite implementation.
In some cases, the developers develop a feature or fix an issue reported by the tester with minimal knowledge of requirement. They may implement the feature without knowing about the actual requirement of the application. It is very difficult for the QA to segregate the issues especially if the issue occurs in the implementation part. The better approach is to break down a feature into small sub features and utilize the concept of unique feature flags in each and every feature. This will help you to separate all the issues and make gradual builds whenever a need arises. This method also helps you to reduce the integration issues when the final feature i.e., the feature which is the combination of all sub features, is pushed into the production branch.
There are some cases where there are interdependencies between the features. In this case, developers have to wait for the completion of the features. You can overcome this situation by the proper use of stubs and feature flags and thus, the developers can avoid the blocking situation. These stubs can be used as dummy in the implementation of the required feature and once the implementation is complete the developer can remove the stubs. This development activity needs careful planning of the proper interaction between the developers to make the QA automation best in CI/CD pipeline. Any kind of indulgence in this process will have an impact on overall test schedule.
Several projects have development and QA teams that are situated in different regions. In this case, proper co-ordination is required to ensure that every team member is in sync. Better transparency facilitates effective processes. This is where the CI can be very powerful as it provides greater level of transparency within the key stakeholders in the team.
With CI, the whole team gets an idea about the overall status of the tests and what all things need to be done to improve the efficiency of the tests. It is necessary to have every team member working on the same page, especially if you are working remotely. Using project management tools could be very effective and it’s a best practice for CI/CD process. By using this project management tool, everyone can view the activities performed by other team members, and also the deadline by which the allocated tasks need to be completed.
It is very important to include the correct members in the QA automation planning, development and execution. The developers in the team can add more information to the test code which is implemented by the test engineers. So, it is better to involve the developers also in the test case implementation. They have a clear picture about the architecture, coding techniques and best practices in software development. At the same time, experienced QA engineers have good understanding of test infrastructure, test frameworks, etc. Therefore, the developers working closely with the QA engineers can reduce the effort and time involved in the test case development to accelerate test automation for CI/CD pipeline.
In some critical situations, the development team can work on the automation code without the participation of the testing team thus, there is no delay in the implementation of automation tests. It is recommended to involve both the developers and QA team for the test case implementation.
Learnings from Previously Executed Projects
Project Planning, requirement gathering, implementation, testing, product deployment, etc. are the different phases of software project. In every phase, you learn new things and can use these findings in your next project for better planning and implementation. Even if the nature of the projects is different, somehow you can still use the best practices to speed up the phases of the current project. In this way, you can also avoid the mistakes that were committed in other projects. It is very helpful to make a note about the QA automation techniques that you have used in the process as a best practice for CI/CD. Here, you can involve your team members i.e., developers, testers to check and understand where the best practices of test automation can be reused. You can increase your productivity by doing this as the team can recreate the test steps again.
All the above-mentioned points would help you to speed up the test automation using best practices for CI/CD pipeline. Speeding up CI/CD pipeline is highly tool dependent. Large number of tools are available for CI/CD. However, you should select the correct tool based on the budget, requirements and experience. Few of the commonly used CI/CD tools are Jenkins, Travis CI, Gitlab, TeamCity, Codeship, Circle CI, etc.
Version control is a feature which is used in DevOps to track the code changes. Regardless of the versioncontrol tool being used, the use of development and testing environments are common for all team members. The development team will create different ‘development branches’ for different customers. However, the code will be pushed to the production server/staging server. It would be better to make the staging environment same as that of the production environment.
This method helps to deploy the working code from staging server to production server. Developers can create and set up new environments with a click here using tools like Jenkins. Use of tools which are common to the development team and DevOps team are very important in streamlining the best practices of CI/CD process.
You can encounter some scenarios where the QA team finds some issues which were not in the previous build release. The reason behind the issue can sometimes be a bug fix that is pushed in the new release software which is undergoing the test. Sometimes, doing RCA (Root Cause Analysis) is very time consuming and you cannot allow missing that valuable feature for a long time in the next release.
In such situations, the developer who pushed that fix is able to roll back his changes so that the release is not stopped. Also, the developer gets some extra time to review and change his implementation. This roll back is not possible without a version control system. This type of roll back option is not only possible in source code, but can be extended to documents, presentations, flow diagrams, etc.
The client had manual deployment processes and they followed Manual Smoke, Sanity and Regression Testing, which was taking longer test execution time and thus, adversely affecting faster release to the market. They were using excel sheets for managing tasks and the work flows which was a time consuming activity. Too much time was spent in preparing the dashboards and charts manually, to track the project progress.
The existing software release process did not have a well-defined branching strategy for the source code management and the releases. The client had a manual deployment process and there was no CI/CD implemented. There was no build back-up and roll back strategy and proper testing/QA environments were not available.
After a detailed analysis and understanding of the customer requirements, we came up with a cost-effective testing solution which included code management, release management, task management, automation, reports etc. We implemented the branching strategy for the source code management. All the environments required for testing were standardized and testing is integrated into the software delivery pipeline.
We utilized the Continuous Integration (CI) and Continuous Delivery (CD) features of DevOps, which reduced deployment time so that both development and QA has more time for testing and quality control issues. Using Continuous Integration (CI), we automated the code integration from multiple developers into a shared repository, multiple times a day. Each of these integrations is then verified by an automated build and the automated tests, which enabled quick error detection and fixes. With Continuous Deployment (CD), the application was made deployable automatically at any point of time to the QA, UAT or production environments once the software passes all automated tests.
By implementing the branching strategy, we ensured that the developers could work in isolation on the new features. This reduced the risk of code overwriting by the developers in the team. Code quality / coding standards were improved by reinforcing the code review process before merging the code into the stable branch from the feature branches. We completely eliminated the confusion of finding which code has been deployed to each of the environments like QA, UAT and Production. As the changes are very small and specific, testability of the build was increased. This enabled more accurate feedback to the Development Team and the quality releases.
Early bug detection and fixing them in the early stages only enabled the developers to spend more time on the feature development than debugging and bug fixing all the time. Automated unit tests and the static analysis enabled the developers to identify and fix the bugs in the early stages of the project. By implementing CI, the Dev Team was able to identify the integration problems and the code merging conflicts early in the stages of development. This made the integrations easy and thus, enabled the rapid software releases. Integration bugs were detected and fixed early. This increased the build quality and helped the customer to save both time and money.
Automated build and deployment eliminated the manual intervention in the deployment process. The speed of the deployments increased with 0% deployment process errors. The automated build pipelines ensured good quality software releases all the time. This has ensured faster time-to-market with quality. QA efforts have been reduced by 50% due to the automated tests implementation and the early defect detection. As a result, QA Team got time to dive deeper into the issues, test more positive and negative scenarios which improved the overall quality of the tests.
Automated build, deployment and testing reduced the resource dependencies and made the new resource on-boarding very easy. Our solution improved the productivity of the team and the team started taking up more features for the development and testing. The Mean Time to Resolution decreased drastically due to the quicker fault isolation.
QA automation is an important part of the overall test strategy. However, it has to be planned and executed in careful manner. You have to identify the different aspects of test automation where you can neatly use resources which are in hand rather than investing in in-house testing infrastructure. Cloud-based testing could affect the overall test automation strategy adversely due to the reliability and scalability aspects. All these improvements not only encourage the members of the team, but also assist in obtaining the best practices for CI/CD pipeline.