In today’s competitive and dynamic digital world, companies want to release their software and applications to end users as fast as possible without compromising on the quality. Over the last couple of years, we have seen the shift from manual testing to automation and how enterprises are increasingly adopting test automation to accelerate product development lifecycle. We have to agree that automation is playing a pivotal role in making the entire test execution faster and reliable. We are witnessing some interesting breakthroughs in the field of automation like Parallel Execution, Natural Language Processing (NLP), Machine Learning, Artificial Intelligence (ML/AI).
Parallel execution is an interesting area which requires comparatively less effort to implement and maintain, as compared to NLP or ML/AI. The concept has been around for quite some time and is pretty simple.
To better understand the concept, imagine we are currently running 10 web-based automation tests sequentially in a single browser, where each test takes exactly 1 minute to execute. Hence the total execution time is 10 minutes in sequential execution mode. With parallel execution implemented, we can run all 10 tests in parallel. Hence the total execution time is 1 minute. In this example, we can see a drastic reduction of 90% in the total execution time. This means that we can identify issues and report status 90% faster in parallel execution mode as compared to the sequential run. Is it that simple and effective? Let’s find out.
The Challenge – How to Reduce the Test Execution Time?
With this key benefit of parallel execution where we push our code and infrastructure to function concurrently, we can set new benchmarks for our automation capability.
So, is it worth to implement parallel execution in all our projects? Well, it depends.
If we are going to build a framework from scratch, then the possibilities for parallel execution support are high since we can design the framework and tests to support parallel execution mode right from the start. This also instills the specific scripting mindset early in the involved people. If we are dealing with an existing sequentially running framework, then we need to analyze the existing framework, test cases to see if it’s a viable candidate.
Recently one of our clients wanted us to help them improve their existing framework to bring down the execution time. The regression suite was taking close to 7 hours to complete and this would hamper the quick release goal since it would take more than a day to execute, triage the regression failures and issues. One among the provided suggestions was to implement parallel execution. During the research phase, there was a question dangling in our minds – Is it worth to implement parallel execution in an existing complex framework ? It took a while to figure out the answer and to decide the road map.
Let’s first go through few basic evaluation criteria and then come up with the feasibility analysis and roadmap.
Evaluation Criteria for Parallel Test Execution
As we discussed previously, we can implement the parallel execution support right from start, if we are newly developing a framework. However, in case of an existing framework, we need to consider the below points and derive the answers post the framework analysis:
- Does the framework architecture support parallelism?
- Does the framework primarily rely on static variables and static classes?
- Does the framework use single instance of browser in helper classes and not support multiple browser instances operations?
- Are the framework configurations hard coded and reusable by concurrent processes?
- Is the framework tightly coupled with a specific test framework library version, which prevents parallel execution upgrades / updates?
- Does the framework have extensive logging capability and debug support?
- Does the framework have the ability to generate well defined consolidated reports or reports per instance?
- Does the test support parallelism?
- Are the tests scripted down as independent atomic cases?
- Can the tests run independently and do not depend on the outcome of any other test cases?
- Does each test have its own test data to work upon?
- Does the test have tight dependency with application features? Like few test scenarios cannot be run simultaneously like application reset and data creation.
- Does each test have its own test setup and teardown?
- Does the test take time for execution – Is it a simple UI validation or complex functional validation?
- Does the infrastructure support parallelism?
- Does the automation infrastructure have good configuration – Memory and processor cores?
- Does the CI/CD support proper cleanup if anything goes wrong in execution and does not block subsequent runs – Kill drivers, instances etc.?
- Do we have any specific type of testing to be targeted?
- Is only a specific set of test case required to be faster – Like Sanity or Regression and is the client satisfied with overnight automation run for full functionality?
- Are there any specific type of testing cases present like browser compatibility testing? If customer needs to verify their application in different browsers or different browser versions, then parallel execution would make sense for faster execution.
Once we have these basic evaluation outcomes, we can decide on which type of parallel execution to implement or adapt to.
Types of Parallel Execution
We know that parallel execution means multiple test cases running in parallel. This could be test cases of same module or different modules.
- Multiple module parallelism.
- Test level / Instance level parallelism within a module.
Let’s take an example to understand these types. Consider two modules for automation – Module A and Module B. Different types of parallelism applicable here are:-
- All test cases of module A will run in one browser sequentially while all the cases of Module B will run in another browser instance sequentially.
- Test cases of module A will run in two browsers. Once module A case executions are completed, the test cases of module B will run in these two browsers.
Out of these two types, module wise execution is slightly better since the test cases of same module will run sequentially in one browser and hence, we can expect less interruption between tests of same module. This also reduces the effort to make test cases independent within the module since we are running sequentially as it is today. If multiple modules can’t be executed at the same time, then we can opt for tests parallelization within a module.
How Parallel Execution Reduces Test Execution Time Drastically
Now that we are familiar with the evaluation criteria and different types of parallelism, let’s discuss about the problem statement discussed earlier. Let’s analyze the evaluation criteria performed for the client’s framework, outcomes and possible alternatives for implementing parallel execution.
- The framework makes use of static variable and static helper classes which make it impossible to parallelize across multiple browser instances.
Solution: Framework level changes are required to support parallelism. Concurrently used static methods and variables needs to be non-static. Helper classes should have proper constructors for browser initialization and objects. Implementation of nested sealed class for Test setup and cleanup so that each test will have its own set of objects. Logic to signal tests to wait if some global operation has to be performed E.g. Database restore.
- Module wise parallelism is not possible because before each module, we have to clear the data base due to rule dependencies and can’t be omitted.
Solution: Since each module requires database restore before execution, two modules cannot run at same time. Otherwise Database restore before running Module B will run when Module A test cases are running. This would result in data loss and test failure. Hence, we must focus on parallelizing the tests within each module.
- Test cases are not purely atomic. Test case depends on previous ones to get the required test data.
Solution: Need to change the logic and make tests independent by making use of class and test setup, tear down annotations.
- Test data is shared across test cases. Multiple test cases use same test data for modification and hence actual test outcome is surely going to differ from expected.
Solution: Create test data which is specific for test cases and only to be used by the specific test.
- Numerous of the test cases are GUI specific and requires only few seconds to complete.
Solution: Since we need separate browser instance for each case, there is a delay of opening the browser, logging in, creating test data and navigation to required page before performing the actual test case execution. We can add a logic in base class to reuse the already opened browser to save some time. But in our case, the delay always outweighs the actual test case execution time. Because of this inevitable delay even though we reuse the browser, we cannot save much time by using parallel execution for these sequential tests. E.g. If we are to check the GUI elements in the landing page or dashboard, sequential execution mode will login and navigate to the required page. Other tests can directly start checking for the expected GUI elements and finish. In case of parallel execution, we know that each test must be independent. Hence each test has to login, navigate to page, verify the element and logoff. In this case, sequential run would be more practical and easier unless we club the validations and reduce the test cases.
From the above evaluations, we can now finalize the approach for parallel test execution:
- Framework changes needed.
- Helper classes and existing classes need to be tweaked to offer independent working objects.
- Test changes required for independency.
- Test data segregation required.
- Omit modules with quick GUI validations – Let it run sequential if the parallel execution overhead is more than the actual time saving.
- Parallelize only the modules which are having tests with longer duration.
Using the above approach and framework changes, we were able to implement parallel execution using NUnit, reducing test executing time by ~3 hours. The existing team could then follow the new pattern. This solution will eventually help the client improve the ROI and save more time, since the regression suite is meant to grow over time.
As we have explored, parallel execution is definitely worth trying out. But before jumping on to a full-fledged implementation, we must evaluate the pros and cons first and make sure that the benefits are worth its technical overhead and complexities. If we are starting a new framework development, then we should definitely keep parallel execution support in mind. For an existing framework, we should compare the level of code modification efforts required against the total time savings and an initial POC with few classes would help arrive at a ballpark estimate.
If implemented correctly, parallel test execution can drastically reduce the execution time and thus facilitate faster releases with good quality (of course, based on the test coverage as well). If parallel execution is incorrectly implemented or implemented in a non-viable scenario, it would not only waste the implementation effort but also will be a nightmare with frequent test case failures and debugging. We should first analyze the current scenario, do research on the existing framework and its parallel execution practicality and measure the benefits. If it’s worth the hassle, we should surely go for it !!
Renjith R. Nair
Test Lead, RapidValue