great place to work

An Essential Guide to Effective Test Automation Leveraging Open Source

An Essential Guide to Effective Test Automation Leveraging Open Source

Introduction - Test Automation

Mankind is stepping into a new era completely owned by automation and the advent of mobility and digital technologies has simplified life by providing easy access to any individual to access technology without any bias. In the business environment, this pushes companies to deliver high-quality applications in less time with limited resources. This has ultimately led to a competition to deliver secure and advanced digital solutions without breaking the requirements and guidelines of the clients. Projects developed under such pressure have failed to meet objectives and suffer from vital schedule and budget slippage because of a delay in discovering the defects. This results in an increased percentage of ‘Defect Leakage’ and eventually customer dissatisfaction due to sub-par products. Organizations have begun to adopt automation to attain better product quality and develop secure, faster and useful mobile applications. This whitepaper endeavors to provide insights from our experiences on working with automation and elaborates on its importance with reference to certain case studies.

The Road that Led to Test Automation

As the saying goes, “Necessity is the mother of invention”, automation too rose from the need to simplify particular processes that were time-consuming. Testing mobile applications are more complex than testing desktop and web applications. This is because, mobile applications need to be tested on a variety of software platforms and versions, on varied hardware and custom factors and also under different network connectivity conditions. In addition to this, the frequent OS updates, the introduction of new devices and the customer expectation of experiencing quick upgrades, would require additional test cycles. This gave rise to test automation where special tools are used to control the execution of tests and actual results are compared with expected results. It can automate certain repetitive yet essential tests in a formalized testing process already in place, or add additional testing that would have been otherwise difficult to perform manually. Organizations have been opting for automation due to the following reasons:

  • Automated testing simplifies low-level interface regression testing which is difficult to perform manually and also aids in finding errors in the testing. This saves a lot of time that is spent on performing repetitive and boring manual testing.
  • Manual testing can be performed only with a limited number of physical devices, automation could employ any number of physical devices as it supports cloud integration.
  • While in case of manual testing, if there is a change in software, one has to run all the tests from the beginning while in case of automated tests, this is not required as the tests are already set up. It also offers a standard and structured framework to incorporate test executions.
An Essential Guide to Effective Test Automation Leveraging Open Source

What Makes Test Automation Important?

During the initial stage of project development, one needs to cover only a limited number of test cases and hence manual testing would be of help in such a scenario. However, as the project progresses, automation would become an inevitable component. As mentioned earlier, test automation saves the time that would have been otherwise used for running regression tests, re-entering the same test data and checking against coding standards. It fuels better consistency in results which could be tracked through the continuous execution of scripts. It also supports static measures and coverage and also helps one in understanding application behavior. The detailed reports generated by it provides statistics and graphs about test progress, incident rates and performance and also allows real-time report collaborations with others in the project. The automated test cases are more reliable and the scripts are scalable anytime in case of new enhancements in the project. When the test cases pass, it builds up the confidence of the developer as it suggests that the new code meets the test requirements and does not interrupt or degrade any existing features.

Open Source Tools and its Impact on Test Automation

Identifying the right tool is a crucial step in guaranteeing success in test automation. Enabling intelligent automation, these tools are pivotal in the DevOps toolchain. Choosing the right tool might be tricky owing to the wide range of choices available in the market. Also, there is no ‘one size fits all’ feature and hence it is important to choose a tool that has the potential to address any testing challenge and can take advantage of the upcoming trends in the digital world. Some of the top open source tools are the following:

1. Selenium – Initially developed in 2004 by Jason Higgins as an internal tool at ThoughtWorks, it was later open sourced in the same year. It is undoubtedly one of the best open source tools that supports quite a lot of programming languages, browsers, and multiple system environments. With the flexibility that it provides to the testers, Selenium acts as the core framework for other open-source automation tools.

2. Appium – The Appium philosophy is that testing native apps shouldn’t require including an SDK or recompiling an app. With its primary focus on mobile applications, it provides full access to back-end APIs and DBs from test code while automating mobile apps from any language and test framework. It is easy to install and user-friendly, making it a much-preferred testing tool.

3. Katalon Studio – Built on top of open-source automation frameworks like Selenium and Appium, it is an automation testing solution developed by Katalon LLC. Advanced testers use it for saving time otherwise used on building new libraries and maintaining their scripts, non-programmers use it to easily start an automation testing project. Katalon Analytics provides comprehensive views of test execution reports that include metrics, charts, and graphs.

4. Watir – An acronym that is derived from, ‘Web Application Testing in Ruby’, Watir is an open-source testing tool that supports cross-browser testing including Firefox, Opera, headless browser and IE. It consists of several smaller projects with the most important ones being watir-classic, watir-webdriver and watirspec. Watir also integrates with BBD tools like RSpec, Cucumber and Test/Unit and also supports data-driven testing.

5. Protractor – It is a NodeJS program written in JavaScript and that identifies the web elements in AngularJS applications which is otherwise not possible by using JUnit or Selenium WebDriver. It combines powerful technologies like Selenium, Jasmine and the like to work as a solution integrator. Protractor is reserved not only for AngularJS applications but also for writing automated regression tests for normal web applications.

The ABC’s of Automation Test Coverage

While analyzing the existing testing tools, we observed that most of the tools provide maximum test execution coverage for applications. However by using an automation tool one cannot automate 100% functionality of an application, especially if the application has any third party integrations. Automation test coverage is calculated using the following formula:

Automation Test Coverage = (Total No. of test cases automated / Total No. of test cases identified for automation) * 100

Here are the percentage of execution that were tracked in Appium and Selenium tools based on our experience with multiple projects.

A coverage of 80% in execution was observed using Selenium.
Total No. of test cases automated = 960
Total No. of test cases identified for automation = 1200
Automation Test Coverage = (960 / 1200) * 100 = 80%

A coverage of 75% in execution was observed using Appium.
Total No. of test cases automated = 600
Total No. of test cases identified for automation = 800
Automation Test Coverage = (600 / 800) * 100 = 75%

Automation test coverage often refers to a set of automation scripts that are written against the requirement specifications. The software quality could be measured through a specific analysis employing automation test coverage. Automation tools easily execute thousands of different complex test cases during every test run providing coverage that is otherwise impossible with manual tests. Testers thus find themselves with more time that could be utilized in creating new automated tests and deal with complex features. Automation test coverage also evaluates the ability of the tool that can automate complex scenarios thus helping us to select a better tool. In some cases, paid tools might have more test coverage on the complex features of application because they provide methods or commands to handle the flows. Whereas in case of trial version tools, we may fail to reach the expected test execution coverage as it does not provide any methods or commands as in the case of paid tools.



Risks Involved in Test Automation

Risks in automation can be divided into three major points:

1. Risk while selecting a tool – As a trial version does not offer full coverage, companies blindly go for paid versions. When some companies purchase paid tools without understanding their maturity level, it could result in a loss in profit. Thus before purchasing and implementing automation tools, one must possess the ability to operate them with appropriate coding knowledge and skills. In fact, the success of the testing process relies on the efficiency with which it is handled. Unrealistic ideas surrounding the tools ultimately leads to failure. Likewise underestimating the time and effort needed to achieve significant and continuing benefits from the tool could also be a risk. With the world moving at a rapid pace, one must anticipate changes and possess a proper estimation to incorporate the changes. Also one must be careful while selecting an automation framework as an unsuitable framework will lead to loss of time and effort. Therefore before selecting a tool, one must have a clear idea about the organization’s maturity, areas within the organization where tool support will help improve testing processes.and the requirements and objectives that a tool can meet.

2. Risk while configuring the tools in different environments – It is possible to face risks while configuring the Appium, Selenium tools in Windows as well as Mac OS. However, configuration issues do not pose a major risk as these can be resolved immediately. Some issues that one could face include incorrect details or missing details in environment variables, device OS version conflicts or Xcode version conflict.

3. Risk while test execution using tools, especially in the case of mobile applications – Here are some of the issues that we have had to face while executing scripts in mobile applications, processes thereby, accelerating continuous delivery.

  • Appium instructor does not start: Faced while executing the Appium tool for iOS or Android mobile applications. This was due to issues related to the WebDriver Agent.
  • Failed to start instruments: Issue faced in Appium tool
  • Failed to start an Appium session: Faced usually while launching an iOS application using Appium tools when instruments crashed on setup.
  • Elements not found: This issue is faced in both Appium and Selenium where the locator fails to find the elements. An element could not be located on the page using the given search parameters. It could have been a locator issue or an incorrectly passed parameter.
  • Issues with configuration files: This might happen in all types of tools.
An Essential Guide to Effective Test Automation Leveraging Open Source

Test Automation Optimization using Automation Framework

A test automation framework is a set of expectations, ideas, and practices which provides you support for automation testers to run the test in a flexible manner and also helps in establishing a work platform for automated testing. Frameworks that are application independent are preferred and testers expect the automation tools to be compatible with the selected automation framework. Thus automation framework is the system in which the tests are automated and the tools follow the standards and strategies that are designed in the framework.

Previously, the environment around the test automation development did not have a structural approach to illustrate function libraries for reporting, error handling and driver scripts. The framework plays an important role in automation test development by providing an outline of the overall test structure, ensuring consistency of testing and minimizing the amount of code for development and thereby enabling less maintenance. Some companies use ‘a single standard’ test automation framework that includes a test library. The test library is created following a standard design and development process with proper documentation, standard scripting, and team consistency. The scripting standard should be maintained across the whole framework behind the library creation and the project team should follow the defined scripting standards. This would help in avoiding duplication of code and in providing consistent coding across the team. In this type of framework scripts are separated from the data stored in different files and modifications are not made in the script. However, the data is manipulated for testing with multiple input values. Implementation and establishment of libraries and utility components across the teams will help in reducing the effort needed for repeated regression suites and thus enabling maximum re-usability.

Some of the automation frameworks that are available in the automation testing world are:

Linear Scripting – Also called as ‘Record and Playback’, model it is a simple framework model. In this model, testers can manually record the actions and scripts are automatically generated when the recording stops. Automation expertise is not required for this framework model.

Test Library Architecture Framework – This framework is also called as ‘Structured Scripting’ model and testers employing this model initially perform record and playback. Later the common tasks are grouped into functions and these functions are called by the main test script named ‘driver’. An advantage of this model is the ease with which the scripts could be managed.

Data-driven Testing framework – Test cases reside in test scripts and test data is kept separately in an external file (Excel files, text files, CSV Files, ODBC Sources). The variables could be loaded into the scripts from external files when required and could be used as input and verification values. The advantages of this framework are that the test scripts do not affect the test data and that the test scenarios can be executed by updating test data in the external data file.

Keyword-Driven or Table-Driven Testing Framework – This framework requires one to develop data tables and keywords. It works independently of a test automation tool used to automate them. The three components of this framework are keyword, application map and component function. Keyword is an action that can be performed on a GUI component. An application map provides named references for GUI components. Component functions actively manipulate or interrogate GUI component. Data table contains the action to be performed on GUI objects and corresponding arguments. Advantages of this framework include the reusability of code and test tool independency.

Hybrid Test Automation Framework – As the name suggests, it is a combination of one or more frameworks. The main intention is to mitigate weaknesses of automation flows and other frameworks by adding more frameworks as adding framework will improve the strength of the framework as well as the automation flows. It is the most often implemented framework by companies.

Companies with an automation framework and optimized automation testing reap the following benefits:

  • Establishing a single standard across the organization helps in following a standard process as compared to pre-empted ad-hoc processes that do not yield results.
  • Independence from individual coding standards and the utilities that have been created. The effort by new members could be reduced with the help of complete documentation.
  • The companies collect complete requirements for automating and providing complete test coverage with the help of the framework. This minimizes the testing efforts during the later stages of the releases.
  • The framework would have the capacity to adopt future enhancements and ensure consistency of testing.

Use Case of Different Open Source Test Automation Tools

Using Selenium to automate – Selenium supports multiple desktop browsers as well as mobile web. The latest version of Selenium, 3.141.59 is observed to work great with the latest version of Firefox, however, will result in random failure if used with Firefox35 or below. An area that is considered to be risky is the configuration of Webdrivers for multiple browsers. However, this could be tackled through proper scripting including a configuration file. Companies employ selenium features such as ‘WebDriverWait’, ‘ExpectedConditions’ and ‘Explicit Wait’ to ensure that the scripts do not break due to fast execution or execution of the next line before finishing the first one. Other important features are Action Classes and JavaScript Executors that support Scroll, Mouse Over, Ajax Handling and Alert Handling activities. Users are supposedly 90% to 95% satisfied with the tool. Selenium provides options to locate UI elements and to compare expected test results against actual application behavior. One of its key features is the support it provides in executing one’s tests on multiple browser platforms. Selenium is composed of different tools with their own roles:

Selenium RC: An older concept that is not widely used in the current automating world, Selenium RC merged with a powerful tool called Selenium Webdriver.

Selenium Webdriver: Most of the testers are using this tool for automating their web applications.

Selenium IDE: A Firefox plugin that provides an easy-to-use interface for developing automated tests. It has a recording feature that records user actions as they are performed and exports them as usable script in one of the many programming languages that could be later executed.

Selenium-Grid: It allows to run tests in parallel, that is, different tests could be run at the same time on different remote machines. (Hub-Node Model)

The benefits of Selenium are that it is open source and supports all browsers like IE, Firefox, Mozilla, and Safari. It also supports all operating systems and programming languages such as Java, Ruby, C#, and Python. It enables one to run multiple tests at a time and runs test in parallel at remote machines using Selenium-Grid.

The streamline working flow of Selenium tool is as follows:

Using Appium to automate – Appium helps companies to establish an automation foundation for both iOS and Android and helps build strong relationships with big clients. It provides features like inspector support that helps in determining the elements that are implemented in the application (say buttons, text fields, labels, etc). It supports the features provided by Selenium and also enables ‘Record and Playback’. It is an open source, cross-platform test automation tool for native and hybrid mobile apps that supports iOS, Android and Firefox OS platforms. It takes selenium commands from the app and converts those into a readable format for UI automation. It does not require recompiling or modifications to app and allows writing tests in one’s favourite programming languages and testing frameworks (Java, Python, Ruby, Node, PHP, C #, and Perl) using the Selenium API. It could run locally or remotely and also provides detailed reports.

The internal streamlined working of Appium tool is as follows:

Appium Instructor generates scripts and these will be sent to the Appium server. The server will generate the corresponding script in UI automation and the UI automation script will be executed in the simulator/devices. About 75% of satisfaction could be achieved through the usage of this tool for automating applications.

Using Protractor to automate – This tool provides automation support for different AngularJS applications. The process of protractor installation is relatively easy. Initially one could face a challenge in finding a good IDE for protractor. Webstorm is a good IDE to configure AngularJS and Protractor concepts. This tool has similar features as Selenium such as element locators and the like. To install Protractor first we need to install NodeJS and configure its path properly. Later we need to open a command prompt and enter the following command to install the protractor globally:

npm install -g protractor

We can install the protractor locally in our project directory. For that we need to type the following command in the command prompt of the project directory:

npm install protractor

All the tests are created in protractor in the form of JavaScript files and run using the following command in the command prompt:

protractor <name of the configuration file>

The important features of the tool are as follows:

  • An end-to-end testing framework for AngularJS applications and works as a solution integrator by combining powerful tools and technologies such as NodeJS, Selenium, WebDriver, Jasmine, and Cucumber.
  • Protractor speeds up testing by avoiding the need for lots of ‘sleeps’ and ‘waits’ in tests as it optimizes sleep and wait times.
  • Built on the top of Webdriver JS and Selenium server, it provides new locator strategies and functions that are helpful to automate the AngularJS application.
  • Introduced new and simple syntax to write tests and allows to run tests targeting remote addresses.
  • It has the ability to take advantage of Selenium-Grid to run multiple browsers at once.

The streamline working flow of Protractor is as follows:

An Essential Guide to Effective Test Automation Leveraging Open Source

Benefits of QA Automation

How profitable is switching to ‘Automation Testing’ is one of the most frequently asked questions. Here are some of the benefits rendered by automation testing that helps in the growth of businesses from all over the world.

1. Cost Optimization – Handling customer issues, escaped bugs and poor performance have been reported to be some of the most time-consuming tasks. Switching to automation reduces the time taken to run tests and fix glitches, thereby reducing the cost incurred. Though the initial investment is high, test automation helps in saving the project and support costs in the longer run.

2. Improved Efficiency – As mentioned earlier, manual testing is a dreary task that proves to be vexing for the testers. The much accurate test automation is a one-stop solution that saves time and also allows the testers to focus on other important tasks. Since testing consumes a major portion of the time of the overall development cycle, reducing the time spent in it would ultimately result in improved efficiency.

3. Reduction of Defects – Unlike manual testing, test automation efficiently detects escaped bugs and run routine checks to improve performance. This fuels the development speed and improves quality while ensuring that there is a reduction in the number of glitches the next time. The early detection of the bugs make them easier to fix and also reduces the money spent on it.

4. Faster Time-to-Market – Organizations that employ manual testing take longer time-to-market as they spend a lot of their time on repetitive testing. This time is saved in test automation where there are certain tools and scripts to ensure that the quality of standard is maintained. With faster release cycles and faster feedback on quality, test automation reduces the time-to-market by a considerable amount.

5. Increased Test Coverage – Another feature that sets apart test automation from manual testing is the fact that the former has the ability to examine an application to ensure that it is working as expected. Also, the facility to execute several complex tests increases the overall test coverage which is otherwise impossible with manual testing.

Conclusion - Test Automation using Open Source

With the reliability and efficiency that is promised through test automation, business value is expected to skyrocket with its implementation. DevOps and Cloud Computing have become familiar names in the IT household as companies compete to outrun each other by adopting the latest technologies. The process of development, operation and quality assurance has undergone a complete makeover with automation and it has been so widely adopted that quality assurance cannot be imagined without test automation. In addition to that, the use of open source testing tools has made software to be accessible to all. They provide the freedom to work without expensive licenses and are regularly modified by testers. Since choosing a tool is a very crucial part to understand requirements, design test cases and much more, it is advised to choose an open source tool that is user-friendly. While some developers are still apprehensive of switching to test automation, the initial costs are quite negligible when compared to the ease and convenience that comes with it.

Sanoj Swaminathan
Test Architect,
RapidValue Solutions

If you’d like to know more about Test Automation Solution and Services, please reach out to us at

How can we help you?
Please share your details to continue to read the whitepaper.