One of the advantages of executing tests in parallel is it speeds up the execution time of Tests which in turn facilitates speeding up the Continuous Integration Process.
Parallelism in Continuous Testing cloud
Continuous Testing cloud and hosted services supports parallelism of Appium or Selenium tests by the implementation of SeeTestGrid the grid is an extension of the current Selenium Grid model, and it supports queuing tests and allocating agents based of desired capabilities to a test Session. seetest.io has the capability to run concurrent tests for one or more users in your project.
Grid Execution provides more details on this concept.
Parallelism in Automated Tests
Just having the support of parallelism in the Continuous Testing is not enough. In order to execute tests in parallel for a specific client session,
- Multi-Threading support of the language in which tests are developed can be used.
- Parallel execution feature of Test Framework using which tests are developed can be made use of and is the preferred solution.
We also need to ensure that to develop tests which can be executed simultaneously, we need to develop tests adhering to some best practices.
- Independent Tests Methods: Tests should be independent of each other.
- Usage of thread safe references: Tests should use thread safe variables. For example, avoid the usage of static references in the tests.
- Repeatable: Tests should return always the same results for the same version of the application and test inputs.
As an example, let's discuss the support of parallel execution of tests in TestNG framework. Concepts discussed in the section below will kick-start the process of building a reliable Parallel Tests.
Parallelism using TestNG
The quick and easy way :
Now that you are all set, with basics of Parallel test execution. Let's try out a sample to demonstrate these concepts.
- Find the sample test in our git repository.
- Fork the repository.
- Follow the steps in Readme packaged in the sample project and run the test using command gradlew runTestsParallel.
Before we discuss parallel execution, here are some basics of TestNG.
TestNG provides a parallel attribute in testng.xml. Basically, this is a way to instruct TestNG's engine to run tests in multiple threads.
The behavior of the thread execution depends upon what value this attribute is assigned in testng.xml.
|Value||Definition example in testng.xml||Description|
|methods||Run all your test methods in separate threads.|
|tests||Run all the methods in the same <test> tag in the same thread|
|classes||Run all the methods in the same class in the same thread, but each class will be run in a separate thread.|
|instances||Run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads|
Here is a simple example of testng.xml with parallel attribute.
Although this looks simple yet the onus is on Test Developers to develop robust and reliable parallel execution test mechanism. Let's take an example to elaborate on this.
Consider the following scenario,
The parallel attribute is defined with a value of methods. This means that every test method defined in TestNG class will be executed in a separate thread.
Appium or Selenium Driver is setup using @Beforeclass Annotation. This means that the Driver is set up on once for a TestNG class.
The test developed using the strategy above will not give reliable results.
The reason being that Appium Driver created in @BeforeClass annotated function is shared in all the test methods. However, every test method is run in separate thread simultaneously. Sharing the same Driver instance will make it thread-unsafe resulting in unreliable results.
So in such cases, it is best to use a parallel attribute with "classes" as value, this will ensure that all test methods are run in same thread within a TestNG class.
If you choose to parallelism at methods level, Web Driver needs to be set up in every test methods to make it thread-safe. Of course, this becomes very resource intensive since every test method will create a driver instance.