This article builds on the article, Parallel Tests Execution and focuses more on the best practices to execute tests in parallel.
In the original article, there are mentions of these practices, we will discuss those with more detail.
The example provided in this article is based on our Sample code. Below are the steps to get the sample code.
Listed below are good practices that will go a long way for a successful implementation of parallel execution of tests.
- Include parallelism at test design/development stage
- Independent Test methods
- Parallelism and Logging
- Avoid usage of static variables in tests
- Parallelism and Test Drivers
Include parallelism at test design/development stage
As discussed in the original article, running parallel tests strategy cannot be employed at the final stage of test development and by just running tests in parallel. Parallelism needs to be thought at an early stage of the design and development of the Test Suite.
Although most of the Test framework and their language provides a way to run a test in parallel, it has observed that this is not enough. Significant changes need to be done in Tests to make this work.
Independent test methods/Tests
Tests should have a specific object and should not depend on another test. This is a very important consideration and every test developer needs to be educated about this.
Let's discuss this with an example.
Two test methods i.e makePaymentsData and eriBankLogin are dependent on each other.
The test makePaymentsData assumes, that eriBankLogin will be called before and hence the state of the user will be logged in. However, when tests are run in parallel or the order of the execution changes then makePaymentTests will fail.
The sample code performs login operation @BeforeMethod, which ensures it will be always called before executing any test method.
To conclude, It is of utmost importance to develop a method which is independent of each other.
Parallelism and Logging
When tests are run in parallel, it's not easy to figure out what exactly is happening at execution time. The tester needs to know the sequence of the test suite and individual tests.
Generally, logs provide this information. Most of the popular languages achieve parallelism using "Threads". Hence it is a matter of utmost importance that Thread Identifier should find a place in the logs.
Log generated when test executed in parallel WITHOUT thread identifier.
Log generated when test executed in parallel WITH thread identifier
As we can see from above, the second snippet of the log is much clearer to a user as it clearly gives an indication that "AndroidTestNGExampleTest" is executed in thread "11".
There is also more clarity to the user how the tests were executed by the run-time engine.
Avoid usage of static variables in tests
Using static variables should be avoided because in a parallel execution environment this will invariably result in conflicts.
Let us consider the following example.
The code snippet is a Factory class that returns a driver which is static i.e one per class.
Assume two different tests try to make use of it.
Let's consider Class A and Class B belong to Test1 and Test2 respectively and both tests are run in parallel.
This would mean test methods, testA and testB both will grab the driver simultaneously and will lead to conflicts.
Many such examples can be given which shows that using static variables is not a good idea for parallel execution of tests and hence should be avoided.
Parallelism and Test drivers
Automation tests for Mobile and Web applications are generally achieved using test drivers provided by libraries like Appium/Selenium.
Test driver for a Mobile application and browser
Parallelism can be achieved at different levels, Let us consider a test suite whose composition looks like the diagram below.
With regards to parallelism, Here are important observations.
- Individual test methods like test-1-A and test-1-B can be run parallel.
- Tests, Test-1 and Test-2 can be run parallel. In this case, individual test methods will be run in the parent thread.
Choosing the first strategy will mean that the test driver needs to be created in the test methods to avoid thread conflicts. This is becoming counterproductive because this will lead to frequent acquisition and release of devices affecting the performance of the test.
It is hence recommended from seetest.io perspective to keep parallel execution in between tests i.e multiple tests running in parallel which means test drivers to be setup per Test.
Test driver for Browser only application
seetest.io cloud also provides browser only instances for testing.
Since browsers are less resource-intensive, it is fine to develop test suites with a view to run individual test methods in parallel.
In such scenarios language concept such as Java's ThreadLocal is a good choice which ensures returned drivers are per thread. Even if testA and testB are executed in parallel they will use a webdriver per thread thus avoiding conflicts.