Page tree
Skip to end of metadata
Go to start of metadata

One of the advantages of executing tests in parallel is it speeds up the execution time of Tests which in turn facilitates speeding up Continuous Integration Process.

Parallelism in Experitest cloud

Experitest cloud and hosted services supports parallelism of Appium or Selenium tests by the implementation of SeeTestGrid the grid is an extension of the current Selenium Grid model, and it supports queuing tests and allocating agents based of desired capabilities to a test Session.  seetest.io  has the capability to run concurrent tests for one or more users in your project.

Grid Execution provides more details on this concept. 

Parallelism in Automated Tests

Just having the support of parallelism in the Experitest is not enough. In order to execute tests in parallel for a specific client session,

  1. Multi-Threading support of the language in which tests are developed can be used. 
    or
  2. Parallel execution feature of Test Framework using which tests are developed can be made use of and is the preferred solution.

We also need to ensure that to develop tests which can be executed simultaneously, we need to develop tests adhering to some best practices.

They are,

  • Independent Tests Methods: Tests should be independent of each other.
  • Usage of thread safe references: Tests should use thread safe variables. For example, avoid the usage of static references in the tests.
  • Repeatable: Tests should return always the same results for the same version of the application and test inputs.




As an example, let's discuss the support of parallel execution of tests in TestNG framework. Concepts discussed in the section below will kick-start the process of building a reliable Parallel Tests.

Parallelism using TestNG

The quick and easy way :

Now that you are all set, with basics of  Parallel test execution. Let's try out a sample to demonstrate these concepts. 

  1. Find the sample test in our git repository.
  2. Fork the repository.
  3. Follow the steps in Readme packaged in the sample project and run the test using command gradlew runTestsParallel.


Before you proceed with fetching the sample tests, you will have to do two more things, very simple but important ones:

  1. If you plan to test a native or hybrid app - Upload your app to your project
  2. Fetch your access key




Further Reading 

Before we discuss parallel execution, here are some basics of TestNG.

TestNG provides a parallel attribute in testng.xml. Basically, this is a way to instruct TestNG's engine to run tests in multiple threads. 

The behavior of the thread execution depends upon what value this attribute is assigned in testng.xml.

ValueDefinition example in testng.xmlDescription
methods<suite name="My suite" parallel="methods">Run all your test methods in separate threads.
tests<suite name="My suite" parallel="tests">Run all the methods in the same <test> tag in the same thread
classes<suite name="My suite" parallel="classes">Run all the methods in the same class in the same thread, but each class will be run in a separate thread.
instances<suite name="My suite" parallel="instances">Run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads


Here is a simple example of testng.xml with parallel attribute.

TestNG with parallel attribute
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="TestNGRunParallelClassesExample" parallel="classes" verbose="2">
   <!-- preserve-order=true ensures that the order is  -->
   <test name="TestNGRunParallelClassesExample">
      <classes>
         <class name="io.appium.testng.TestNGRunParallelClassesExample">
      </classes>
   </test>
</suite>


Although this looks simple yet the onus is on Test Developers to develop robust and reliable parallel execution test mechanism. Let's take an example to elaborate on this.

Consider the following scenario,

  1. The parallel attribute is defined with a value of methods. This means that every test method defined in TestNG class will be executed in a separate thread.

    Parallel at mehods level
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
    <suite name="TestNGRunParallelClassesExample" parallel="methods" verbose="2">
       <!-- preserve-order=true ensures that the order is  -->
       <test name="TestNGRunParallelClassesExample">
          <classes>
             <class name="io.appium.testng.TestNGRunParallelClassesExample">
          </classes>
       </test>
    </suite>
  2. Appium or Selenium Driver is setup using @Beforeclass Annotation. This means that the Driver is set up on once for a TestNG class.

    WebDriver Creation in @BeforeClass
    public class TestBase {
    	protected AndroidDriver<AndroidElement> driver = null;
    		@BeforeClass
    		public void setUp() {
    			driver = new AndroidDriver<>(new URL("https://cloud.seetest.io/wd/hub"), dc);
    		}
    		@Test
    		public void test1() {
    			driver.findelement(By.xpath("//*[@id='test1']"));
        	} 
    		@Test
    		public void test2() {
    			driver.findelement(By.xpath("//*[@id='test2']"));
        	} 
    }

The test developed using the strategy above will not give reliable results.

The reason being that Appium Driver created in @BeforeClass annotated function is shared in all the test methods. However, every test method is run in separate thread simultaneously. Sharing the same Driver instance will make it thread-unsafe resulting in unreliable results.

So in such cases, it is best to use a parallel attribute with "classes" as value, this will ensure that all test methods are run in same thread within a TestNG class. 

If you choose to parallelism at methods level, Web Driver needs to be set up in every test methods to make it thread-safe. Of course, this becomes very resource intensive since every test method will create a driver instance.

WebDriver at @Test
public class TestBase {

	private AndroidDriver<AndroidElement> driver = null;
	
		@Test
		public void test1() {
			AndroidDriver<AndroidElement> driver = new AndroidDriver<>(new URL("https://cloud.seetest.io/wd/hub"), new DesiredCapabilities());
			driver.findelement(By.xpath("//*[@id='test1']"));
    	} 

		@Test
		public void test2() {
			AndroidDriver<AndroidElement> driver = new AndroidDriver<>(new URL("https://cloud.seetest.io/wd/hub"), new DesiredCapabilities());
			driver.findelement(By.xpath("//*[@id='test2']"));
    	} 
}




  • No labels