TestNG-failed.xml and Optional Parameters

While debugging failures in some of the tests I came across one issue in TestNG Framework. The issue is related to optional parameters.

Problem Statement

Let me give an example to explain the issue.

Sample Test Class:

public class TestSampleAPI  extends TestBase
{
 @BeforeMethod
 @Parameters("BeforeTestParam")
 public void setUp(String param)
 {
 
 }
 
 @Test
 @Parameters ({"GlobalParam","number1", "number2", "result"})
 public void testAdd(String globalparam,int number1, int number2, @Optional String result)
 {
  Assert.assertEquals(0, 1);  //make sure that this test always fail.
 }
}

Sample XML File corresponding to above Test Class

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" verbose="1" parallel="tests" thread-count="10">
<parameter name ="GlobalParam" value="global" />
<parameter name="BeforeTestParam" value="testvalue" />
 
  <test name="Test1"  preserve-order="true">
        <parameter name="number1"  value="1"/>
 <parameter name="number2"  value="2"/>
 <parameter name="result"  value="3"/>
    <classes>
      <class name="test.TestSampleAPI"/>
      <methods>
        <include name="testAdd"/>
      </methods>
    </classes>
  </test>
 
   <test name="Test2"  preserve-order="true">
        <parameter name="number1"  value="1"/>
 <parameter name="number2"  value="3"/>
    <classes>
      <class name="test.TestSampleAPI"/>
      <methods>
        <include name="testAdd"/>
      </methods>
    </classes>
  </test> 
 
</suite>

When the above xml file is run as TestNG Suite; both the tests fail (for obvious reasons). Following is the testng-failed.xml file that is generated...

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="10" name="Failed suite [Suite]" parallel="tests">
  <parameter name="result" value="3"/>
  <parameter name="number2" value="3"/>
  <parameter name="number1" value="1"/>
  <parameter name="GlobalParam" value="global"/>
  <parameter name="BeforeTestParam" value="testvalue"/>
  <test name="Test1(failed)" parallel="tests" preserve-order="true">
    <parameter name="result" value="3"/>
    <parameter name="number2" value="2"/>
    <parameter name="number1" value="1"/>
    <parameter name="GlobalParam" value="global"/>
    <parameter name="BeforeTestParam" value="testvalue"/>
    <classes>
      <class name="test.TestSampleAPI">
        <methods>
          <include name="setUp"/>
          <include name="setUp"/>
          <include name="testAdd" invocation-numbers="0"/>
        </methods>
      </class>
    </classes>
  </test>
  <test name="Test2(failed)" parallel="tests" preserve-order="true">
    <parameter name="number2" value="3"/>
    <parameter name="number1" value="1"/>
    <parameter name="GlobalParam" value="global"/>
    <parameter name="BeforeTestParam" value="testvalue"/>
    <classes>
      <class name="test.TestSampleAPI">
        <methods>
          <include name="setUp"/>
          <include name="setUp"/>
          <include name="testAdd" invocation-numbers="0"/>
        </methods>
      </class>
    </classes>
  </test>
</suite>

Now look at the above xml file carefully; total 5 global parameters including the one which was optional !!

  <parameter name="result" value="3"/>  <!-- this was declared optional in one of the test Methods !! -->
  <parameter name="number2" value="3"/>
  <parameter name="number1" value="1"/>
  <parameter name="GlobalParam" value="global"/>
  <parameter name="BeforeTestParam" value="testvalue"/>

When this testng-failed.xml will be run as TestNG Suite (as is normally done to rerun failures), probably the tests which are assuming default value of optional parameter (i.e. the ones in which value of default parameter is not explicitly specified) will never pass as those tests will get some junk value from global parameters list.

For example in the above case, on rerun Test2 will take 'result' parameter value as '3' (i.e. <parameter name="result" value="3"/>), which might ultimately lead to failing this test again.

Possible Workaround

The workaround, that I am using currently, is to remove the list of global parameters from testng-failed.xml before running it as TestNG Suite. 

If interested, here is the complete implementation ...

public class FailedTestsXMLModification 
{
 /*
  * There is bug in TestNG which caused tests with optional parameters to fail when executed using
  * testng-failed.xml.
  * 
  * This method is targeted to resolve that bug; 
  */
 
 public static void main(String[] args)
 {
  if(args.length!=1)
  {
   System.out.println("Wrong number of arguments provided!!");
   System.out.println("Please provide path of tesng-failed.xml file.");   
   System.exit(1);
  }
 
  String path= args[0];  
 
     try
     {
      //reading testng-failed.xml file
      File file = new File(path);
      int ch;
      StringBuffer strContent = new StringBuffer("");
      FileInputStream fin = new FileInputStream(file);   
   while ((ch = fin.read()) != -1)
    strContent.append((char) ch);
   fin.close();  
     String content=strContent.toString();
 
     //use regular expression 
        content=content.replaceAll("(<suite.*>)\\s+(<parameter.*/>\\s+)*", "$1\r\n");
 
       //writing back in the testng-failed.xml file
        FileWriter fstream = new FileWriter(path);
        BufferedWriter out = new BufferedWriter(fstream);
        out.write(content);
        out.close();
     } 
     catch (FileNotFoundException e) 
     {
   e.printStackTrace();
  }
     catch (IOException e) 
     {
   e.printStackTrace();
  }
 }

This solution is working fine for TestNG 6.3.1 because in this version, a failed test carry list of all parameters (global and local both) that will be required for its execution (for all- @BeforeTest, @BeforeMethod,actual test, @AfterMethod, @AfterTest etc). The sample testng-failed.xml file I pasted above was generated by TestNG 6.3.1 only.

But this solution will not work with TestNG 6.7 as in this version, a failed test carry only list of local parameters but not of global parameters.