How to Retry Failed Tests in Selenium with TestNG: Complete Steps to use IRetryAnalyzer

Intermittent test failures are one of the biggest pain points. In this blog post, I’ll explain how to retry failed tests in Selenium with TestNG. This ensures your test suite is more resilient. It also makes your test suite more trustworthy. These failures can occur due to network issues, browser timeouts, or momentary server delays. Such situations don’t reflect actual defects in the application. Such non-deterministic test failures slow down the development process and create noise in the results.

The good news is, TestNG provides a built-in mechanism to handle flaky tests by automatically retrying failed tests. Let’s see how to retry failed tests in selenium with TestNG. You can also refer to the official TestNG documentation here.

Why Retry Failed Tests?

Before diving into the implementation, let’s understand why retrying failed tests is important:

1. Reduced Flakiness: Tests that fail due to temporary issues can be retried, reducing the noise in test results.

2. Increased Stability: Retrying tests can help stabilize your CI/CD pipelines.

3. Improved Confidence: Developers and stakeholders can trust test reports more when transient issues are resolved automatically.

[ca-sidebar id="9836"]

Setting Up Retry Logic in TestNG

TestNG provides a way to implement retry logic by using the IRetryAnalyzer interface. This interface allows us to define the conditions under which failed tests should be rerun.

Step 1: Add TestNG Dependency to Your Project

Before we start with the implementation, ensure that you have TestNG added to your project’s dependency management. For Maven projects, you can include it as follows:

<!--how to retry failed tests in selenium with TestNG.-->
<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.7.0</version>
    <scope>test</scope>
</dependency>

Step 2: Create a Retry Analyzer Class

The retry logic is implemented in a class that implements the IRetryAnalyzer interface. This class will define the maximum number of retries and increment the retry count after each failure.

// Using IRetryAnalyzer to retry failed tests in selenium with TestNG.

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {
    
    private int retryCount = 0;
    private static final int maxRetryCount = 3;  // Define how many times you want to retry
    
    @Override
    public boolean retry(ITestResult result) {
        if (retryCount < maxRetryCount) {
            retryCount++;
            return true;  // Test will be retried
        }
        return false;  // No more retries
    }
}

Here, the retry method returns true if the test needs to be retried, and false when the maximum retry count is reached.

Step 3: Associate Retry Logic with Test Cases

Now that the retry logic is implemented, the next step is to associate it with test cases. This can be done at the test method level using the @Test annotation with the retryAnalyzer parameter.

//retry failed tests in selenium with TestNG

import org.testng.annotations.Test;

public class SampleTest {

    @Test(retryAnalyzer = RetryAnalyzer.class)
    public void testMethod() {
        System.out.println("Executing Test");
        assert false;  // Forcefully fail the test to demonstrate retry
    }
}

In this example, the testMethod will be retried up to three times before it is finally marked as failed.

Step 4: Configure Retry at a Global Level (Optional)

If you have several tests and you want to apply retry logic globally, you can use a listener to automatically apply the retry mechanism to all test methods, rather than annotating each test individually.

//retry failed tests in selenium with TestNG

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

public class RetryListener implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

Step 5: Register the Retry Listener in TestNG XML

To apply the retry listener to all tests, register it in the TestNG configuration file (testng.xml):

<suite name="Test Suite">
    <listeners>
        <listener class-name="RetryListener" />
    </listeners>

    <test name="Sample Test">
        <classes>
            <class name="SampleTest" />
        </classes>
    </test>
</suite>

With this setup, every test case in the suite will automatically retry on failure.

Best Practices for Retrying Failed Tests

While retrying failed tests can reduce flakiness, it’s important to keep some best practices in mind:

1. Limit Retries: Avoid setting retries too high (e.g., retrying more than 3 times) as it can mask genuine failures and increase execution time.

2. Identify Root Cause: Repeated retries should not become an excuse to avoid investigating flaky tests. Make sure to analyze persistent issues.

3. Test-Specific Retries: Use retry selectively on flaky tests instead of applying it globally. This prevents unnecessary retries on tests that rarely fail.

Integrating Retry Logic with CI/CD

Retrying failed tests is especially useful in continuous integration pipelines. When integrated with CI tools like Jenkins, Azure DevOps, or GitLab, this technique ensures that transient issues don’t cause build failures. This way, only genuine failures will cause the pipeline to break, saving time and effort.

You can integrate your Selenium test suite with CI/CD by adding a step in your pipeline configuration (Jenkins pipeline as an example):

pipeline {
    agent any
    stages {
        stage('Run Tests') {
            steps {
                sh 'mvn clean test'
            }
        }
    }
    post {
        always {
            junit '**/target/surefire-reports/*.xml'
        }
    }
}

To Sum Up:

• Use IRetryAnalyzer to retry failed tests.

• Avoid over-reliance on retries and focus on fixing flaky tests.

• Ensure integration with your CI/CD pipelines to prevent transient failures from affecting the build process.

With this strategy, you can increase test suite stability and build confidence in your automated tests.

Retrying failed tests is a crucial technique in modern test automation, especially when working with flaky tests or unreliable environments. By implementing these techniques to retry failed tests in Selenium with TestNG, you can drastically improve the reliability of your test suites and maintain the integrity of your CI/CD pipelines.