Handling Dynamic Pagination in Web Tables using Selenium with Java

 In web automation, working with tables is a common task. Web tables often have paginated structures, especially when dealing with large datasets. The challenge comes when we need to interact with tables that span across multiple pages. In this blog post, we’ll explore how to handle dynamic pagination in web tables using Selenium with Java.

What is Dynamic Pagination?

Dynamic pagination is a technique used to divide large data sets into pages, with users able to navigate between pages using "Previous" and "Next" buttons, or selecting a page number directly. Handling this efficiently is crucial when testing applications, as data may not all be visible at once.

The Challenge

When working with dynamic pagination, Selenium must handle:

  1. Navigating through pagesClicking “Next” or “Previous” until the required data is found.
  2. Handling dynamic changes: The number of pages may vary based on different conditions like filtering or sorting.
  3. Interacting with table rowsExtracting data or interacting with elements across different pages.

Approach

Let’s walk through a sample approach to handling dynamic pagination.

Step-by-Step Guide


1. Set Up Selenium WebDriver

Start by setting up your Selenium environment with Java. You'll need to have:

  • Java Development Kit (JDK)
  • Selenium WebDriver libraries
  • TestNG (optional for structured testing)
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import java.util.List;

public class PaginationHandling {
    public static void main(String[] args) {
        // Set up WebDriver (Assuming Chrome in this case)
        System.setProperty("webdriver.chrome.driver", "path-to-your-chromedriver");
        WebDriver driver = new ChromeDriver();
        driver.get("url-of-your-web-table");

        // Implement logic here
    }
}


2. Locating Table Rows and Pagination Controls

The first task is to locate the web table and identify the rows. Here’s how you can locate the table body and rows:

// Locate the table body

WebElement tableBody = driver.findElement(By.xpath("//table/tbody"));


// Fetch all rows within the table

List<WebElement> rows = tableBody.findElements(By.tagName("tr"));

You’ll also need to identify the pagination controls, such as "Next", "Previous", or page numbers:
// Example for locating Next button
WebElement nextButton = driver.findElement(By.xpath("//a[contains(text(),'Next')]"));

// Example for locating page numbers
List<WebElement> pages = driver.findElements(By.xpath("//a[@class='page-link']"));


3. Loop Through Pages and Extract Data

Now, we can create a loop to navigate through all pages and extract the required data.

public static void paginateAndExtract(WebDriver driver) {

    boolean hasNextPage = true;


    while (hasNextPage) {

        // Fetch and process table rows

        WebElement tableBody = driver.findElement(By.xpath("//table/tbody"));

        List<WebElement> rows = tableBody.findElements(By.tagName("tr"));


        for (WebElement row : rows) {

            // Extract data from each row (e.g., columns)

            List<WebElement> columns = row.findElements(By.tagName("td"));

            System.out.println(columns.get(0).getText()); // Example: print the first column

        }


        // Try to find the "Next" button and move to the next page

        try {

            WebElement nextButton = driver.findElement(By.xpath("//a[contains(text(),'Next')]"));

            if (nextButton.isEnabled()) {

                nextButton.click();

                Thread.sleep(2000);  // Wait for page to load

            } else {

                hasNextPage = false;

            }

        } catch (Exception e) {

            hasNextPage = false;  // No more pages

        }

    }

}


4. Handling Edge Cases

  • Last Page: Ensure you check whether the "Next" button is disabled or no longer exists, as this indicates that you’re on the last page.

  • No PaginationIf the table fits on one page, your script should handle this gracefully by checking if pagination exists at all.

  • Data Search: If you’re looking for specific data within the table, you can add an extra check within the loop. For example, if a row contains specific text, you can stop further navigation.

for (WebElement row : rows) {
    List<WebElement> columns = row.findElements(By.tagName("td"));
    if (columns.get(0).getText().equals("Specific Value")) {
        System.out.println("Found the value, stopping search.");
        hasNextPage = false;  // Stop pagination
        break;
    }
}


Example Scenario: Scraping a Web Table

Let’s consider an example of scraping product data from an e-commerce website, where product listings are displayed in a paginated table. You want to scrape product names and prices across all pages.

public static void scrapeProductTable(WebDriver driver) {

    boolean hasNextPage = true;


    while (hasNextPage) {

        // Locate the table rows

        List<WebElement> rows = driver.findElements(By.xpath("//table/tbody/tr"));


        for (WebElement row : rows) {

            List<WebElement> columns = row.findElements(By.tagName("td"));

            String productName = columns.get(1).getText();  // Assuming the 2nd column has product names

            String productPrice = columns.get(2).getText(); // Assuming the 3rd column has prices


            System.out.println("Product: " + productName + ", Price: " + productPrice);

        }


        // Move to the next page

        try {

            WebElement nextButton = driver.findElement(By.xpath("//a[contains(text(),'Next')]"));

            if (nextButton.isEnabled()) {

                nextButton.click();

                Thread.sleep(2000);  // Allow the next page to load

            } else {

                hasNextPage = false;

            }

        } catch (Exception e) {

            hasNextPage = false;  // No more pages

        }

    }

}


Best Practices

  1. WaitsUse implicit/explicit waits where necessary to ensure pages load before interacting with elements.

  2. Pagination Checks: Before attempting pagination, verify if pagination exists, as some tables may not be paginated under certain conditions.

  3. PerformanceMinimize page refreshes by extracting all necessary data before moving to the next page.


Followers