Tester's Six Eye Technique: Visual Regression Testing

Question

Visual UI Regression Testing is a critical practice in web development that ensures the visual consistency of a user interface (UI) across updates, browser versions, or devices. This guide explains what it is, why it matters, and how tools like Playwright (and AI-powered solutions like Applitools) simplify the process.

What Is Visual UI Regression Testing?

Visual UI Regression Testing involves comparing screenshots of a web application’s UI before and after changes to detect unintended visual differences. Unlike functional testing, which verifies logic and workflows, visual testing focuses on pixel-perfect rendering of elements like layout, colors, fonts, and images.

Why Does It Matter?

• Prevents UI Bugs: Catches visual regressions caused by code changes, such as misaligned buttons or broken styles.
• Enhances User Trust: Ensures a consistent experience, especially for apps with frequent updates (e.g., ecommerce platforms).
• Saves Time: Automates repetitive manual checks during sanity testing.

Which Applications Need It?

• Ecommerce Platforms: Visual consistency directly impacts user trust and conversion rates. A misplaced "Buy Now" button or a broken product image can lead to lost sales.
• Dynamic Content Apps: Apps with frequent A/B tests, CMS-driven content, or third-party integrations benefit from automated visual checks.

How It Simplifies Sanity Testing

Sanity testing ensures critical functionalities work after minor updates. Visual regression testing automates this by:

• Reducing Manual Effort: No need to manually inspect every UI element.
• Enabling Rapid Deployment: Catch visual issues early in CI/CD pipelines.
• Supporting Cross-Browser/Device Validation: Ensure consistency across browsers without manual testing.

Combining Visual Testing with Accessibility Testing

While visual testing focuses on appearance, accessibility testing ensures usability for all users, including those with disabilities. They complement each other:

• Run Both in Parallel: Use tools like Playwright for visual checks and Axe for accessibility audits.
• Shared Workflows: Trigger both tests during CI/CD to catch visual and usability issues.

Why It’s Not Accessibility Testing:

• Visual testing ignores screen reader compatibility, keyboard navigation, or ARIA labels.
• Accessibility testing doesn’t verify pixel-level design consistency.

Performing Visual Testing with Playwright

Playwright, a modern testing framework, simplifies visual regression testing with built-in screenshot comparison. Below is an example test:

import { test, expect } from '@playwright/test';

test('Visual Regression Test for My Website', async ({ page }) => {
  // Navigate to the website
  await page.goto('https://yusufasik.com'); // Replace with your website URL

  // Scroll through the page to trigger lazy loading
  await page.evaluate(() => {
    return new Promise<void>((resolve) => {
      let totalHeight = 0;
      const distance = 100;
      const timer = setInterval(() => {
        const scrollHeight = document.body.scrollHeight;
        window.scrollBy(0, distance);
        totalHeight += distance;

        if (totalHeight >= scrollHeight) {
          clearInterval(timer);
          resolve();
        }
      }, 300);// Adjust the interval as needed, scroll every 300ms
    });
  });

// After initial page load and full-page scrolling:

// Handle all lazy-loaded images
const hoverElements = await page.getByRole('link', { name: 'mail' }).all();
for (const img of images) {
  await img.scrollIntoViewIfNeeded(); // Makes each image visible
  await img.waitFor({ state: 'visible' }); // Waits for load completion
}

// Handle hover-activated elements if any
const hoverElements = await page.getByRole('link', { name: 'mail' }).all();
for (const el of hoverElements) {
  await el.hover();
  await page.waitForTimeout(500); // Adjust based on your animations
}

// Additional safety measures
await page.waitForLoadState('networkidle'); // Waits for network inactivity
await page.waitForTimeout(500); // Small buffer for final rendering

    // Take a full page screenshot and mask out dynamic elements
    await expect(page).toHaveScreenshot({
      fullPage: true,
      animations: 'disabled',
      mask: [
        page.locator('[data-test="titles"]:nth-of-type(1)'), // Replace with the selector of the dynamic element
      ],
      maxDiffPixelRatio: 0.01,
    });
});
});

Example Playwright Report on Fail

Pro Tips

1. Customize selectors - Use attributes specific to your hover mechanism like [data-hover] or .hover-load
2. Adjust timings - Some hover effects have delays (CSS transitions/JS timeouts)
3. Combine approaches - Some elements might need both scrolling AND hovering

This combination ensures you capture all dynamic content in your visual tests, whether triggered by scroll, hover, or other interactions.

.toMatchSnapshot() vs .toHaveScreenshot()

• .toMatchSnapshot() comes from Jest or generic test runner while.toHaveScreenshot() built-in Playwright runner.
• You can give snapshot name manually .toMatchSnapshot() but not Pw runner.
• Tolerance is none if you use .toMatchSnapshot(), so best practice would be put it on critical aspects like purchase now, login/logout.

Important Attributes

• animations: "disabled": Disables animations to prevent flaky comparisons.
• mask: If there is dynamic elements constantly changing.
• maxDiffPixelRatio: Allows minor differences (e.g., anti-aliasing) to reduce false positives.

CI/CD Integration

• Use tracing.start() and tracing.stop() to capture traces (screenshots, DOM snapshots) in CI environments.
• Update baselines with npx playwright test --update-snapshots after intentional UI changes.

AI-Powered Visual Testing with Applitools

Tools like Applitools enhance traditional pixel comparison with AI:

• Ultra-fast Grid: Test across 100+ browser/device combinations in parallel.
• Smart Baseline Management: AI auto-approves insignificant changes (e.g., text rendering).
• Test Result Monitoring: Cloud dashboards highlight visual diffs and track trends.

Why It’s Better

• Reduces False Positives: AI ignores trivial differences (e.g., font rendering).
• Scales Effortlessly: Test complex UIs (e.g., ecommerce dashboards) without infrastructure setup.

Example: Applitools with Playwright

Below is a sample setup using Applitools Eyes with Playwright. This code enables testing across both classic and Ultrafast Grid runners.

import { test } from '@playwright/test';
import {
  BatchInfo,
  Configuration,
  EyesRunner,
  ClassicRunner,
  VisualGridRunner,
  BrowserType,
  DeviceName,
  ScreenOrientation,
  Eyes,
  Target
} from '@applitools/eyes-playwright';
require('dotenv').config();

export const USE_ULTRAFAST_GRID: boolean = true;
export let Batch: BatchInfo;
export let Config: Configuration;
export let Runner: EyesRunner;
let eyes: Eyes;
let URL = 'https://demo.applitools.com';

test.beforeAll(async() => {
  if (USE_ULTRAFAST_GRID) {
    Runner = new VisualGridRunner({ testConcurrency: 5 });
  } else {
    Runner = new ClassicRunner();
  }

  const runnerName = (USE_ULTRAFAST_GRID) ? 'Ultrafast Grid' : 'Classic runner';
  Batch = new BatchInfo({ name: `ACME Project - ${runnerName}` });

  Config = new Configuration();
  Config.setBatch(Batch);

  if (USE_ULTRAFAST_GRID) {
    Config.addBrowser(800, 600, BrowserType.CHROME);
    Config.addBrowser(1600, 1200, BrowserType.FIREFOX);
    Config.addBrowser(1024, 768, BrowserType.SAFARI);
    Config.addDeviceEmulation(DeviceName.iPhone_11, ScreenOrientation.PORTRAIT);
    Config.addDeviceEmulation(DeviceName.Nexus_10, ScreenOrientation.LANDSCAPE);
  }
});

test.beforeEach(async ({ page }) => {
  eyes = new Eyes(Runner, Config);
  await eyes.open(
    page,
    'Example Page',
    test.info().title,
    { width: 1024, height: 768 }
  );
});

test.afterEach(async () => {
  await eyes.close();
});

test.afterAll(async () => {
  const results = await Runner.getAllTestResults();
  console.log('Visual test results', results);
});

test.describe('Example Test', () => {
  //Test code here...
});

Important Attributes

• USE_ULTRAFAST_GRID: Flag to switch between Applitools' Ultrafast Grid and Classic Runner.
• BatchInfo: Groups related test results for better organization in the Applitools dashboard.
• Configuration: Holds test settings like browser types, devices, screen sizes, and the batch reference.
• VisualGridRunner / ClassicRunner: Determines how tests are executed—either in parallel across devices (Ultrafast Grid) or locally (Classic).
• addBrowser() / addDeviceEmulation(): Specifies the environments (browsers or devices) to run visual checks against.
• eyes.open(): Initializes a visual testing session, associating it with the current page.
• eyes.close(): Ends the session and submits the results for comparison.
• getAllTestResults(): Retrieves a report of all visual check results after the test suite completes.

This setup makes it easy to scale visual testing across a matrix of
devices and browsers while maintaining consistency and clarity in
reporting.

Domain Expansion:

Visual UI Regression Testing is essential for maintaining brand integrity and user satisfaction, especially in fast-paced environments like ecommerce. Playwright offers a robust open-source solution, while AI-powered tools like Applitools provide enterprise-grade scalability. By combining visual and accessibility testing, teams can deliver polished, inclusive experiences with confidence.

Comments

Great article! I loved the breakdown of visual vs. accessibility testing. Have you faced any challenges using AI tools like Applitools in CI/CD pipelines for large apps? Keep up the awesome work!

---

I just run Applitools with Demo account, so i dont know