Test Reporting

A test suite produces two things: a pass or fail signal, and the information needed to act on it. Most teams get the first and neglect the second. When a nightly run reports “37 failed” and the only clue is Timeout 30000ms exceeded, someone loses a morning re-running tests locally. Good reporting turns each failure into something a developer can act on from the report alone.

Examples use Playwright, whose built-in reporting is strong; the principles apply to any stack.

Who reads reports, and what they need

No single report serves all four. Plan to produce a machine-readable format plus a rich human-readable one, and to summarise for stakeholders separately (the Test Execution Report tool helps with that last step).

Playwright reporters

Reporters are configured in playwright.config.ts, and you can use several at once:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: process.env.CI
    ? [
        ['list'],                                         // live progress in the CI log
        ['html', { open: 'never', outputFolder: 'playwright-report' }],
        ['junit', { outputFile: 'results/junit.xml' }],  // for the CI test tab
      ]
    : [['list'], ['html', { open: 'on-failure' }]],
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  retries: process.env.CI ? 2 : 0,
});

View an HTML report locally with npx playwright show-report. In CI, upload the report folder as a build artifact so anyone can download it.

JUnit XML is the lingua franca of test results. Almost every CI server, test management tool and dashboard can read it, so produce it from every suite regardless of language: Maven Surefire writes it for JUnit and TestNG, Newman has a junit reporter, and Cypress can use a JUnit reporter too.

Evidence for failures

Traces

A Playwright trace records every action, a DOM snapshot before and after each one, network requests, console logs and the source line. Opening it in the Trace Viewer (npx playwright show-trace trace.zip, or from the HTML report) is usually the fastest way to understand a failure you did not see happen.

Traces cost time and disk space, so record them selectively:

• 'on-first-retry': record only when a test is retried. A good CI default: zero cost for passing tests, full evidence for failures.
• 'retain-on-failure': record every test, keep traces only for failures. Useful when retries are disabled.
• 'on': everything. Use while debugging, not routinely.

Screenshots and video

screenshot: 'only-on-failure' captures the final state of failing tests. Video ('retain-on-failure') helps with animations and timing problems but is heavier; many teams find traces make video unnecessary.

Steps and attachments

Structure long tests with test.step so the report shows where in the journey a failure happened, and attach the data you would want when debugging:

import { test, expect } from '@playwright/test';

test('checkout with a saved card', async ({ page, request }) => {
  const order = await test.step('create a cart via API', async () => {
    const res = await request.post('/api/test/carts', { data: { items: [{ sku: 'KB-101', qty: 1 }] } });
    const body = await res.json();
    await test.info().attach('cart-response', {
      body: JSON.stringify(body, null, 2),
      contentType: 'application/json',
    });
    expect(res.status()).toBe(201);
    return body;
  });

  await test.step('pay with saved card', async () => {
    await page.goto(`/checkout/${order.id}`);
    await page.getByRole('button', { name: 'Pay with saved card' }).click();
  });

  await test.step('see confirmation', async () => {
    await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
  });
});

For API tests, attaching the request and response of a failing call is often all the evidence a developer needs. Remember to mask tokens and personal data before attaching.

Annotations and tags

Tags and annotations make reports filterable and explain skipped or known-failing tests:

test('export to CSV', {
  tag: ['@reports', '@slow'],
  annotation: { type: 'issue', description: 'https://tracker.example.com/QA-1432' },
}, async ({ page }) => {
  // ...
});

Run a subset with npx playwright test --grep @reports, and filter the HTML report by tag.

Merging sharded reports

When a suite is split across machines with --shard, each machine has only part of the results. Use the blob reporter on each shard, collect the blobs, then merge them into one report:

# on each shard
npx playwright test --shard=2/4 --reporter=blob

# after all shards finish, with every blob in ./all-blob-reports
npx playwright merge-reports --reporter html ./all-blob-reports

merge-reports also accepts other reporters (for example --reporter junit), or a config file via --config when you want several reporters with options. The CI/CD Integration lesson shows the full GitHub Actions workflow.

Flakiness and trends

With retries enabled, Playwright marks a test that failed and then passed as flaky, distinct from passed and failed. This is valuable information, not noise to hide.

A single report answers “what happened in this run”. Questions like “is this test getting flakier?” or “which tests are slowest?” need history. Options:

• Your CI’s built-in test analytics, fed by JUnit XML.
• A test reporting or test management tool that stores results per run.
• Allure Report (open source), which builds history and trend views from its result files; adapters exist for Playwright, JUnit, TestNG, Cypress and others.
• A small custom pipeline: the json reporter’s output stored per run and queried later.

Useful measures to track per test and per suite: pass, fail and flaky counts, duration, and how long a test has been failing or quarantined. Watch the trend rather than any single number.

Practice exercise

A team’s CI runs 600 Playwright tests in 4 shards. Developers complain that failures are hard to investigate: there is no single report, CI logs only show Error: expect(received).toBe(expected) for API tests, and nobody knows which tests are flaky. Write the reporter and use configuration, describe the CI artifact steps in words, and change this API assertion so its failure message is useful:

expect(res.status()).toBe(200);

Practise test structure with steps and attachments in the Playwright Lab.

// playwright.config.ts (relevant parts)
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: process.env.CI ? [['blob'], ['github'], ['list']] : [['html', { open: 'on-failure' }]],
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
});

expect(res.status(), `GET ${res.url()} returned ${await res.text()}`).toBe(200);

Key takeaways

• Reports serve different audiences: developers need evidence, CI needs machine-readable results, leads need trends, stakeholders need summaries.
• Combine reporters: list for logs, html for people, junit for CI, blob plus merge-reports for shards.
• Record traces on first retry and screenshots on failure; add test.step, attachments and assertion messages so failures explain themselves.
• Treat flaky results as signal, track them over time, and fix or quarantine them with an owner.