LLMs in Test Automation

The previous lesson used LLMs at authoring time: a person asks, the model drafts, the person reviews. This lesson is about putting LLM calls inside the automation system, where they run without a person watching every response. That raises the bar: the integration needs the same engineering discipline as any other dependency that can be slow, expensive, wrong or unavailable.

Where an LLM can sit in the pipeline

Author → Run tests → Collect results → Triage failures → Report → Fix
  ▲                                       ▲              ▲
  │ drafting (previous lesson)            │ classify,    │ summarise for
  │                                       │ cluster      │ humans

Useful placements, roughly from most to least established:

1. Summarising failures for humans: turning a stack trace, error message and recent log lines into a short explanation in the CI report.
2. Triage assistance: suggesting whether a failure looks like a product bug, a test bug, an environment problem or flakiness, and grouping similar failures across a run.
3. Explaining diffs: summarising what changed in a large API response or snapshot diff.
4. Natural-language test steps interpreted by a model at run time.
5. Semantic assertions: asking a model whether output “means” the right thing.

The first three are advisory: if the model is wrong, a human reading the report notices and the pipeline result is unaffected. The last two put the model in the execution or verdict path, which is where the risk lives.

Design rules for LLM integrations

1. Keep the deterministic path deterministic

The test’s pass or fail verdict for conventional functionality should not depend on a model call. Add LLM output alongside results (a triage label, a summary), not instead of them.

2. Ask for structured output and validate it

Free text is hard to use programmatically. Ask for JSON with a fixed set of allowed values, then validate. Many providers offer a structured-output or JSON mode; use it when available, but still validate, because your code should not depend on a single provider feature behaving perfectly.

3. Always have a fallback

Models time out, rate-limit and return malformed output. Every call needs a timeout and a deterministic fallback such as "unknown", so a model outage never breaks the pipeline.

4. Control cost and volume

• Only call the model where it adds value: for failed tests, not every test.
• Truncate inputs: the last 50 log lines are usually more useful than 50,000.
• Cache results keyed on a hash of the input; identical failures across retries or shards need one call.
• Set a per-run budget and stop calling the model once it is spent.

5. Protect data

Logs and traces can contain tokens, cookies, emails and customer data. Redact before sending, use an approved provider or a self-hosted model, and know the provider’s retention terms.

6. Measure it like any other component

If the triage label is wrong half the time, it is noise. Build a small labelled set of past failures and measure accuracy whenever you change the prompt or model.

Building a failure-triage helper

The helper below classifies a failed test into one of four categories. It is provider-agnostic: callModel is a stub you connect to whichever model you are permitted to use.

// triage.ts
import { createHash } from 'node:crypto';

export type TriageLabel = 'product-bug' | 'test-bug' | 'environment' | 'flaky' | 'unknown';
const LABELS: TriageLabel[] = ['product-bug', 'test-bug', 'environment', 'flaky'];

export interface FailureInput {
  testTitle: string;
  error: string;
  logTail: string; // last N lines of relevant logs
  passedOnRetry: boolean;
}

export interface Triage {
  label: TriageLabel;
  reason: string;
  source: 'rule' | 'model' | 'fallback';
}

// Stub: connect to your provider's SDK or a local model.
export async function callModel(prompt: string): Promise<string> {
  throw new Error('callModel not implemented');
}

const cache = new Map<string, Triage>();

function redact(text: string): string {
  return text
    .replace(/Bearer\s+[A-Za-z0-9._-]+/g, 'Bearer [REDACTED]')
    .replace(/[\w.+-]+@[\w-]+\.[\w.-]+/g, '[EMAIL]');
}

function withTimeout<T>(p: Promise<T>, ms: number): Promise<T> {
  return Promise.race([
    p,
    new Promise<T>((_, reject) => setTimeout(() => reject(new Error('timeout')), ms)),
  ]);
}

export async function triage(f: FailureInput): Promise<Triage> {
  // Deterministic rules first: cheap, explainable, testable.
  if (f.passedOnRetry) {
    return { label: 'flaky', reason: 'Passed on retry', source: 'rule' };
  }
  if (/ECONNREFUSED|ENOTFOUND|502 Bad Gateway|503 Service Unavailable/.test(f.error)) {
    return { label: 'environment', reason: 'Network or upstream error', source: 'rule' };
  }

  const input = redact(`${f.testTitle}\n${f.error}\n${f.logTail.slice(-4000)}`);
  const key = createHash('sha256').update(input).digest('hex');
  const cached = cache.get(key);
  if (cached) return cached;

  const prompt = `Classify this automated test failure.
Allowed labels: ${LABELS.join(', ')}.
Respond with JSON only: {"label": "<one allowed label>", "reason": "<one sentence>"}

${input}`;

  let result: Triage = { label: 'unknown', reason: 'Model unavailable', source: 'fallback' };
  try {
    const raw = await withTimeout(callModel(prompt), 15_000);
    const parsed = JSON.parse(raw);
    if (LABELS.includes(parsed.label) && typeof parsed.reason === 'string') {
      result = { label: parsed.label, reason: parsed.reason.slice(0, 300), source: 'model' };
    }
  } catch {
    // keep the fallback; never fail the pipeline because triage failed
  }
  cache.set(key, result);
  return result;
}

Points to notice:

• Rules before the model. Retries and network errors are classified deterministically. The model only sees the ambiguous cases.
• Redaction runs before anything leaves the process. The regexes here are a minimal example; real redaction needs patterns for your own secrets and identifiers.
• Validation rejects any label outside the allowed set, so a creative response becomes unknown, not a new category.
• The source field makes it visible in reports whether a label came from a rule, the model or the fallback.

Measuring the helper

Collect 50 to 100 past failures where the team already knows the real cause. Run the helper over them and compute accuracy and a confusion matrix:

// evaluate-triage.ts
import { triage, type FailureInput, type TriageLabel } from './triage';

interface LabelledFailure extends FailureInput { expected: TriageLabel }

export async function evaluate(dataset: LabelledFailure[]) {
  const confusion: Record<string, number> = {};
  let correct = 0;
  for (const item of dataset) {
    const { label } = await triage(item);
    if (label === item.expected) correct++;
    const cell = `${item.expected} -> ${label}`;
    confusion[cell] = (confusion[cell] ?? 0) + 1;
  }
  return { accuracy: correct / dataset.length, confusion };
}

The confusion matrix matters more than the headline accuracy. Calling a product bug “flaky” is far more costly than the reverse, because it teaches people to ignore real failures. Re-run this evaluation whenever you change the prompt, the model or the rules, and keep the dataset in version control. This is a small version of the evaluation discipline covered in Testing LLM Applications.

Self-healing locators

Self-healing tools try to recover when a locator no longer matches: they compare the current page with what was recorded when the locator last worked and pick the most similar element. Healenium, an open-source project for Selenium, works this way: it stores successful locators and, on a failure, searches for the best match and continues the test, recording the healed locator for review. Newer tools apply LLMs to the same problem.

Self-healing reduces breakage from cosmetic changes, but it has a real failure mode: it can heal to the wrong element and turn a genuine regression into a passing test. Keep it auditable:

• Report every heal, with old and new locator, in the test output.
• Treat heals as maintenance tasks: a person reviews and updates the source locator.
• Do not allow healing in tests that verify the very element being changed.
• Prefer resilient locators (roles, labels, test IDs) so healing is rarely needed.

Practice exercise

Your team wants to add a step to CI where a model reads each failed Playwright test’s error and trace summary and posts a comment on the PR saying what probably went wrong. List the design decisions you would make to keep this safe, cheap and trustworthy.

Show a model answer

1. Advisory only: the comment never changes the job result; failures still fail.
2. Scope: only failed tests, de-duplicated by a hash of the error, capped at a maximum number of model calls per run.
3. Input: test title, error message and a truncated log tail; no full traces, screenshots or environment variables.
4. Redaction: strip tokens, cookies, emails and any known secret patterns before sending; use an approved provider.
5. Structured output: JSON with an allowed category and a one-sentence reason, validated, with a fallback of “no suggestion” on timeout or invalid output.
6. Transparency: the comment says it is an automated suggestion and links to the report and trace.
7. Fork safety: do not run the step for PRs from forks, where secrets are unavailable and untrusted content could be crafted to manipulate the prompt.
8. Measurement: a labelled set of past failures to measure accuracy before rollout and after any prompt or model change; drop the feature if it is not helping.

Key takeaways

• Put LLMs where a wrong answer is cheap: summaries, triage suggestions and grouping, alongside deterministic verdicts.
• Ask for structured output, validate it against an allowed set and always have a timeout and fallback.
• Apply deterministic rules first, and control cost with scoping, truncation, caching and budgets.
• Redact sensitive data before it reaches any model.
• Measure the integration on a labelled dataset and watch the costly confusions, not just accuracy.
• Self-healing (for example, Healenium for Selenium) must report every heal for human review.