Skip to content
SDETRole
SDETRole

Learning path 07 · Intermediate → Advanced

AI in Software Testing

A practical, hype-free path through AI in quality engineering: using LLMs to assist test work, agentic browser automation and MCP, and how to test and evaluate LLM-powered applications. Each lesson marks what is established practice and what is still experimental.

Start the path

0/11 lessons completed

What you will be able to do

  • Separate established AI testing practices from experimental ones and explain the risks of each
  • Draft test ideas, cases and data with an LLM and review the output systematically
  • Explain how AI agents and Playwright MCP drive a browser through accessibility snapshots
  • Evaluate how self-healing locators work and the safeguards that stop silent false passes
  • Design an evaluation suite for an LLM feature with golden datasets and LLM-as-judge
  • Test LLM-powered features for non-determinism, guardrails, prompt injection and cost

Prerequisites

  • Experience writing automated tests in any framework
  • Basic API testing knowledge for the LLM application lessons
  • No machine learning background required

Who this path is for

QA engineers and SDETs who want to use AI tools responsibly in their work, and testers asked to test features built on large language models.

Syllabus: 11 lessons

  1. 01 Introduction to AI-Powered Testing A hype-free map of AI in quality engineering: what LLMs are good and bad at, established vs emerging practices, and the risks testers must manage.
  2. 02 AI-Assisted Test Generation Use LLMs to draft test ideas, test cases, test data and automation code, with prompts that produce reviewable output and checks that catch bad generations.
  3. 03 LLMs in Test Automation Where LLM calls belong inside a test automation system: failure triage, log summaries and natural-language steps, plus design rules for reliability, cost and privacy.
  4. 04 Agentic AI for Testing How AI agents plan and act through tools to test software, the real risks (non-determinism, cost, hallucinated assertions, credentials) and where agents help today.
  5. 05 MCP for QA Workflows What the Model Context Protocol is, how Playwright MCP gives AI assistants a browser, how to configure it, and how to use MCP servers in QA work safely.
  6. 06 AI-Powered Browser Automation How AI drives browsers: accessibility snapshots vs screenshots, Playwright MCP, agentic loops, natural-language actions, Playwright's agent features, and how to keep results trustworthy.
  7. 07 Self-Healing Locators How self-healing locators work (element fingerprints, similarity scoring, fallbacks), Healenium as an open-source example, the risks of silent heals and safeguards that keep tests honest.
  8. 08 Testing LLM Applications A practical strategy for testing LLM-powered features: deterministic layers, eval suites with golden datasets, sampling for non-determinism, guardrail tests and CI.
  9. 09 LLM Evaluation Design trustworthy LLM evaluations: quality dimensions, datasets, reference metrics, LLM-as-judge and its biases, RAG metrics and comparing versions without fooling yourself.
  10. 10 AI Application Reliability Make LLM-powered features dependable: handling non-determinism, timeouts, retries and fallbacks, output validation and guardrails, cost and latency budgets, monitoring and evals in CI.
  11. 11 Model Testing and Validation Test machine learning models like a QA engineer: data splits and leakage, overfitting, precision, recall, F1 and calibration, drift, fairness checks, robustness and how this differs from LLM evals.

Two sides of AI in testing

This path covers both directions. The first is using AI to test: AI-assisted test generation, LLMs inside test automation, agentic AI for testing, AI-powered browser automation, MCP for QA workflows and self-healing locators.

The second is testing AI: testing LLM applications, LLM evaluation, AI application reliability and model testing and validation. Begin with the introduction to AI-powered testing, which sets out what LLMs are good and bad at.

How the lessons treat hype

Each lesson marks what is established practice and what is still emerging. Generated tests still need review, agents are non-deterministic and cost money per run, and a locator that "heals" silently can turn a real regression into a pass. The lessons show concrete checks for each risk, for example asserting on structured output with a JSON Schema rather than trusting free text, or logging every heal for human review. SDETRole itself does not use AI to check your work: labs are executed or validated by deterministic code.

Mistakes teams make when adopting AI in QA

  • Accepting generated test cases without checking them against requirements, which adds volume but not coverage.
  • Running an LLM-judged evaluation once and treating the score as stable. Sample several runs and compare versions on the same dataset.
  • Giving an agent or MCP server production credentials or broad file-system access.
  • Testing an LLM feature only through the UI, instead of testing deterministic layers such as prompt construction and output parsing directly.

Suggested study plan and interview preparation

Allow two to four weeks. If you mainly need to test an LLM feature at work, read the introduction and then jump to testing LLM applications and LLM evaluation. Read testing LLM applications in practice for a worked strategy, keep the LLM-as-judge and prompt injection glossary entries handy, and finish with the AI in software testing interview questions, which increasingly appear in SDET loops.

Interviewers rarely want tool names. They ask how you would know an AI-generated test is correct, or how you would test a chatbot whose answers change between runs. Strong answers describe deterministic checks first, evaluation datasets second and human review where judgement is needed.

Type to search across SDETRole.

Full-text search by Pagefind · runs in your browser Open search page →