AI in Software Testing Interview Questions

Use LLM assistants to draft test ideas, boundary cases, test data, boilerplate and first-pass code, then review everything as if a junior wrote it: check against the requirements, run it, and make sure assertions are meaningful. Never paste secrets or customer data into tools your company hasn't approved.

Generated tests can look correct while asserting little, encode current buggy behaviour as expected, hallucinate APIs, ignore framework conventions, add flaky waits and duplicate coverage, and leak data into external tools. Mitigate with review, mutation-style checks that tests can fail, linting, and clear ownership.

When a locator fails, the tool compares the element's stored attributes (id, text, classes, position, neighbours) with candidates on the page, scores similarity and uses the best match, sometimes updating the locator. Risks: clicking the wrong element, masking regressions and unreviewed drift.

An AI test agent is an LLM in a loop that calls tools, such as a browser exposed through the Model Context Protocol (Playwright MCP). It helps explore apps, reproduce bugs and draft tests, but is still experimental as an unsupervised gate: nondeterministic, costly and needing review.

Test the deterministic parts (retrieval, tools, auth, UI, APIs) conventionally, and evaluate model behaviour statistically on a versioned golden dataset with rubrics, automated graders and human review. Add safety and prompt-injection tests, regression evals on every change, and production monitoring.

Layer methods: deterministic checks for format and required facts, reference-based metrics (exact match, BLEU/ROUGE, embedding similarity), LLM-as-judge with rubrics and controls for its known biases, and human review. Run them on a versioned golden dataset as regression evals on every change.