Reliability Engineering for QA

Testing before release reduces risk; it cannot remove it. Production has real traffic, real data, real infrastructure failures and combinations no test environment reproduces. Reliability engineering, popularised by Google’s Site Reliability Engineering (SRE) books, gives teams a shared language for how reliable a service needs to be and practices for keeping it there.

For QA engineers and SDETs, this is a natural extension of the job: from “does it work before release?” to “is it working for users now, and how quickly would we know if it stopped?”

SLIs, SLOs and SLAs

Service level indicators (SLIs)

An SLI is a carefully chosen measurement of service behaviour as users experience it. Good SLIs are usually ratios of good events to total events:

• Availability: successful requests divided by valid requests (for example, responses that are not 5xx).
• Latency: requests served faster than a threshold divided by all requests (for example, the share of checkout API calls under 500 ms).
• Correctness: the share of responses with correct results, such as orders priced correctly.
• Freshness: the share of reads returning data updated within a time window.

Measure as close to the user as practical (at the load balancer, or from real user monitoring in the browser) rather than deep inside one server, which can be healthy while users see failures.

Service level objectives (SLOs)

An SLO is a target for an SLI over a time window, for example: “99.9% of checkout requests succeed, measured over a rolling 30 days” or “95% of search requests complete in under 300 ms over 28 days”.

Choose SLOs based on what users need and what the business can sustain, not on what the system happens to achieve today. 100% is the wrong target for almost everything: users cannot tell the difference between very high reliability and perfect reliability because their own networks and devices fail more often, and chasing 100% stops all change.

SLAs

An SLA is a contract with customers that specifies consequences (such as service credits) when a level is missed. SLAs are business and legal documents. Internal SLOs are usually stricter than external SLAs so the team gets a warning before the contract is breached.

Error budgets

The error budget is the complement of the SLO: the amount of unreliability the team is allowed in the window.

For a 99.9% availability SLO over 30 days:

Error budget = 1 - 0.999 = 0.1% of requests (or of time)
Time-based:  30 days x 24 h x 60 min = 43,200 min
             43,200 min x 0.001       = 43.2 minutes of full outage per 30 days
Request-based: with 10,000,000 requests in the window,
             0.001 x 10,000,000       = 10,000 failed requests allowed

The budget turns reliability into a decision tool:

• Budget remaining: the team can ship features, run experiments and take measured risks.
• Budget exhausted or burning fast: prioritise reliability work, slow down risky releases and investigate.

This is an agreement, normally written as an error budget policy, between product and engineering. It replaces arguments about “is this stable enough to release?” with a shared, measured answer.

Burn rate is how fast the budget is being consumed relative to the window. A burn rate of 1 uses exactly the whole budget over the window; a burn rate of 10 would exhaust a 30-day budget in 3 days. Alerting on burn rate (often with a fast window and a slow window together) catches serious problems quickly without paging people for brief blips.

Where QA fits

• Help define SLIs that reflect real user journeys, not just server health.
• Treat SLO breaches and fast budget burn as signals for test strategy: what did our tests miss, and at which level should a test have caught it?
• Use the error budget when discussing release risk: a team with little budget left should weigh a risky change differently.

Synthetic monitoring

Synthetic monitoring runs scripted transactions against production (or a production-like environment) on a schedule, from one or more locations, and alerts when they fail. It detects outages even when there is little real traffic, such as at night, and tests journeys end to end from outside your infrastructure.

Your end-to-end tests are a good starting point:

import { test, expect } from '@playwright/test';

test('synthetic: search and view product', { tag: '@synthetic' }, async ({ page }) => {
  const start = Date.now();

  await page.goto('https://www.example-shop.com/');
  await page.getByRole('searchbox', { name: 'Search products' }).fill('mug');
  await page.getByRole('searchbox', { name: 'Search products' }).press('Enter');
  await expect(page.getByRole('heading', { name: /results for "mug"/i })).toBeVisible();

  await page.getByRole('link', { name: /blue mug/i }).first().click();
  await expect(page.getByRole('button', { name: 'Add to basket' })).toBeEnabled();

  test.info().annotations.push({ type: 'duration_ms', description: String(Date.now() - start) });
});

Principles for synthetic checks:

• Read-only or clearly marked. Prefer journeys that do not change data. Where a check must write (sign in, add to basket, even place an order), use dedicated synthetic accounts flagged so their activity is excluded from analytics, fulfilment and billing.
• Small and stable. A handful of critical journeys, with strong locators and no fixed waits. A flaky synthetic check pages someone at 3 a.m. for nothing; that destroys trust fastest.
• Alert on patterns, not single failures. Require consecutive failures or failures from multiple locations before alerting.
• Record timings so you see slow degradation, not just outright failure.

Schedule them with your monitoring platform, a CI scheduled workflow or a Kubernetes CronJob. The cron expression explainer helps check schedules.

Chaos engineering basics

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions. Netflix’s Chaos Monkey, which terminated instances in production, made the idea famous. The core is a scientific method, not random destruction:

1. Define steady state: measurable normal behaviour, ideally your SLIs (for example, checkout success rate and latency).
2. Form a hypothesis: “If one of the three inventory service pods is killed, checkout success stays above 99.9% and p95 latency stays under 800 ms.”
3. Introduce a realistic fault: kill a pod, add network latency to a dependency, make a downstream service return errors, fill a disk, expire a certificate.
4. Observe the difference between steady state and the experiment.
5. Learn and fix: missing timeouts, retries without backoff, no circuit breaker, alerts that did not fire, runbooks that were wrong.

Safety rules:

• Start small and outside production: in a test or staging environment, one fault at a time.
• Limit the blast radius: one instance, one percentage of traffic, one region, a short duration.
• Have a stop button: the ability to end the experiment immediately and roll back.
• Tell people: on-call engineers and stakeholders should know when experiments run.

Tooling ranges from simple to sophisticated. For testers, a good first step is fault injection at the network level in an integration environment: Toxiproxy, for example, sits between your service and a dependency and can add latency or cut connections on demand. Kubernetes-native tools such as Chaos Mesh and LitmusChaos run experiments on clusters. Use what your platform team supports.

Testing in production with feature flags and canaries

Some things can only be verified with production traffic and data. The goal is to do this deliberately and with limited exposure.

Feature flags

A feature flag (feature toggle) lets you deploy code in a disabled state and enable it at run time, for everyone, for a percentage of users or for specific users such as internal staff and test accounts.

This separates deployment from release:

• Deploy the code dark; enable the flag only for test accounts; run your tests against production with those accounts.
• Gradually increase the percentage while watching SLIs.
• Turn the flag off instantly if something goes wrong, without redeploying.

Testing implications:

• Test both flag states, plus the transition, since users may be mid-session when it flips.
• Flags multiply combinations. Keep the number of long-lived flags small, and remove flags once a feature is fully released; stale flags are technical debt and a source of surprising bugs.
• Make the flag state visible in test reports, so you know which variant a test ran against.

Canary releases

A canary release deploys a new version to a small subset of servers or traffic first, compares it with the stable version, and promotes it only if it behaves at least as well.

            ┌────────────────────┐
  traffic ──┤ 95% → stable v2.13 │ ── compare error rate, latency, business metrics
            │  5% → canary v2.14 │
            └────────────────────┘
       healthy? → 25% → 50% → 100%       unhealthy? → route all traffic back to v2.13

The comparison should use the same SLIs as your SLOs, and ideally automated analysis with clear thresholds. Testers contribute by defining what “healthy” means for the changed feature (for example, payment success rate, not just HTTP 200s) and by running targeted smoke tests against the canary.

Practice exercise

Your team runs an online booking service. Recently, a payment dependency slowed down for two hours at night and nobody noticed until morning, while customers abandoned bookings. Propose SLIs and an SLO for the booking journey, calculate its 30-day error budget in minutes, and describe the synthetic check, chaos experiment and release practice you would add.

Show a model answer

SLIs

• Booking availability: successful booking confirmations divided by booking attempts, excluding user errors such as validation failures.
• Booking latency: the share of booking submissions confirmed within 2 seconds.

SLO: 99.5% of booking attempts succeed and 95% confirm within 2 seconds, over a rolling 30 days.

Error budget (availability): 1 - 0.995 = 0.5%. Time-based, 43,200 minutes x 0.005 = 216 minutes per 30 days. The two-hour incident used more than half the month’s budget, a strong signal for reliability work.

Synthetic check: every 5 minutes from two locations, a synthetic account searches, selects a slot and goes through payment to the final confirmation in test mode (or the furthest step the payment provider allows), recording timings. Alert after failures in consecutive runs or from both locations, and on sustained latency over threshold. This would have caught the slowdown at night.

Chaos experiment (staging): hypothesis “if the payment dependency adds 3 seconds of latency, bookings time out gracefully after 5 seconds, users see a retry message, and an alert fires within 10 minutes”. Inject latency with a network fault tool on one instance for 15 minutes, with a stop button. Expected findings: missing timeouts or alerts.

Release practice: ship payment-related changes behind a feature flag, enable for internal accounts first, then canary at 5% with automated comparison of booking success rate and latency before promotion. Alert on error budget burn rate.

Key takeaways

• SLIs measure what users experience, SLOs set targets over a window, and SLAs are contracts with consequences.
• The error budget (1 minus the SLO) turns reliability into a shared decision tool; burn-rate alerts catch fast consumption.
• Synthetic monitoring runs small, stable, scripted journeys against production on a schedule with dedicated accounts.
• Chaos engineering is hypothesis-driven fault injection with a small blast radius and a stop button, starting outside production.
• Feature flags separate deploy from release; canaries compare new and stable versions on real traffic before full rollout.
• Production practices complement pre-release testing; they do not replace it.