Kubernetes Fundamentals for QA

Kubernetes (often written k8s) is a system for running containers across a cluster of machines. You describe the desired state in YAML (“three copies of the checkout service, reachable on port 8080”) and Kubernetes works continuously to make reality match it: starting containers, restarting crashed ones, spreading them across machines and routing traffic.

You do not need to be a cluster administrator to work as an SDET, but if your product runs on Kubernetes you will regularly need to find out why a test environment is broken, read logs from a service, run a suite inside the cluster, or understand why a deploy did not pick up a change. This lesson covers that working knowledge.

The objects you will meet

Pods

A pod is the smallest unit Kubernetes runs: one or more containers that share a network address and can share storage. Usually one pod runs one main container. Pods are disposable. When a pod dies, Kubernetes does not repair it; a controller creates a new one with a new name and a new IP address. Never hard-code a pod name or IP in a test.

Deployments

A deployment manages a set of identical pods for a long-running service. It specifies the image, the number of replicas and how to roll out changes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: qa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:2.14.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5

When the image tag changes, the deployment performs a rolling update: new pods start, and old ones are removed once the new ones are ready. The readiness probe decides “ready”. If it is missing or too shallow, traffic reaches pods that cannot serve it yet, and your tests see intermittent errors during every deploy.

Services

Pods come and go, so other components need a stable address. A service gives a set of pods (selected by labels) one DNS name and virtual IP and load-balances across them:

apiVersion: v1
kind: Service
metadata:
  name: checkout
  namespace: qa
spec:
  selector:
    app: checkout
  ports:
    - port: 80
      targetPort: 8080

Inside the cluster, other pods reach it as http://checkout from the same namespace or http://checkout.qa.svc.cluster.local from anywhere. From outside the cluster, traffic usually arrives through an Ingress or Gateway resource, or a LoadBalancer service, depending on how the cluster is set up.

Namespaces

A namespace is a logical partition of the cluster. Teams commonly use them to separate environments (qa, staging) or to create short-lived ephemeral environments per pull request (pr-1234). Names only need to be unique within a namespace, and access rules and resource quotas can be applied per namespace. Most kubectl commands take -n <namespace>.

ConfigMaps and Secrets

Configuration is injected through ConfigMaps (plain settings) and Secrets (credentials). When a test environment behaves differently from staging, compare these first. Note that Kubernetes Secrets are only base64-encoded by default, not encrypted; access to read them should be restricted.

kubectl essentials

kubectl is the command-line client. These commands cover most debugging:

kubectl config current-context                # which cluster am I talking to?
kubectl get namespaces
kubectl get pods -n qa                        # status, restarts, age
kubectl get pods -n qa -l app=checkout -o wide
kubectl describe pod checkout-7d9f8c-abcde -n qa   # events: pull errors, probe failures, OOM kills
kubectl logs deployment/checkout -n qa --since=15m
kubectl logs checkout-7d9f8c-abcde -n qa --previous  # logs of the crashed container
kubectl get events -n qa --sort-by=.metadata.creationTimestamp
kubectl rollout status deployment/checkout -n qa
kubectl get deployment checkout -n qa -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl port-forward service/checkout 8080:80 -n qa

How to read the common pod states:

Status	Usual meaning
Pending	Not scheduled yet: insufficient resources, or waiting for a volume
ImagePullBackOff / ErrImagePull	Wrong image name or tag, or no permission to pull it
CrashLoopBackOff	The container starts and exits repeatedly; read logs --previous
Running but not ready	The readiness probe is failing; check describe events
OOMKilled (in describe)	The container exceeded its memory limit

The jsonpath command above answers a question testers ask constantly: “which version is actually deployed?” Check it before filing a bug against a build that was never rolled out.

port-forward tunnels a local port to a service or pod, so you can point a local test run or API client at a service that is not exposed outside the cluster.

Running tests as a Job

A Job runs pods until a task completes successfully, which matches the shape of a test run. Running tests inside the cluster lets them reach internal services by DNS name and keeps traffic off the public internet.

apiVersion: batch/v1
kind: Job
metadata:
  name: e2e-run-1842
  namespace: qa
spec:
  backoffLimit: 0                  # do not retry the whole suite on failure
  activeDeadlineSeconds: 1800      # hard timeout: 30 minutes
  ttlSecondsAfterFinished: 86400   # clean up the Job a day after it finishes
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: e2e
          image: registry.example.com/e2e-runner:1842
          env:
            - name: BASE_URL
              value: http://checkout.qa.svc.cluster.local
            - name: TEST_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: e2e-credentials
                  key: password
          resources:
            requests:
              cpu: "1"
              memory: 2Gi
            limits:
              memory: 4Gi

Key fields:

• restartPolicy: Never and backoffLimit: 0 mean a failed run is reported as failed rather than silently re-run. Retrying the whole suite hides flakiness; let the test runner handle per-test retries if you need them.
• activeDeadlineSeconds is the job-level timeout.
• ttlSecondsAfterFinished stops finished jobs piling up in the namespace.
• Secrets come from a Kubernetes Secret, not from the image or the YAML file.
• Resource requests matter: browsers are memory-hungry, and an under-provisioned pod gets OOM-killed mid-run.

Run it and wait for the outcome from CI:

kubectl apply -f e2e-job.yaml
kubectl wait --for=condition=complete job/e2e-run-1842 -n qa --timeout=30m
kubectl logs job/e2e-run-1842 -n qa

kubectl wait exits with an error if the timeout passes without the condition being met. A failed job never becomes complete, so a robust script also checks for condition=failed (for example, by polling kubectl get job ... -o jsonpath for the status) instead of waiting out the full timeout. Pods are ephemeral, so copy reports out before they disappear: have the runner upload them to object storage or your test management tool, or use kubectl cp while the pod still exists.

A CronJob runs a Job on a schedule using cron syntax, which is a simple way to run a nightly regression or a synthetic check inside the cluster.

Browser grids on Kubernetes

Large UI suites often need many browsers at once. Selenium Grid can run on Kubernetes: a hub (router, distributor and session queue) receives WebDriver sessions and hands them to browser node pods, and the number of nodes can be scaled to match demand.

The Selenium project maintains official Docker images and a Helm chart for this in the SeleniumHQ/docker-selenium repository. Helm is a package manager for Kubernetes: a chart is a parameterised bundle of manifests. Use the chart’s own README and values file for configuration, including its documented autoscaling options, rather than hand-writing grid manifests or copying flags from old blog posts, because options change between chart releases.

From the test side nothing changes: your tests use RemoteWebDriver pointed at the grid’s service URL, as covered in Cloud Testing Platforms.

What testers should ask about a Kubernetes environment

• Is every service’s readiness probe meaningful (does it check dependencies), or does it return 200 as soon as the process starts?
• Are test environments created per PR in their own namespace, and are they torn down afterwards?
• Are resource limits in test similar to production? Performance results from a namespace with tiny limits mean little.
• How do I find out which image version is running in each environment?
• Where do logs go after a pod is replaced? (kubectl logs only shows current and previous containers; long-term logs live in a logging system.)

Practice exercise

After every deploy to the qa namespace, your E2E suite fails for about a minute with 502 and 503 errors, then recovers. Separately, a nightly test Job sometimes ends with no logs and the pod shows OOMKilled. Explain the likely causes, how you would confirm them with kubectl, and the fixes.

Show a model answer

Errors after deploy. Likely causes: the deployment has no readiness probe, or one that passes before the app can serve (for example before migrations or cache warm-up), so the rolling update sends traffic to pods that are not ready and removes old ones too early. Tests may also have started before the rollout finished.

Confirm with kubectl describe deployment checkout -n qa (look for the probe definition), kubectl get events -n qa during a deploy, and kubectl rollout status deployment/checkout -n qa timing compared with test start time.

Fix: add a readiness probe that checks the app’s real dependencies; have the pipeline run kubectl rollout status (or equivalent) before starting tests; consider adjusting rollout settings so old pods stay until new ones are ready.

OOMKilled Job. The browser container exceeded its memory limit; with no logs written to storage, the evidence disappeared with the pod.

Confirm with kubectl describe pod <job-pod> -n qa (last state: OOMKilled) and the limits in the Job spec.

Fix: raise the memory request and limit, reduce workers per pod (more pods, fewer browsers each), and have the runner stream logs and upload reports as it goes rather than only at the end.

Key takeaways

• Pods are disposable; deployments keep the right number running; services give them a stable DNS name; namespaces separate environments.
• Readiness probes decide when pods get traffic, and weak probes cause “flaky” failures during every deploy.
• get, describe, logs --previous, get events, rollout status and port-forward cover most debugging.
• Run suites as Jobs with backoffLimit: 0, a deadline, resource requests and secrets from Secrets.
• For Selenium Grid on Kubernetes, use the official docker-selenium Helm chart and its documentation.
• Always check your kubectl context before running anything that changes state.