Response times degrade badly under load. How do you find the bottleneck?

Finding a bottleneck is an investigation, not a guess. The approach I follow: verify the test, find the point where behaviour changes, locate the saturated resource, prove it, fix one thing, and measure again.

1. Verify the test is valid

• Is the load generator the bottleneck? Check its CPU, memory and network. In k6, dropped iterations or many VUs blocked waiting are hints; in JMeter, a GUI-mode run or heavy listeners can throttle the generator. Run JMeter 5.6 in non-GUI mode (jmeter -n -t plan.jmx -l results.jtl -e -o report/).
• Is the workload realistic? Unique data (no artificial cache hits or hot rows), think time, correct mix.
• Are errors being counted? Fast errors can make latency look better; timeouts can make it look worse.

2. Find the knee

Plot throughput and latency against load. Typically throughput rises linearly, then flattens while latency climbs sharply. The load level at that knee is where some resource saturates. A step or ramp profile makes it visible. Also note which percentiles move: everything shifting together points to a shared resource such as CPU; p99 exploding while p50 stays flat suggests queuing, lock contention or GC pauses.

3. Locate the saturated resource

For each component in the request path, check utilisation, saturation and errors (Brendan Gregg’s USE method):

• CPU: high utilisation and run-queue length; throttling against container CPU limits in Kubernetes is easy to miss.
• Memory and GC: frequent or long pauses, heap growth, swapping, OOMKilled restarts.
• Pools and queues: database connection pool exhausted (threads waiting for a connection), HTTP client pools to downstream services, web server worker threads, message queue lag. A pool at 100% with waiters is a classic cause of latency cliffs while CPU looks idle.
• Database: slow query logs, missing indexes, lock waits, deadlocks, N+1 query patterns that scale with load.
• Downstream dependencies: third-party APIs, rate limits, slow internal services; a timeout or retry policy may be multiplying load.
• Network and infrastructure: load balancer limits, bandwidth, DNS, TLS handshakes without connection reuse, autoscaling that reacts too slowly.

4. Use tracing and profiling

APM or OpenTelemetry distributed traces show where time is spent inside a slow request: which span grows under load. A profiler (flame graph) on the hot service shows which code paths consume CPU or block. Correlate trace timestamps with the load test timeline.

5. Prove it, fix it, re-test

Form a hypothesis (“the connection pool of 10 is exhausted at 120 req/s”), confirm with metrics, change one variable (pool size, index, cache, query), and re-run the same test. Removing one bottleneck often reveals the next; that is expected. Record each finding with before/after numbers.

Common mistakes

• Reporting “the API is slow at 500 users” without server-side evidence.
• Tuning several settings at once, so you can’t tell which helped.
• Running against an environment with different sizing or data volume and extrapolating linearly.
• Ignoring coordinated omission: closed-model tools that wait for slow responses send fewer requests and under-report latency.

What interviewers look for

• A structured method rather than guessing.
• Validation of the test setup and the load generator.
• Specific resources to check, especially pools, GC and the database.
• Tracing/profiling evidence and one-change-at-a-time verification.