Throughput

Throughput is how much work a system gets done per unit of time: requests per second (RPS) for an API, transactions per second (TPS) for a payment system, messages per second for a queue consumer. It measures capacity, while latency measures how long each individual piece of work takes. You need both, plus the error rate, to describe performance.

How throughput, latency and concurrency relate

For a steady system, Little’s Law links the three:

concurrent requests in flight = throughput × average response time

Example: 200 requests/second × 0.25 seconds = 50 requests in flight

With virtual users that pause between actions, think time adds to the cycle: throughput ≈ users ÷ (response time + think time). So 100 users with a 0.5 s response time and 4.5 s of think time generate about 20 requests per second, not 100. This is why “we tested with 1,000 users” means little until you also state the think time and the resulting request rate.

What throughput looks like under rising load

As load increases, throughput first rises in step with it. At some point a resource saturates (CPU, a connection pool, a single-threaded component), and throughput flattens while latency climbs steeply. Push further and throughput can actually fall, because the system spends its effort on queuing, retries and timeouts. Finding that plateau is the main output of a stress test, and it defines practical capacity.

Testing tips

• State targets as rate plus latency. “500 RPS with p95 under 300 ms and under 1% errors” is testable. “Handles high traffic” is not.
• Count successful work only. A server that answers 2,000 RPS with half of them 500 errors is not doing 2,000 RPS of useful work.
• Use arrival-rate load where possible. Generating a fixed request rate shows directly whether the system can sustain it; a fixed pool of users will quietly send fewer requests as responses slow down.
• Check the load generator. If the machine running the test is at 100% CPU, you measured the generator’s throughput, not the system’s.
• Mind rate limits. A gateway returning 429 caps measured throughput; decide whether you are testing the limiter or the service behind it.