Design testing for an event-driven microservices system (Kafka, asynchronous flows)

Event-driven systems move the hard problems from request/response into time: messages arrive late, twice or out of order, and failures surface far from their cause. A good strategy tests those properties deliberately rather than hoping end-to-end tests stumble on them.

Example system

Order service publishes OrderPlaced to Kafka. Payment, inventory and notification services consume it and publish their own events (PaymentCaptured, StockReserved). The order service consumes those to move the order to CONFIRMED.

Clarify

• Which broker and delivery guarantees are configured? Kafka consumers are typically at-least-once, so duplicates must be expected.
• How are schemas managed: Avro or Protobuf with a schema registry, or JSON?
• Are there ordering requirements per entity (per order ID)? Is the partition key chosen to provide that?
• How are failures handled: retry topics, dead-letter topics, compensating events (sagas)?

Layer 1: component tests per service

Treat each service as a black box with event inputs and outputs. Run it with a real broker in a container, stub HTTP dependencies, then publish an input event and assert on output events and database state.

@Test
void reservesStockWhenOrderPlaced() {
    String orderId = UUID.randomUUID().toString();
    producer.send(new ProducerRecord<>("orders", orderId, orderPlacedJson(orderId, "SKU-1", 2)));

    await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {
        assertThat(stockRepository.reservedQuantity("SKU-1", orderId)).isEqualTo(2);
        assertThat(testConsumer.eventsOfType("StockReserved"))
            .anyMatch(e -> e.orderId().equals(orderId));
    });
}

Awaitility polls until the assertion passes or the timeout expires, which is faster and more reliable than Thread.sleep. Unique IDs per test let tests share a broker without reading each other’s messages.

Layer 2: message contracts

Most breakages in event systems are schema changes: a renamed field, a changed enum, a removed property that a consumer still reads.

• With a schema registry, enforce a compatibility mode (for example backward compatibility) and check new schemas against the registry in CI before merge.
• Consumer-driven contract testing tools such as Pact support message pacts, where the consumer states which fields it needs and the producer verifies it can produce them.
• Version events explicitly when breaking changes are unavoidable, and test that consumers handle both versions during the transition.

Layer 3: delivery semantics and failure modes

These deserve named tests, because they rarely appear in happy-path flows:

• Duplicates: publish the same event twice; the consumer must be idempotent (one reservation, one email).
• Ordering: OrderCancelled arriving before OrderPlaced is processed. Verify behaviour per the design, and that events for the same order share a partition key.
• Poison messages: malformed payloads go to a dead-letter topic after configured retries without blocking the partition.
• Downstream failure: database down or HTTP dependency failing; the consumer retries with backoff and does not commit the offset prematurely, so no message is lost.
• Restart and rebalance: kill a consumer mid-processing and check no events are lost or double-processed beyond what idempotency handles.
• Sagas and compensation: payment fails after stock was reserved; the reservation is released.

Layer 4: a few end-to-end flows

In a shared or ephemeral environment, run a handful of critical flows: place an order and poll the API until it reaches CONFIRMED within an agreed time. Propagate a correlation ID through every event so that a failure can be traced across services in logs or distributed tracing (for example OpenTelemetry).

Non-functional

• Load tests measure throughput and end-to-end latency (from publish to final state), plus consumer lag under load and recovery time after a backlog.
• Monitor consumer lag, dead-letter topic volume and processing errors in production with alerts.

What interviewers look for

• Recognition that at-least-once delivery means duplicates and idempotent consumers are the norm.
• Schema and contract management as a first-class concern.
• Correct async assertion technique, not sleeps.
• Specific failure-mode tests: ordering, poison messages, rebalances, compensation.
• Observability via correlation IDs and consumer lag.

Common mistakes

• Relying only on end-to-end tests with long fixed waits.
• Testing with an in-memory fake that hides partitioning and rebalancing behaviour.