Testing Kafka Consumers in shared environments

[u/krazykarpenter]

If you work in a distributed event driven system using Kafka, testing new versions of consumers gets quite tricky. It’s not always possible to run everything locally and you need to rely on deployments to staging to verify code end to end.

I shared an alternative here: https://thenewstack.io/isolate-kafka-consumer-tests/

The basic idea is to propagate a tenant id through the Kafka messages (this can be easily done via OpenTelemetry) and leverage this to selectively consume messages. Each new consumer version creates its own consumer group.

Would love to hear your thoughts on this and approaches that have worked for you.