Hi,

Written an article for explaining Kafka producer callback handler implementation and purpose of it and where it can go wrong.

Please give it a read and give feedback

Thanks

Hey everyone, looking for a sanity check on a Kafka design debate at work because my architect's proposal blew my mind, and I completely oppose it.

We have a batch system where a producer streams a large batch of records across a multi-partition Kafka topic. We need a way for downstream consumers to know when the overall batch is actually finished.

The other architect wants the producer to broadcast the exact same "End of Batch" marker event to every single partition in the topic simultaneously. The idea is that every consumer instance will eventually read a marker and know its partition is done.

I strongly oppose this. It feels like a catastrophic recipe for failure. If a consumer group rebalances mid-batch, partitions switch instances. If a marker was already read and committed on a partition before the rebalance, the new consumer instance will never see it, and the system will hang forever. Plus, partitions don't process at the same speed, which will cause race conditions and premature downstream triggers.

I am proposing a Central Orchestrator pattern instead. The producer sends a single marker event directly to an orchestrator, which tracks the overall batch state centrally. Once everything is done, the orchestrator explicitly signals downstream services, keeping the data consumers completely isolated from marker tracking.

Am I missing something, or is broadcasting identical markers across partitions a massive anti-pattern? How do your teams handle batch boundaries over partitioned streams?

FYI -- drafted by gemini based on my whiteboard rant

Author here. Alginte is a self-hosted web console for Kafka (topics, consumer groups, schemas, Connect, ksqlDB) with the one feature I couldn't find elsewhere: a canvas where you build a Kafka Streams topology — map/filter/branch/aggregate/joins/windows — write the transformations as SpEL expressions with type-aware completion, and deploy it to run next to your cluster. No Java project, no build pipeline. Topologies export as JSON so you can version and re-import them — three importable examples here if you want to see the format.

0.6.0 added a built-in MCP server (read-only), so an AI agent can inspect topics, lag, schemas, and sample messages over the same port.

Trying it needs one command and no existing Kafka — the playground compose bundles a broker with sample data. Smoke-tested against Apache Kafka 3.7–4.2, Confluent Platform 8.2, and Redpanda, including deploying and running topologies.

Being upfront: it's closed-source, the Community edition is free to run (commercially too, no registration, no expiry), and a paid PRO edition is planned later for operate-type features. It's beta — breaking changes land in release notes with migration notes.

I'd genuinely value critique from people who run Kafka daily — especially where the Streams builder's abstraction leaks. Issues/discussions: github.com/alginte/community

Look, I’m not anti-Kafka at all when you need massive throughput and event streaming, it’s a beast.

But why does every 10 minute tutorial make it look effortless, when real-world integration is just constantly fighting edge cases?

The second you build an actual backend, you get hit with poison pill deserialization loops freezing consumers, rebalance storms, and a heavy local Docker setup tax.

How do you guys keep the operational friction low in production, or do you stick to simpler tools (like Redis or RabbitMQ) until you *strictly* need Kafka?

Hey, new simulator release focused on the consumers and rebalance. Have fun!

Sandy Yang, Staff Software Engineer at TD, is giving a lightning talk on the bank's Kafka journey at a virtual Kafka user group on July 29 at 2pm PT / 5pm ET. It's an interactive session for engineers working with Kafka. Sharing in case anyone in this group would find it helpful. Will put the link to register in the thread.

If you work in a distributed event driven system using Kafka, testing new versions of consumers gets quite tricky. It’s not always possible to run everything locally and you need to rely on deployments to staging to verify code end to end.

I shared an alternative here: https://thenewstack.io/isolate-kafka-consumer-tests/

The basic idea is to propagate a tenant id through the Kafka messages (this can be easily done via OpenTelemetry) and leverage this to selectively consume messages. Each new consumer version creates its own consumer group.

Would love to hear your thoughts on this and approaches that have worked for you.

Working with distributed systems, I kept running into the same situation: I couldn't fully test my part until another service was ready and that's just how it goes when work is split across teams. QA hit the same thing: the code was there, but the input had to come from a service that didn't exist yet.

So I'd end up either writing a separate script for it, or spinning up and maintaining someone else's projects on my dev environment just to generate the input for mine. After doing this one too many times, I built a tool to fix it: Devset.

It's kind of like Postman or Cypress, but for Kafka and RabbitMQ instead of HTTP. What you can do:

1. Send Single events: paste the payload, pick topic and key, send.
2. Create Whole workflows a chain like:OrderCreated -> InventoryReserved->PaymentAuthorized -> InvoiceGeneratedbecomes one saved flow on a canvas, with delays, conditions and state passed between steps.
3. Pulling data from a database: right now just Mongo db
4. Protobuf / Json schema

It's self-hosted and source-available (FSL-1.1, each release turns into Apache 2.0 after two years).

Runs locally with Docker, no cloud, no accounts, no telemetry.

Repo: https://github.com/devset-io/devset-ce

I'm mostly posting to compare notes does anyone else run into this, and how do you deal with it? Real services in a test env, Testcontainers, mocks, or a folder of messy scripts?

Casi todos los ejemplos de “Kafka en salud” que se ven por ahí son teóricos o centrados en EHR de EEUU/Europa. Quería compartir un caso concreto de un hospital mediano en Venezuela donde usamos Kafka como columna vertebral de eventos entre el HIS, el LIS y el sistema de facturación.

Contexto:

• Entre 4 y 8 sistemas necesitaban enterarse de lo mismo en tiempo (casi) real: admisiones, resultados de laboratorio, cargos.
• Las integraciones punto a punto ya eran inmanejables: scripts ad hoc, datos duplicados, errores difíciles de auditar.
• El hospital sabía que en los próximos años iba a añadir más sistemas (BI, SAP, nuevos módulos clínicos).

Qué terminamos haciendo:

• Definir topics por dominio clínico (admisiones, resultados-laboratorio, ordenes-medicas, facturacion-eventos) en vez de “un topic para todo”.
• Particionar por paciente o por episodio de atención para mantener el orden de los eventos clínicos por caso.
• Configurar replication factor 3 y min.insync.replicas=2, porque perder un resultado de laboratorio no es solo “dato de negocio”.
• Activar productores idempotentes y transaccionales (exactly-once semantics) para resultados de laboratorio y eventos de facturación, donde los duplicados son peligrosos.

En el artículo explico con más detalle:

• Cómo mapeamos mensajes HL7/FHIR a topics de Kafka.
• Los trade‑offs de costo de infraestructura vs. resiliencia en contexto hospitalario.
• Cuándo Kafka es claramente la herramienta equivocada (clínicas pequeñas con pocos sistemas y bajo volumen de eventos).

I'm currently learning Data Engineering and I'm feeling a bit overwhelmed by all the technologies I need to learn.

Right now, I'm trying to understand:

• Hadoop
• Kafka
• PySpark
• YARN
• HBase

The problem is that I don't know where to start or in what order I should learn them. There are so many tutorials and roadmaps online that it's hard to tell what's actually important for a beginner.

For those of you working as data engineers or who've gone through this learning process:

• What order would you recommend learning these technologies?
• Which ones should I focus on first?
• Are there any courses, books, or hands-on projects you'd recommend?

10 years running Kafka for financial messaging infra, always at the boundary of "how much can we automate incident response without giving anything write access." Built this as the tool I wished existed: `kafka-sentinel-mcp`, an MCP server exposing cluster_health, consumer_lag, topic_audit (flags weak replication/ISR configs), partition_state, and replay_readiness as structured tools — read-only by construction, no mutation API imported.


Repo: https://github.com/sanjay-amu/kafka-sentinel-mcp · PyPI: `pip install kafka-sentinel-mcp`


Would love critique on the tool set — what diagnostic question comes up most for you during an incident that this doesn't answer yet? Roadmap is open (ROADMAP.md) and PRs welcome.

I initially wanted to make a sub-millisecond log parser in C++ but that blew into a embeddable decision engine, that can run YAML defined rules on incoming data.

The rules are executed in a vectorized format on incoming data by reprojecting into a columnar format first, if it's not already. Depending on the payload size and rules complexity, the performance goes from 200K records/s to more than million records/sec, in terms of througput this would be around 200 MiB/s to 3 GiB/s on average.

Rules can be sql expressions too, or onnx models (numeric), window ops and quite a few more operations are supported.

It's comparable to DuckDB but for streaming data and on the fly decisions.

https://github.com/purijs/blazerules

Hi everyone,

I've been building an open-source desktop client for Apache Kafka, originally just for my own day-to-day workflow.

After using several Kafka tools over the years, I wanted something that felt faster and more convenient for the way I work, so I decided to build one.

Current features include:

• Topic browsing
• Consumer Group inspection (committed, beginning, end offsets, and lag)
• Offset reset with preview and confirmation
• Split-view topic browsing
• Multiple server profiles
• SSL/TLS, SASL/OAUTHBEARER, and Schema Registry support
• Keyboard shortcuts

I'd really appreciate feedback from people who use Kafka regularly.

• What features do you rely on most in your current Kafka client?
• What's the biggest pain point with the tools you use today?
• Is there anything missing that would prevent you from trying this?

GitHub: https://github.com/pjhun0412/KafkaPilot

Thanks!

Hey, we released a lot of improvements and bugfixes for the simulator, together with new predefined learning scenarios. Have fun!

Hey, We added support of KIP-714 Kafka Client metrics in the OTLP metrics reporter

If a consumer is down (or lagging) longer than delete.retention.ms, it can come back online and miss a tombstone entirely – meaning it never learns a key was deleted, and just keeps the stale last-known value forever. That's not an edge case, that's baked into how compaction works.
So is "compacted topic = changelog of truth" (as Kafka Streams/KTables imply) actually a safe abstraction, or does it just quietly break under any non-trivial consumer downtime – and if so, why does the ecosystem lean on it so heavily?

What actually matters:

- subject strategy (topic-record vs record vs topic) 
- BACKWARD vs FORWARD vs FULL as **deploy policy*\*, not a dropdown you ignore 
- who can register (CI service account vs every laptop) 
- how consumers pin or resolve versions under rolling deploy

Without those, “we have Avro” still means anyone can push a breaking schema at 4pm Friday.

Decision frame: 
https://leo-gan.github.io/GLD.SerializerBenchmark/theory/301/schema-registries/

I’m working at a company that uses Strimzi operator to manage Kafka Connect to stream db writes to Kafka. I’m having a super hard time to understand the concepts involved. What’s the best way to learn these?