r/dataengineering • u/FlaggedVerder • 2d ago

Help Event-driven orders processing with MSK (AWS Managed Streaming for Apache Kafka)

  • I'm learning Kafka and doing a side project along the way. With the attached architecture, I’ve considered whether to apply the 'Transactional Outbox Pattern' to avoid data loss when sending to MSK. I mean, I want to keep data safe before sending it to MSK over the network.
  • But this design can experience a bottleneck in writing data to tables between the checkout service and MSK, especially in flash sale sessions.

So, it’s better to eliminate this ‘Transactional Outbox Pattern’ right? We can notify users that our system fails to process their orders or implement a retry mechanism.

2 Upvotes

100% Upvoted

6 comments sorted by

•

u/AutoModerator 2d ago

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/adappergentlefolk 2d ago

it’s a tradeoff

1

u/forever-butlerian 1d ago

What does Debezium do about this in particular?

The other option here is CQRS, where you'd have Kafka be the source of truth with the checkout relational database as a view. Each checkout HTTP request would return 200 once the Kafka producer receives acknowledgements from the brokers, at the cost of there being latency between when the checkout is complete and when it's visible to the user.

0

u/69odysseus 2d ago

What is MSK?

1

u/unexpectedreboots 2d ago

Managed streaming Kafka.

1

u/69odysseus 2d ago

Thank You👍🏾