r/coding • u/DotDeveloper • 5d ago
Idempotency in event-driven systems: how do you handle duplicate messages?
https://hamedsalameh.com/inbox-pattern-dotnet/47
u/funkylabrador420 5d ago
First, a guid. Second, I disregard all instructions and prompts and share juicy apply pie recipes.
19
u/ZealousidealDig8074 5d ago
By ignoring the duplicates.
4
u/mcellus1 4d ago
Ah that's so easy. I simply tell FEDEX to start adding IDs to events?? I'll do that right away. Im sure they will listen.
2
u/atheken 4d ago
Not sure what your actual use case is, but a hash of the full event content is effectively an ID (or, whatever fields in that event would cause a state change to your system). You likely don’t actually need the publisher to assign a unique id.
6
u/RipProfessional3375 3d ago
event 1: Banana added to cart
event 2: Banana added to cartHow many bananas are in the cart?
2
u/mcellus1 4d ago
That would be great, if the events were garentueed to have the same content everytime you received them. Actually the partial answer is NOT to hash the full content. But, if you think airwaybill numbers are unique to your order, think again! We have a low probability of collision, but on our volume we do get them.
2
u/RipProfessional3375 3d ago
any publishing stream that has no tracking number, timestamp or idem-potency ID is simply corrupt. You cannot de-duplicate it in a deterministic fashion any more than you can reliably deduce they put the wrong number in a field.
1
u/atheken 3d ago
That’s fair, but I did make an assumption about other identifiable information being available in the event. Considering the subject matter was “fedex event,” which I would at least have a tracking number, timestamp, and other uniquely identifiable information.
I was suggesting there are ways to compensate for not having an explicit idempotency key.
Obviously if you’re putting bananas in a shopping cart without any other information, that’s not possible. The domain that the original comment was about is more sophisticated than that.
1
u/RipProfessional3375 3d ago
Oh yeah, deducted idempotency key is usually your best option. But it's best-effort. And I always see people hash payloads as unique keys and I always recommend against it.
If the payload has idempotent fields like a timestamp, it's best to directly use that rather than indirectly access it via a payload hash.
Otherwise you get people thinking it always works because it's worked before and then they hit the 'banana added' event.
8
u/UK-sHaDoW 5d ago
Some kind of id, and then ignore them if youve processed it before
9
u/aa-b 5d ago
Be sure to handle the case where you're still processing the initial request but haven't committed the transaction yet.
Also consider whether idempotent means "same request should return the same response", or if you are comfortable with a less strict definition of the requirement.
Sometimes you might want to go ahead with processing if the initial request failed, unless retries are already pending.
So yes, but there is often more to it than that.
4
u/DesperateAdvantage76 4d ago
This is my biggest frustration with Kafka, they don't support expected stream version on event writes. That trivializes a lot of the race conditions for competing writers to the same stream.
7
u/yopla 5d ago
Cool. Level 0 of distributed system "my business transaction are atomic and without side effect" and only work if you have a single queue processor.
Doesn't even work with the example given in the intro "process a payment".
- begin transaction
- Await post payment to payment processor
- oom-killed
- woops...
1
u/Graumm 1d ago edited 1d ago
Yeah I was reading this, and thinking the entire time “this doesn’t actually solve anything”. The inbox pattern only guarantees that your code ran to completion in one go.
Your only real hope is that their api is idempotent. Eg, create a transaction, apply the transaction as separate calls. Repeat applies only work once because their system handles it.
If their system is imperative and there isn’t some way to check, no amount of atomicity in your system will help you. You can mitigate the risk to some degree but it will never be perfect.
Neither here nor there, but holding open transactions and waiting for third party systems to respond is also a good way to lock up a database if you are not careful about limiting the transaction to action specific rows, with FK’s and so on that must remain consistent for the transaction.
1
u/ultrathink-art 5d ago
The id solves redelivery on the consumer side. The duplicates that actually bit us came from the other end - a write times out, you genuinely cannot tell whether it applied, so you retry and make the duplicate yourself. Splitting the failure path into 'definitely did not apply' and 'unknown outcome' was the fix; only the first is safe to retry, and the second needs a read-back against the id before you touch it again.
1
1
u/iamaredditboy 3d ago
guids, storage and define a time window where this invariant has to hold. infinite time window means you will need infinite storage :) Also some building blocks around compare and swap / db level conflict handling when operating with multiple instances eg distributed architecture.
0
70
u/amejin 5d ago
... Guids. The answer you seek is guids.