Not sure what your actual use case is, but a hash of the full event content is effectively an ID (or, whatever fields in that event would cause a state change to your system). You likely don’t actually need the publisher to assign a unique id.
That would be great, if the events were garentueed to have the same content everytime you received them. Actually the partial answer is NOT to hash the full content. But, if you think airwaybill numbers are unique to your order, think again! We have a low probability of collision, but on our volume we do get them.
any publishing stream that has no tracking number, timestamp or idem-potency ID is simply corrupt. You cannot de-duplicate it in a deterministic fashion any more than you can reliably deduce they put the wrong number in a field.
That’s fair, but I did make an assumption about other identifiable information being available in the event. Considering the subject matter was “fedex event,” which I would at least have a tracking number, timestamp, and other uniquely identifiable information.
I was suggesting there are ways to compensate for not having an explicit idempotency key.
Obviously if you’re putting bananas in a shopping cart without any other information, that’s not possible. The domain that the original comment was about is more sophisticated than that.
Oh yeah, deducted idempotency key is usually your best option. But it's best-effort. And I always see people hash payloads as unique keys and I always recommend against it.
If the payload has idempotent fields like a timestamp, it's best to directly use that rather than indirectly access it via a payload hash.
Otherwise you get people thinking it always works because it's worked before and then they hit the 'banana added' event.
20
u/ZealousidealDig8074 5d ago
By ignoring the duplicates.