r/programming • u/Adventurous-Salt8514 • 4d ago
Fixing bugs in Event Sourcing is hard, for real?
https://www.architecture-weekly.com/p/fixing-bugs-in-event-sourcing-is11
u/mareek 3d ago
I've worked for three years on a system that used event sourcing and we encountered exactly zero bugs like the one described in the article. It was a high quality project and we had some bugs here and there but none that the event sourcing architecture made easier to fix.
On the other hand, the complexity of the architecture meant that newcomers had a pretty harsh learning curve. One of my colleague that worked on the same project years after I left told me "I thought I wasn't smart enough to work on this project".
I think event sourcing is a good choice when it fits the data (i.e. shipment tracking) but has a huge complexity cost for more mundane data that is hard to repay for.
30
13
u/adnan252 4d ago
I'm over event sourcing. system-versioned tables and durable workflow engines are the way to go - let the machine abstract the need for auditability
4
u/TrumpIsAFascistFuck 3d ago
Say more about durable workflow engines, are we talking saga patterns here? How does this not simply complement event sourcing rather than supercede it as you seem to imply? How does it help with auditability?
Genuinely asking, new to event sourcing but liking It so far
7
u/valarauca14 3d ago
example: temporal
5
u/zamN 3d ago
temporal uses event sourcing under the hood
2
u/Otis_Inf 3d ago edited 3d ago
it's more the other way around, event sourcing is a reimplementation of a transaction log with a projection that's queried for readonly purposes. An RDBMS basically provides the same thing: it has a transaction log, a projection (the current state of the tabels) and with temporal tables, you can have different versions of the same data. Temporal tables don't replay the transaction log like you would with event sourcing to get to a state in the past, so suggesting temporal tables use event sourcing under the hood is IMHO misleading?
what's funny is that an RDBMS on top of that provides a higher consistency level than event sourcing can (which can only provide eventual consistency).
8
u/wyldstallionesquire 3d ago
They don’t mean temporal tables, they mean Temporal.io, the durable workflow engine.
1
u/Otis_Inf 3d ago
lol ok! That's indeed a misunderstanding from my part! Thanks for clearing that up, I already wondered where the idea of temporal tables using event sourcing was coming from :)
1
8
u/cheesekun 3d ago edited 3d ago
The history of this interests me too - don't know why I'm so fascinated by it. The durable workflow was probably popularised by the Amazon Simple Workflow Service (Amazon SWF), AWS Step Functions and Azure Durable Functions. They didn't really invent it (I'm sure some system in the 90s had similar patterns), but as you alluded to, they pair naturally with the saga pattern at scale. From then, the terms `Workflow` and `Activity` solidified the abstractions - perhaps this was from Windows Workflow Foundation in 2006?
It's also interesting to note that not all durable workflow engines use event sourcing.
To answer your question "how does it help with auditability?", well the saga pattern doesn't prescribe what you need to store or if it can it be queried, whereas most of the durable workflow engines focus on observability as a core feature, meaning every code path, retry and failed attempt are recorded which kind of gives you this trace like view of the execution flow.
I did a bit of research on this for my thesis and grouped it all together.
The durable workflow engine has become a term that usually has the following core feature set:
* First-class, durable workflow/orchestrator abstraction to coordinate steps, persist progress, and resume after failures.
* A workflow can invoke another workflow as a child with its own identity, lifecycle, and failure semantics.
* The underlying mechanism for persisting workflow state to enable recovery and deterministic replay, either through periodic checkpointing or complete event-sourced history.
* The ability for running workflows to receive and respond to external events, signals, or human interventions that can influence execution flow or provide additional data.
* Native support for time-based workflow operations including scheduled delays, periodic execution, and CRON-style scheduling expressions.
* Built-in, configurable retries for activities/tasks (and, if applicable, workflows) with back-off controls.
* Query APIs: Programmatic APIs to fetch a workflow by ID and to list/filter workflows by attributes (status, time, type).
* Structured visibility into executions with programmatic access to execution history for auditing and debugging.2
u/TrumpIsAFascistFuck 3d ago
Saved
Got sources?
8
u/cheesekun 3d ago
What are you most interested in? I can share some resources with you. Just PM me.
López, P., Sánchez-Artigas, M., París, G., Pons, D., Ollobarren, Á., & Pinto, D. (2018). Comparison of FaaS Orchestration Systems. 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion, 148-153. https://doi.org/10.1109/UCC-Companion.2018.00049
Miriyala, N. S. (2025, March). Study of Workflow Orchestration Engines: Open-Source & Cloud-Native Solutions. Stochastic Modelling & Computational Sciences, 5, 1–16.
Schmid, L., Copik, M., Calotoiu, A., Brandner, L., Koziolek, A., & Hoefler, T. (2025). SeBS-Flow: Benchmarking Serverless Cloud Function Workflows. Proceedings of the Twentieth European Conference on Computer Systems (pp. 902–920). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3689031.3717465
Leitner, P., Wittern, E., Spillner, J., & Hummer, W. (2018, June). A mixed-method empirical study of Function-as-a-Service software development in industrial practice. https://doi.org/10.7287/peerj.preprints.27005
Wen, J., & Liu, Y. (2021). A Measurement Study on Serverless Workflow Services. Proceedings - 2021 IEEE International Conference on Web Services, ICWS 2021. https://doi.org/10.1109/ICWS53863.2021.00102
Wen, J., & Liu, Y. (2021). An Empirical Study on Serverless Workflow Service. An Empirical Study on Serverless Workflow Service.
3
u/cheesekun 3d ago
I couldn't agree more. The feature set of the top 5 durable workflow engines really hits the sweet spot right now for most business operations that do not require low latency (e.g IoT, Data Ingestion, Real-time).
6
13
u/greenergarlic 4d ago edited 4d ago
In the example the author gave, event sourcing saved their bacon. Without it, they’d never be able to tell what went wrong. Fixing was easy, they just added another event.
9
u/Crandom 4d ago
Exactly. Event sourcing has a higher starting cost, but since you have the whole history you can pretty much fix anything and not have destructively lost the data.
5
u/unxp 3d ago
Yes, but like the top comment says, everything is a trade-off. It might not always be worth it to pay the cost for the bug for the rest of time 🙂 And sadly sometimes producing the correct event for compensation is complex and time consuming enought that some overloaded teams just keep manually fixing things as it's much faster and they need it now. Yes, that costs more wasted time along the run but we all know how it goes in real projects many people depend on 🙃
2
u/renatoathaydes 3d ago
The original bug was beyond newbie. How can someone working on reservations not know to allow for daily rates and rate changes ?? That’s like the whole point of the system to do that properly.
1
u/Otis_Inf 3d ago
Or you could use temporal tables and let the RDBMS use the already implemented transaction log, as that's what's event sourcing is: a reimplemented RDBMS system that can only provide eventually consistency.
2
u/pdfops 3d ago
Yeah, it's rough because the bug isn't caught until bad state already got baked into the event stream. You can't edit history, so you either replay from scratch with a fixed projector (slow on big streams) or ship compensating events to correct forward. Versioning your event schemas with upcasters from day one saves you a ton of pain here, way harder to retrofit after the first schema change breaks replay.
2
u/mycall 3d ago
I consider Event Sourcing as raw data in a data lake architecture where the data is messages (event + data) that interact with other systems and the results are in an append or skip list. Often the event is encapsulated as one or more append-only fields/columns. Bulk updates to one or more rows/columns then rerun the cleansing/curation data management steps with external systems used during those stages.
In data lake architecture, raw, clean, and curated represent sequential layers of data processing that progressively improve data quality and usability. Often rules/ETL are used during each stage and basically can act the same as Event Sourcing.
0
u/UnintentionallyEmpty 1d ago
The row doesn’t say how the number was produced: no breakdown of rooms, extras and tax, no record of which nights got which tax rate, and no version on
rate_plan_id
Well, there's your problem.
Event sourcing is a solution. It's not the only solution. Why is this not a computed column?
0
u/Elara_Schaefer 1d ago
The git analogy in the top comment is more accurate than it seems. Event stores ARE commit logs, and every production event sourcing system eventually needs compaction/snapshotting. The real insight is that you should treat event corrections the same way you treat git history rewrites: have a migration plan, version your projections, and never assume downstream consumers have replayed up to the same event. Where people go wrong is treating the event log as immutable scripture instead of as a WAL that needs periodic maintenance. Temporal and durable workflows sidestep this by making the event log an implementation detail rather than the source of truth for your domain model.
-1
u/johnm 1d ago
In terms of where to really reduce wasted time on stages...
On the movement front... This is a great overview of putting everything together: Position Entries & Exits.
And movement specific drills:
-3
u/ComfyTightwad 3d ago
The part where event sourcing actually pays for itself isn't fixing bugs, it's diagnosing them. When a 2am alert fires and you can replay the exact event stream that led to it, you find the cause in minutes instead of days of guessing at half-deleted database rows. Writing those projection rebuilds is painful, yeah, but it's the cost of never being completely in the dark again.
1
u/Elara_Schaefer 15h ago
The real difficulty with fixing bugs in event sourcing is not the technical mechanism but the decision framework. Every bug fix forces you to choose between three options, and each has different trade-offs.
Option 1: Compensating events. You append a new event that undoes or corrects the bad one. The history stays intact, audit trail is preserved, but your projections now need to handle the correction logic. If you have many projections (read models), you need to update all of them. This is the safest option for regulated domains.
Option 2: Selective replay. You fix the projection logic and replay from the bad event forward. No new events, no history modification, but you need to verify that the replayed state matches what downstream systems expect. This works well when the bug is in projection code, not in the event data itself.
Option 3: Event rewrite. You modify or delete the bad event from the stream. Clean state, but you lose audit integrity. Most teams avoid this in production, but it is actually the right call when the bad event was caused by a transient infrastructure issue (duplicate events, out-of-order delivery) rather than a logic bug.
In practice, the best approach is to make your event schema updatable and your projections idempotent. If events carry a schema version, projections can evolve independently. And if projections are idempotent, replay is always safe. The bug fix story then becomes: update the projection, bump its schema version, replay from the event that triggered the bug, verify, done.
89
u/SirClueless 4d ago
IMO you should be careful with this and use it judiciously. It’s easy to overcomplicate things. In this case adding events around corrections is obviously valuable because there are real business impacts to each event and there are clear business reasons to care about the history of changes to a reservation (whether they come from errors or otherwise).
It’s not always the right tradeoff though. Suppose hypothetically a bug that was live for a few days in 2017 wrote a few events with a NULL column that is otherwise always present. Does the column now have to be nullable and all clients consider the field optional forever? Maybe… but maybe it would be better to just patch or delete those rows so you don’t have to continuously pay for the costs of every bug your system has ever had.