This is the first entry in a five-part series about event sourcing:
- Why I Finally Embraced Event Sourcing—And Why You Should Too
- What is event sourcing and why you should care
- Preventing painful coupling
- Event-driven microservice in a monolith
- Get started with event sourcing today
A project I’m working on processes files in multiple phases. To help users track progress, I built a status page that shows details like this:
- html
| File Name |
Pages |
Percent complete |
Last updated |
| 5466-harbor-4542.pdf |
23 |
33% |
two minutes ago |
| 5423-seeds-5675.pdf |
35 |
50% |
five minutes ago |
| 9021-lights-3980.pdf |
19 |
100% |
30 seconds ago |
- alignment
- normal
After using it for a while, the team had a request: they wanted to see how long each file took to process. That seemed like a useful addition, so I set out to add it.
One way to track processing time is simple: create two new columns in the database called `start_time` and `end_time` and populate them as the documents are being processed. Then, subtract `start_time` from `end_time` to get the duration. If `end_time` doesn’t exist, subtract `start_time` from the current time.
That works well—for new files.
But what about files that have already been processed? How do we estimate their duration?
The Common Problem: Data Loss in Traditional Systems
This is a familiar challenge. Over and over in my career, I’ve seen business requirements change. Maybe our understanding of the project improves, or we discover a better way of doing things. But there’s always a frustrating reality: any new behavior we introduce can only apply going forward. The existing data is locked in its current form.
Why? Because traditional applications lose information.
The database holds the latest version of every row, and when updated, the row overwrites the older information with the new.
The Game-Changer: Event Sourcing
My application doesn’t have this problem.
That’s because I built it using event sourcing—a pattern where instead of just storing the latest state of the system, we store every change as a sequence of events.
With event sourcing, I had data going back to day one. That meant I could calculate the duration for every file submitted to the system.
In just a few minutes, I adjusted some code, ran some tests, and confirmed that I could retroactively compute durations for all past files, even ones that had failed partway through processing.
Then came my favorite moment.
Since my status page updates live via HTMX over a server-sent-events connection, I watched in real-time as durations magically appeared next to every file. The data had always been there, I just added a new way to present it.
And for the first time in my career, I didn’t have to say, “We can do this going forward, but…”
Why I Wish I Had Used Event Sourcing Sooner
I first learned about event sourcing over a decade ago. The concept fascinated me, but I was hesitant to use it in a production system.
Then, after getting laid off last year, I finally had time to experiment with it on side projects. That gave me the confidence to introduce it to a project at work.
And I wish I had done it years ago.
What Is Event Sourcing?
Event sourcing is a way of building applications that never lose data.
Traditional applications update records in place. If a user removes an item from their cart, the application updates the database to reflect the new state. But that means we’ve lost valuable history—we can’t tell what was removed or when.
With event sourcing, every change is stored as an immutable event. Instead of just storing the final cart contents, we store every action. A user’s shopping cart interaction could look like this:
- html
| Event ID |
Cart ID |
Event Type |
Data |
Timestamp |
| 23 |
1234 |
CartCreated |
{} |
2025-01-12T11:01:31 |
| 24 |
1234 |
ItemAdded |
{“product_id”: 2} |
2025-01-12T11:02:15 |
| 25 |
1234 |
ItemAdded |
{“product_id”: 5} |
2025-01-12T11:05:42 |
| 26 |
1234 |
ItemRemoved |
{“product_id”: 2} |
2025-01-12T11:06:59 |
| 27 |
1234 |
CheckedOut |
{} |
2025-01-12T11:07:10 |
- alignment
- normal
To get the current cart state, we replay the events in order.
This simple shift—from storing state to storing history—changes everything.
“But Isn’t That Slow?”
Surprisingly, no.
Replaying events for a single entity is incredibly fast—it’s just a simple query that retrieves rows in order. I’ve been told that retrieving and replaying hundreds of events[hundreds]{Chances are you won't be building history from hundreds of events. There's a concept called "closing the books" that will keep your event streams small.} is faster than most SQL statements with a join clause.
And when you need to query large amounts of data, like for a dashboard or reporting status for a number of items, event-sourced applications create read models—precomputed views optimized for fast retrieval.
So while a CRUD-based system needs complex queries to piece together data stored across tables, an event-sourced system has the same data ready to go.
Why I Love Event Sourcing
No More “Going Forward” Caveats
The biggest win? When business needs change, I don’t have to tell stakeholders, “We can only do this for new data.”
Instead, I can just replay the history and calculate what they need—even for data that existed before we thought to track it.
Microservices Without the Complexity
At one point, this project I'm working on had five separate event-driven microservices, each serving specific purposes.
After adopting event sourcing in one of those services, I looked to see if we could simplify the project by relocating code to the event-sourced system and have the code subscribe to events.
To my surprise, I realized we could incorporate all of them into one service that was simpler to understand, maintain, and deploy.
Blazing fast views
By incrementally updating read models custom-made for your complicated web pages and API endpoints, those views that used to take time to render can return quickly.
I've seen this approach turn an expensive file-creation-and-download action into a simple static file transfer.
It Just Feels Right
I can’t overstate how satisfying it is to trust that no data is ever lost.
No more digging through logs to reconstruct what happened. No more uncertainty when debugging. The entire history is right there.
Should You Use Event Sourcing?
Absolutely. Yes.
Hear me out. I'm not saying you should go and rewrite your production app. I'm saying that you should use it.
Try it out on a small side project so you know what it's like.
I kept thinking I needed to build a full-blown application and write my own implementation of the event sourcing mechanisms before I would feel comfortable trying it for a job. As a result, it took me over a decade before I even tried it.
Instead, I ask that you be open about the idea of event sourcing, read this series, think about it, try it on as an exercise, and let me know what you think.
I’ve been inspired by Adam Dymitruk and Martin Dilger who both own consulting agencies that use event sourcing in for every project… even those focused on high-frequency trading. They've been operating this way for over a decade and have learned how powerful the pattern is and how to keep it simple. I'll be sharing what I've learned from them over the next few posts.
But for some perspective, after nearly 20 years of writing software and a few years of coaching people on how to write software, the way I write code has changed drastically in this last year.
Check back for the next post, where I'll get more into the practical details of it.
Read more...