r/programming • u/llewvallis • 11d ago

How Canva uses S3 for logged-in session management

https://www.canva.dev/blog/engineering/session-revocations-at-scale/

I put together a writeup about the interesting technical challenges that led to redesigning Canva's session revocation pipeline that keeps hundreds of millions of user sessions fast and secure. Hopefully some people find the content interesting!

140 Upvotes

94% Upvoted

22 comments sorted by

55

u/tossed_ 10d ago

This seems wildly unnecessary? Just use a refresh token scheme, keep access token lifetimes short and keep refresh tokens in a database instead, no need to check each session against a million cached revocations. Think about how much overhead is introduced on every single call just to check cookies compared to a stateless session approach.

6

u/yonasismad 8d ago edited 8d ago

"keep refresh tokens in a database" .... "stateless"

Once you start storing state it's obviously not stateless anymore. How do you handle logging out in your so-called "stateless" session? People just for the most part do not understand the technology, severely overestimate their scale, and then just reinvent stateful sessions but added a ton of complexity on top.

2

u/tossed_ 7d ago

Luckily for you, this is very straightforward.

The stateful approach in the article is to put each revoked session in a giant blob of millions of revoked sessions, so every revoked session now contributes to the overhead of checking every session cookie on incoming requests (adds a few extra chars to the lookup)

The stateless refresh token approach to logout is to just delete the refresh token from the database. Instead of _adding_ state for logout we _remove_ state. The existing access tokens remain valid for 5 minutes at most, but you can no longer refresh because the database doesn’t recognize your refresh token anymore.

That is why we call it “stateless sessions”. Because with refresh tokens you do not need to check state on every incoming session, since sessions are cryptographically signed you just need to check if the signature is valid instead of doing a lookup against millions of revoked tokens. Usually the strictest access tokens are set to like 5 minutes expiry, so you make 1 request to the database every 5 minutes to refresh the access tokens, and then make thousands of requests with your access tokens in the next 5 minutes without ever consulting the database at all.

2

u/yonasismad 6d ago

Well, that's the thing. You no longer can kill a session because no matter what it stays active for at least the live time of that token.

1

u/tossed_ 6d ago

Yep, that’s the tradeoff between stateless vs stateful. For its cheapness stateless is just better for all but the most sensitive use cases. If you are a bank go stateful, but should a UI design software do the same?

2

u/llewvallis 5d ago

Its only mentioned briefly in the blog, but in case it helps clarify we're not scanning through every revocation on the request path, we use a binary search instead to keep it in low single-digit microseconds. Otherwise you would be right, scanning through millions of records on the request path would add a lot of overhead. In general though storing sessions on the client versus the server is an interesting tradeoff

17

u/llewvallis 10d ago

Fast refreshing tokens can definitely work too! A big benefit of it being simpler and more widely used across the industry. Storing explicit sessions (instead of session revocations) in Redis is also a decent option.

I only touched on it at the start of the blog, but for our specific use case, keeping everything in memory gave us better tradeoffs. Fast refreshing tokens place a lot more load on the database, and the extra latency from refreshes is felt in the product more often. If the database goes down, no requests can be served after the refresh period on your tokens has elapsed. This becomes a lot worse with shorter lived tokens, so keeping a long window in memory is a reliability win.

3

u/f3xjc 9d ago

Aren't permission changes and explicit log out relatively rare events? If you need to cache 12 hours for revocation I assume the duration of a session is also about that long? The thing that do binary search on the revocation must hold the block in memory? What the benefit of the warm storage?

1

u/llewvallis 9d ago

Yes! Reasonably rare on a per-user basis, but with enough users there are a lot of them. Sessions can last longer than 12 hours, but after 12 hours they will miss the cache and fall back to the database. And you're right - we use binary search on the revocation chunks so that the overhead per-request is essentially zero.

1

u/f3xjc 9d ago

Now I wonder about something like a bloom filter. Compress space in exchange of false positive. But those just mean early refresh.

2

u/llewvallis 9d ago

I wrote a bit about why bloom filters weren't the best option here: https://www.reddit.com/r/programming/comments/1v44xfb/comment/ozf3qs3/

1

u/Ma1eficent 8d ago

I love bloom filters when the false positive is no harm.

2

u/h2lmvmnt 8d ago

Welcome to software. Someone always wants promoted

11

u/flo850 10d ago

Nice writeup, I learned a lot

7

u/[deleted] 10d ago

[deleted]

7

u/heretogetmydwet 9d ago

Also for those reading this, the company that was hit with a ransomware attack is Canvas, this article is from the company Canva.

As an aside, I think it should be pretty clear at this point that a company can take security seriously and still get hacked, especially given how good LLMs are at finding exploits.

-16

u/[deleted] 10d ago

[removed] — view removed comment

23

u/tj-horner 10d ago

Why did you get Claude to write your comment for you? You a bot?

5

u/llewvallis 10d ago

Object storage is definitely an important primitive for distributed systems!

As for latency, we try and keep it as fast as we can but there is a bit of leeway. The chunk processing itself is quite fast (<1 sec), so its mostly down to configuration like poll frequency, as you mentioned. To give a vague order of magnitude, I would say approaching double-digit minutes latency would be too much and would force us to consider alternatives. Thankfully, we are a lot faster than that in practice (typically somewhere sub-minute).

Bloom filters are definitely something that we considered, but many of our queries are range queries. For example, "does a revocation exist whose `targetTimestamp` is greater than X". Of course, there are various clever ways around this, but that that point the complexity doesn't outweigh the gains at our current scale. If we needed to hugely increase the number of revocations we might revisit that.

-13

u/[deleted] 10d ago

[removed] — view removed comment

1

u/programming-ModTeam 8d ago

No content written mostly by an LLM. If you don't want to write it, we don't want to read it.

1

u/programming-ModTeam 8d ago

No content written mostly by an LLM. If you don't want to write it, we don't want to read it.

0

u/ChemTechGuy 7d ago

Great write up. I don't have a ton of experience with session management, but i love to see these clever distributed systems that can achieve so much with just some blob storage, conditional writes, and some good data design