Push to Prod

Push to Prod

Home
Archive
About

concurrency-war-stories

Best Questions and Answers to the “Terrifying Netflix Concurrency Bug” Post
Why couldn't we roll back? Weren't we concerned about costs? Why is operating Netflix so complicated? Plus many others.
 • 
Matthew Hawthorne
2
Making an AB Test Allocator 20x Faster Using Non-blocking IO
Many moons ago, I inherited a cross-company AB test and had to figure out how to allocate it effectively.
 • 
Matthew Hawthorne
How Would You Design Amazon S3?
One of my favorite system design questions from a few years ago and a fascinating thing I learned while answering it.
 • 
Matthew Hawthorne
One Key Aspect of Optimizing Computational Throughput
When optimizing throughput, think carefully about how many CPUs you have when adding threads or processes.
 • 
Matthew Hawthorne
How We Built a Self-Healing System to Survive a Terrifying Concurrency Bug At Netflix
Our CPUs were dying, the bug was temporarily un-fixable, and we had no viable path forward. Here's how we managed to survive.
 • 
Matthew Hawthorne
1
When Things Are Slow, Look for Queues
When your system is slower than desired, queues are often heavily involved. Here's an overview of the most common situations.
 • 
Matthew Hawthorne
Metrics Are The Map, Not The Territory
We can never obtain a complete and perfect understanding of what our systems are doing. All we have are incomplete signals to inform our theories.
 • 
Matthew Hawthorne
Comprehension of Concurrency is a Lifelong Journey
Accept that your understanding is imperfect. There is comfort in letting go.
 • 
Matthew Hawthorne
© 2025 Matthew Hawthorne
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture