Please, Stop Talking About "Tradeoffs"
Because sometimes, one pro is worth ten cons
Intro
Hello friends. In the Pragmatic Engineer post I co-wrote a few weeks ago, my main point was that the primary artifact of good architecture is good trades:
Is it cringe to quote myself? Let’s find out:
Good architecture work is about purposefully trading problems you have today for better problems tomorrow.
If you’re not upgrading your problems, you’re just rearranging furniture.
There’s a similar word often used in architectural discussions: “tradeoffs.” But this is not the same thing as a trade. Let’s get into it.
An Example Trade
The final example trade I used in the aforementioned post was shifting from regional to global personalization models at Netflix in 2015. You can read more about it here:
https://netflixtechblog.com/recommending-for-the-world-8da8cbcf051b
Summary: Prior to 2015, Netflix trained separate personalization models per geographic region, such as North America, Latin America, Europe, or Asia. In 2015, they shifted to using global models, meaning that for each logical model, such as “Personalized Video Ranking”, there would be a single global model instead of N regional models trained separately.
Note that I wasn’t working on the Personalization team at Netflix when this decision was made — I joined a few months after. But my first project after joining the team was a direct result of the repercussions: we needed to produce high-quality, regionally diversified training data for global personalization models.
More specifically:
We traded the simplicity of training and serving regional models for the challenge of curating diverse training data for global models.
You can read about all of the product and technical benefits of this decision in the Netflix techblog post — Challenges 1 and 2 are the most relevant ones. But from the purist perspective of “is this a good architectural choice?”, the answer is yes, because we traded our current problems for much better ones.
The work I did, which was intelligently sampling and curating a list of Netflix members from which we’d derive training data, is an exponentially more interesting problem than regionally partitioning model training and serving.
Choosing better problems also provides a multiplicative benefit over time. My focus was on sampling data across geographic regions, so I learned about the nuances of stratified sampling. But then we started thinking about other dimensions to sample for, like tenure and device type. We also started thinking deeply about the sampling requirements of different models, which resulted in many improvements that I’ll discuss in a future post.
“Tradeoffs” is a Limited Worldview
What are people trying to communicate when they discuss the concept of “tradeoffs”?
I believe it’s mainly 2 things, with some overlap:
All decisions have repercussions
There are no perfect solutions
In other words, there is no decision without negative consequences
There are some positives here. “There are no perfect solutions” seems obvious, but needs to be communicated in some form every day in the modern corporate environment.
The ability to identify multiple solutions to a problem and evaluate the pros and cons is an important skill threshold for engineers. Especially when compared with the opposite — immediately latching on to a solution and pursuing it with tunnel vision.
But there are negatives also. Many times, I’ve seen the concept of tradeoffs used to cloud a conversation with complexity rather than making a clear choice and moving forward. This can happen due to fear of failure, which I can empathize with. But it can also be a deliberate stalling tactic used by those who do not want a project to succeed. Manufactured complexity can act as a useful fog of war.
I’ve also seen tradeoffs used to hyper-focus on negative consequences of choices. In the worst case, we optimize for the solution with the fewest number of negatives rather than the highest cumulative impact of positives.
Imagine we were attempting to make a decision about the trade in the previous section, regional vs. global personalization models, through the lens of tradeoffs.
Here’s a list of pros and cons:
Regional Models
Pros
Regionally partitioning training data and inference requests is straightforward.
Cons
Prevents learning of connections between related users and content across regions.
Struggles to maintain local taste variations
Global Models
Pros
Improves global recommendation quality by enabling global learning of user preferences and content similarity.
Cons
Must generate diverse training data (stratified by country, for example) to support balanced global learning.
This is my core point: The language of “tradeoffs” doesn’t account for how superior of a problem generating diverse training data is compared to serving regional models.
It flattens the problem space. It doesn’t have a way of representing the amplitude of one pro or con vs another, and thus the long-term benefit of a decision.
These decisions are often made by executives without much time to think about them. Their job is to make quick decisions, and it is our job to provide them with the requisite information to make them. We bias the information towards the decision that we think is best for us, the company, or, hopefully, both.
It’s very easy to win one of these debates by criticizing a particular solution for having too many negatives, while that same solution has a single positive with 10x the impact of its competitors. Decision making has to be about much more than comparing a flat list of tradeoffs — we’re debating alternate futures, often with notably different cultural and technical topographies.
If you’re interested in reading more about why it’s so easy to lose an architecture debate, check out this post:
Decision Making Is The Skill
Decision making is the hardest skill in software engineering. When faced with a difficult decision, exploring and evaluating the options is a key chunk of work. But at some point, we must move forward.
In my career, I’ve found that moving the vibe of a project from “we should do something” to “we should do this specific thing” to be one of the hardest transitions. I have failed at it many times. There are always so many options, lots of lengthy conversations, and the prospect of making a bad choice is terrifying.
But fear not — there are multiple techniques to move a project forward, despite personal doubts, corporate haters, or limited and imperfect information. “Would this decision upgrade our problems?” is not something that I ever heard explicitly said at Netflix, but it was implied in many proposals and produced a lot of excellent work. Try this approach at your next opportunity and I guarantee it’ll shift the conversation into a better place. At the absolute minimum, it’ll be much more interesting than a flat and monotonous discussion about tradeoffs. Thank you and have a great day.
If you want to read more details about the architectural trades required to build systems of massive scale, check out:
PUSH TO PROD OR DIE TRYING: High-Scale Systems, Production Incidents, and Big Tech Chaos