This is the first article on a series about the most common problems startups face when implementing OKRs. This first one talks about the challenges in using OKRs to run product teams. Stay tuned for others as we release them!
A couple of months ago, I spent two days at The Creamery, in San Francisco, helping more than twenty fellow Y Combinator founders get their company's OKRs right. I sat with teams that are building social networks, games, D2C products, cybersecurity APIs, and even helping improve education in Africa, and after two grueling 10-hour days, I decided to write articles about the most common challenges they faced, and this is the first article on that series.
The number 1 challenge these founders had was setting OKRs for their software product teams. It usually looked like this: confusing OKRs with task lists defining features to be delivered. And it's honestly a really tough challenge we've also faced a lot at Qulture.Rocks. But we're learning our ropes and I think we can now help other teams who face the same troubles.
In product teams, OKRs are not about features to be shipped
One big challenge is that OKRs are a tough discipline, and most people get it wrong when trying to use it in product teams. The OKR ecosystem also doesn't help - a quick search over the internet led me to a competitor's website, where I found the following example as inspiration for a good OKR [1]:
"Objective: Implement new 360-degree product planning process
Key Results:
- Document clear role division between sales, marketing, design and (sic) development
- Decide on and document the process of input methods to and from sales, marketing, design and (sic) development back into product management
- Integrate user testing into all activities in product planning and design phase
- Integrate user testing into pre-launch testing phase"
Competitor website :(
Not to mention the re:Work website maintained by Google (the company that has basically helped the OKR gospel spread throughout Silicon Valley) that cites "eat 5 pies" as an example of a great objective, but I'll leave that for another post (if you want to learn the basics of OKRs, read our book, OKRs, From Mission to Metrics).
Anyway, most product team OKRs look something like this:
Objective: Deliver new iOS app
Key results:
- Design first concept
- Code MVP
- Get beta testers
- Publish the app by November on the app stores
What's wrong with it?
(Product) OKRs are all about RESULTS
If you read Inspired, a great book about product management by Marty Cagan, founder of the Silicon Valley Product Group and a big hotshot, you'll get the idea that we should ditch roadmaps as we know them (long lists of features to be built) and replace them with OKRs.
What he's saying is that roadmaps are bad for product teams because they aren't a direction teams can chase. Roadmaps are usually built by a hodgepodge of intuition, hunches, and highest-paid-people-in-the-room opinions.
Instead of a roadmap, product teams and their leaders should set clear OKRs - or results expected - and let their product teams figure out what to do - or what to build - in order to reach them.
Delivering features is an *effort* that will hopefully produce *results* down the line. Let's call features, or the effort product teams put up, "output."
But setting proper OKRs is tough, and usually, teams veer back to some sort of mishmash between OKRs and a roadmap. And roadmaps are collections of efforts.
When we ship a new feature, there's no guarantee it will generate any impact on our business, good or bad. Users may not even use them (and we know the hard way that's frequently the case).
So don't confuse efforts and results, roadmaps and, impact: when running product teams - actually, always -, OKRs should be about results and not efforts. Delivering features is an *effort* that will hopefully produce *results* down the line. Let's call features, or the effort product teams put up, "output."
Financials are not good results to aim for
Marvelous. Now we know that product team OKRs shouldn't look like roadmaps or features to be shipped.
We have to find results with which to work. So what results can a shipped feature produce?
At the highest level, and being rather simplistic, features should produce more revenues or fewer costs for the company that owns the product. That impact on the bottom line can come sooner, such as more conversion for an e-commerce website this quarter, or later, such as more perceived quality on a social network's timeline that will reduce churn and thus produce more revenues in the future.
But these sorts of business results are really hard to relate to or to optimize for. The impact of features is hard to map to the top line or the bottom line of the company. (Of course, some exceptions must exist: when a team works on fixing terrible bugs on the check-out flow of a website, they may impact sales quite directly, in the form of getting customers to close rather than to go shop elsewhere. But generally, things are a bit more nuanced.)
Another problem is that financials are a function of too many variables. We can't attribute to the checkout squad how much revenue customers are bringing into an e-commerce operation. It's also impacted by the product mix, the overall store usability, shipping costs, and speed, only to name a few. It wouldn't be fair nor helpful.
So we need to find other results to base our product OKRs on.
Outcomes
Joshua Seiden has a great book called "Outcomes over Output" that will help us make progress [2]. It's honestly a must-read for all product teams.
Seiden calls business results, like revenues, profits, and costs, "Impacts" (we'll capitalize the terms he uses when using them to his specification). He also calls the stuff product teams build, or features, "Output."
But Seiden doesn't stop there: he suggests there's something in between Outputs and Impact that we should be aware of: Outcomes. The definition of an Outcome is quite simple: "the human behaviors that drive business results." He goes on saying that " we want our customers to log onto our site more often, or put an extra item in their shopping cart, or share an interesting article with a friend, or upload a picture, or complete a task in less time. What do all of these things have in common? They're all measures of customer behavior. They might be small changes in a big system, but they are specific, and they allow our teams the flexibility to figure out the most efficient way to solve the problem, to deliver the behavior change that we seek, and to make a meaningful contribution to the impacts (revenue, profitability) that our executive leaders care about."
In our business (software that helps teams implement 1:1s, ongoing feedback, and peer-to-peer recognition), some examples of Outcomes we build for are:
- Number of feedbacks given by users
- Number of 1:1s done on time by users
- Percentage of users from the "direct report" persona who've gotten feedback in the last 30 days
You get the picture. Outcomes are user behaviors that we believe will drive Impact (or business results). As a rule of thumb, if your OKRs - more specifically, your key results - don't come out of Mixpanel, you're probably doing it wrong.
OKRs, Outcomes, experiments, and MVPs
You read that right: we believe Outcomes will lead to Impact.
What separates great product teams from mediocre ones is how often they get those assumptions right, and/or how quickly they iterate when wrong.
As a rule of thumb, if your OKRs - more specifically, your key results - don't come out of Mixpanel, you're probably doing it wrong.
Tying back to Marty Kagan, the fact that Output doesn't always lead to Outcomes, nor Outcomes to Impact, by the way, is why product teams should prefer OKRs over a roadmap: roadmaps should be flexible, and not set in stone, since they will frequently change in the journey of maximizing for Impact.
Of course, product teams - or most product teams nowadays - will still have some sort of backlog of features to build. But this backlog won't be simply a refined materialization of the roadmap: it will hopefully be a list of features that have been proven, to the extent possible, to generate Outcomes.
The process of finding out if features will indeed produce results is called discovery. Discovery happens via experiments, or the smallest efforts (i.e., time from the team) that can prove our hypotheses of what features will drive Outcomes, and what Outcomes will drive Impact. Product teams should always be running experiments that test those hypotheses, with the least code possible.
By the way, a product team should work on its discovery efforts in parallel to its delivery efforts. Discovery is finding out which features will lead to Outcomes. Delivery is building these features with production-level quality. The good thing is that Outcomes, on which your OKRs should be based, change with much less frequency. They tend to be metrics we'll optimize on the mid to long-run. It's hard to think of Facebook not maximizing the time users spend scrolling on their News Feeds [3], or the number of posts a user likes at a given active session.
Setting Outcome-based OKRs
Ok. Enough with the product-speak. So what does a proper OKR look like for a product team?
Ask yourself what Outcomes do we want to create with this Output (feature)? And then what Impact do we want to create in the business with this Outcome?
Let's say we have a product team at Qulture.Rocks [4] composed of a product manager, a designer, and a few front-and-back-end engineers. Let's also say that Qulture.Rocks' OKRs this quarter revolve around growing MRR from new customers, growing MRR from existing customers (via net expansion), and burning less than a given amount of cash each month, in order to preserve runway. Their objective [5] could look like the following:
Increase the activation rate of technical leaders with our product
Where did this objective come from?
Members of the executive team have found out that leaders in tech teams are a great persona to optimize for: they convert at a low cost (CAC,) and churn very little when properly activated (the process that starts with a sign-up and ends with a happy, active, and retained user). They've also found that they aren't very successful in activating technical leaders: more than 50% of those who sign up end up abandoning the product before experiencing its value.
Ok. Now, what key results [6] does the team have?
Key result: Net activation rate of users classified as "tech leads" from 50% to 70%
What's cool about this OKR is that our product team can run a great number of experiments of things they could do to improve our activation rate. They could create an onboarding flow that's specific to tech leads; they could call each tech lead on the phone to explain the value of the product to them; they could pre-load the application with product team OKRs so that tech leads quickly see value; or they could work on an email campaign to get these users back on the product (resurrect them).
Since OKRs should be aligned up (or unfolded down, whatever you prefer,) this OKR can be aligned - or contribute - to the following OKR:
Objective: Improve our customer retention
Key result: Gross monthly churn rate, from 2% to 1.4%
And this "churn" OKR can be aligned to the following company-wide OKR:
Objective: Improve our MRR in line with the best in the business
Key result: MRR, from $ 240k to $ 330k
What if we already have a roadmap?
Most probably, if you're reading this article you already have a roadmap. So I want to make clear you don't have to chop your roadmap into pieces right away, even though that would be neat.
What you can do is look at what's next in your roadmap. Ask yourself what Outcomes do we want to create with this Output (feature)? And then what Impact do we want to create in the business with this Outcome? These questions will allow you to first set the right frame of mind and, second, check if your roadmap makes business sense. Then you should gradually stop updating your roadmap and setting OKRs instead.
A big challenge for product team OKRs: enterprise software
Last, let's talk about some challenges enterprise product teams may face in this journey of using OKRs to change from Output to Outcomes to Impact.
Getting this right in enterprise software is the hardest for two reasons: the connection of Outcomes and Impacts may be weak or hard to prove, and/or Outcomes may be hard to properly measure. This is true because, respectively, in enterprise software, buyers rarely use the product (check out this tweet), and usage is usually affected by extraneous factors.
To explain the first case, let's imagine Jane the CFO works for Jalmart (fictional characters, please). She buys Oriple ERP because she wants to have all her financial information organized in one place (her employer is getting ready for an IPO.) Now let's move to a product team within Oriple's New Dehli offices, that takes care of a specific feature that handles payable invoices. Let's also say this team wants to improve the rate of invoices properly input on the system within the same active session. If they do this, accounts payable analysts within the companies that use Oriple will probably love how easy it has become to do that workflow. But will that really affect Oriple's revenues? Or its churn rate? Jane the CFO will probably never even see that feature. And it will take A LOT of kicking and screaming from the whole company to get her to switch ERPs, especially after the nightmarish implementation process that Agzzenture ran for two years.
In these cases, the assumption around if an Outcome will generate Impact is hard to test. Product teams will have to rely on their intuition (and market trends that point to the consumerization of enterprise IT) to get work done.
To explain the second case, let's imagine Ned the Analyst. He "uses" Facelook Wordplace because Jowen and Co's HR department implemented the corporate social network company-wide. "Uses" is in quotes because Ned rarely logs in. He thinks it is a nuisance. Come next month the company's CEO, Gretta, announces she's resigning in a weird Wordplace post, and everybody single employee in the company logs in to read it and gossip about it.
Now let's cut to a growth product team within Facelook that handles Wordplace, a young product with about 15 customers. They have an OKR that revolves around resurrecting inactive users and are running several experiments around email campaigns and push notifications. The team thought it had some promising A/B test results on the push front and were about to scope a full-blown feature for the backlog when they learned about the post at Jowen. They'll either have to scrap the OKR altogether and move forward by sheer faith, or stop everything they're doing and wait for more test results.
There's no right answer to this question.
You might think this is uncommon, but there are many enterprise products that have usage metrics polluted by "offline" behaviors, and it can be very hard to clean the metrics and get something meaningful. HR software is especially tricky, as we can attest.
That's it. I hope this helps you use OKRs to run your product teams :) Feel free to contact me at kiko@qulture.rocks if you have any comments or suggestions!
Notes
[1] Maybe they should change their business to SEO consulting ;)
[2] Like we frequently do at Qulture.Rocks, we've purchased five copies of the book for our product team to share.
[3] It takes a "time well spent" movement to change that.
[4] We do have one exactly like that.
[5] Let's remember: an objective is a qualitative goal, usually starting with a verb and ending with a non-numerical description of an aspect of the business to be improved. Some examples: "increase adoption," "reduce our cash burn," or "enhance our customer satisfaction."
[6] Let's also remember: a key result is the sum of a KPI (e.g., "net profit of the company in 2020"), a base level ("100.000") and a target level ("200.000".) We believe objectives should be narrow enough in scope to carry one or two key results at most. If your objective has more than two key results, you've probably too big a scope.
