Should product decisions be driven by data or by gut feeling?
Experts
Measure everything. A/B test relentlessly. Let the numbers guide your roadmap.
Great products come from taste and vision, not dashboards. Trust your instincts.
Lessons from scaling Uber and Opendoor | Brian Tolkin (Head of Product at Opendoor, ex-Uber)
And there turns out there are a decent number of other ways to do that. The first best, most obvious is talk to more customers. But there are other statistical techniques that again, aren't as rigorous or good, but may be possible.
And then I think the last part, which is some part of experimentation is if you just ship something because it's your intuition or it's where you want to see the product go, do you have a reasonable feedback loop to understand whether or not you are correct? So that could be customer support or ticket volume or feature adoption, whatever the case i...
Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor)
When I was at Airbnb, one of the things that I loved most was our experimentation platform, where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Eppo does all that and more, with advanced statistical methods that can help you shave weeks off experiment time, and accessible UI for diving deeper into pe...
Eppo also makes it easy for you to share experiment insights with your team, sparking new ideas for the A/B testing flywheel. Eppo powers experimentation across every use case, including product, growth, machine learning, monetization, and email marketing. Check out Eppo at geteppo.com/lenny, and 10x your experiment velocity. That's geteppo.
The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)
Now, it's also possible to do a complete redesign. Just, as you said yourself, be ready to fail. I mean, do you really want to work on something for six months or a year, and then run the A/B test, and realize that you've hurt revenues or other key metrics by several percentage points? And a data-driven organization will not allow you to launch.
And the key thing is it didn't hurt the user metrics. So it's very easy to increase revenue by doing theatrics. Displaying more ads is a trivial way to raise revenue, but it hurts the user experience. And we've done the experiments to show that.
The ultimate guide to paid growth | Timothy Davis (Shopify)
Eppo also makes it easy for you to share experiment insights with your team, sparking ideas for the A/B testing flywheel. Eppo powers experimentation across every use case, including product growth, machine learning, monetization, and email marketing. Check out Eppo at geteppo.com/lenny and 10X your experiment velocity. That's get E-P-P-O .
Yeah, I mean, there are many ways to judge the effectiveness of growth overall. Like some of the stuff we talked about before, brand metrics, awareness, recall, things along those lines. There are also, I know some companies look at leading indicators like visits, and clicks, or tribute, which of the efforts are linked to or perceived drivers of ac...
Thinking like a gardener, slime mold, the adjacent possible: Product advice from Alex Komoroske
When I was at Airbnb, one of the things that I loved most was our experimentation platform where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Eppo does all that and more with advanced statistical methods that can help you shape weeks off experiment time, and accessible UI for diving deeper into perf...
The other thing you do is once a week or so, you want to make sure that you never propose something in the group that people go, "Eh." You always want to do a thing people go, "Yeah." So what you do is you see that if you're talking to other people, you say, "I wonder if we should have a live conversation every so often, right?" The chats were so f...
Lessons from 1,000+ YC startups: Resilience, tar pit ideas, pivoting, more | Dalton Caldwell (YC)
There was no universe where they would've made up the idea for Segment because they didn't know anything about how analytics worked, but because they were grinding for multiple years and became experts on these things is a side effect of their earlier ideas. They ended up with really good unique insights.
Yeah. And again, it depends on the idea space you're working on. Some are more, some are less. So yeah, it should be a fair amount of time. And nothing substitutes for an actual conversation versus just staring at analytics dashboards.
10 growth tactics that never work | Elena Verna (Amplitude, Miro, Dropbox, SurveyMonkey)
My last one is also that I see something way too often nowadays on growth teams specifically. This is very, very gross problem. And that is too much risk averseness on growth, where you're starting to test everything.
In statistics, six is tricky. Many people take it for face value versus it's just like a directional data point to say there's likely new distribution that has maybe a different mean, and 5% of the chances. If you measure in 95% statistical significance, you might not even be there, and yet really take it for so granted.
35 years of product design wisdom from Apple, Disney, Pinterest and beyond | Bob Baxley
Well, you have to bring a lot of people in once you figure out what you're doing. And so to your point, once you realize you're building Disneyland and you've got the whole thing set and people know what it's about, then they can come in and understand, oh, I'm playing my piece over here.
And then is there something we can do where we just have that debate once and for all, we decide as an organization, we're going left instead of right, and you're absolutely correct. You have to be opinionated, but that's how you're going to win. There's no unopinionated software that's been successful. You have to have a point of view.
How to build a team that can “take a punch” | Hilary Gridley (Head of Core Product, Whoop)
There is some value I think in having a little bit of a chip on your shoulder. You see that people who are really successful, they do have a little bit of like, "I'm going to prove them wrong." And so I don't want to say that you shouldn't think about it at all or you shouldn't care.
I mean, I have a list of myself. I've got a list on my phone of my behavioral activations, and it's things that I know I can do if I start feeling like the walls are closing around me, if I feel myself getting sucked into very low mood or negative thinking, or whatever it is.
How to speak more confidently and persuasively | Matt Abrahams (professor, speaker, author)
Some examples include their priorities in product roadmap are driven by a 100-year vision that comes from Tobi, the CEO. And the core product teams don't have metrics or KPIs. They're essentially banned. And instead, decisions are made based on taste, and intuition, and building towards this long-term vision.
What do you think it takes to make something like that work? Because in a bad case, this team just sits around and builds whatever they want, and the rest of the company's like, "God damn, this sucks. I have to show success in metrics and moving a metric in this team, over there just build beautiful things.
Dylan Field live at Figma's Config: Intuition, simplicity, and the future of design
Dylan Field There's always these areas where things feel murky, and sometimes it's because you just haven't done the work to understand them yet fully. Sometimes it's because no one's done the work to understand them fully.
Then, people would correctly ask me all the why questions for, "Why now? Well, we haven't made Figma Design everything that it needs to be at, why go into this other area? Why is this critical as a company that we do this?" I had a lot of intuition, not a lot of reasoning about it.
Making time for what matters | Jake Knapp and John Zeratsky (Authors of Make Time, Character VC)
Jake Knapp Yeah, so here it is. And you can see that Marie is more likely to swear than Ben apparently. But it's crucial, right? And many founders, they'll have different ... one of the challenges is that for each founder, I think, each set of founders, there's maybe a different way that they phrase what that means. What conviction means to them.
And so what the teams will do is to plot these options on these lenses. They'll almost always create some lenses of their own, some custom lenses that matter to them. They might have one about their conviction. I think there's sort of a humorous one for Mellow where it's like, which founder is like, "F yeah, that's exactly what I want to build.
The rise of Cursor: The $300M ARR AI tool that engineers can’t stop using | Michael Truell
And we think that there are problems with both of those visions. I think that on the chatbot style end of things... And we think it's going to look weirder than both. The problem with the chatbot style end of things is that lacks a lot of precision.
Yep. We think it ends up looking like that, and we're very opinionated that that path goes through existing professional engineers, and it looks like this evolution away from code. And it definitely looks like the human still being in the driver's seat, and the human having both a ton of control over all aspects of the software and not giving that ...
Vision, conviction, and hype: How to build 0 to 1 inside a company | Mihika Kapoor (Product, Figma)
Yeah, absolutely. So, my take is that vision is everything. It is really important to create a vision that you believe in, that your team believes in and that your company believes in. Because the reality of the product development cycle is that it's so messy, it's so chaotic. You're going to have extreme highs and extreme lows.
And so, I think a lot of folks when they think about visioning, they kind of think about, "Okay, how do we start from scratch and learn about the user and then translate that into designs and then translate that into engineering?" And it becomes this very almost linear process.
Peter Deng
Yeah, I mean, just to go further on the Instagram part, the idea was super simple. It was showing photos and visual sharing. But the craft that Mike and Kevin had in putting in the hard work to get the product just right, that's what made it really take off. That's a great example.
And I spent hours obsessing over them because I really wanted to make sure I captured the right essence of what I was trying to say. And I think that crafting is really important when you're working in product, because if you're sitting down and you're writing a vision doc or you're writing a PRD, and if you don't pay attention to the words you use...
232 votes cast
1 comment · Share your perspective
Data is a flashlight, not a compass. It illuminates where you are but doesn't tell you where to go.