While AI has totally collapsed the cost of shipping a piece of software, the cost of shipping the wrong software has never been higher!
Why it matters: The key shift nobody is naming is the move from "can we build this?" to "is this worth building?" The only way to answer the latter question is to work with people, not the technical AI layers. And by the time you know the answer, your user base evaporated into a new tool, or built their own.
An(other) uncomfortable truth is the math: you and your closest competitors are using largely the same models, trained on pretty much the same public internet. The same representations shape the decisions you're both making in Claude or ChatGPT, whether you are asking for feature prioritizations or to evaluate the usability of an interface. Your only differentiator is the information you've put into the model, to make it understand your goals, context, and what to do about the delta.
So we are watching as products across the tech space converge and flatten. Look at your business vertical and think about the five top products: how different are they, really?
Closed loops collapse.
Systems and loops that solely feed on their own outputs collapse, as shown by Shumailov et al in Nature (2024). This means your closed AI loops are not only degrading, your most valuable differentiators are the first things to get tossed. "But wait!" I hear, "Gerstgrasser showed that synthetic data accumulated alongside the real data avoids collapse (Gerstgrasser et al., 2024 preprint)!" Yeah, we agree: a corpus of real user data is required! Let's go get novel training data!
While that finding is focused on LLM training, it rings entirely true about product teams as well. Teams that learn from their own ideas, their own analytics, their own inputs and summaries, they erode over time. This is the value of diversity, not only in the DEI sense, but in the "diversity of ideas" sense. Differentiation requires external inputs – not always easy, but real.
What differentiator is left?
So what differentiators are left? The only proprietary insight I see left, is your customers. Not model-collapsed synthetic users, but real human users who can't see well in the sunlight and sometimes are late to daycare pickup. Actual humans who are solving problems with your products, and can tell you what is working and what is not. Building that differentiator into your shipping loop is your only remaining possible moat.
There is a method to do this research, and it's called: rolling research. This is a standing weekly cadence where product teams build research plans, real users react to the work, and product decisions can be made in hours and days, not weeks and months. Feedback cycles matter, and the best-oriented OODA loop always wins.
And this is the type of research I've run at scale. At Thumbtack I took a superb MeasuringU pilot and turned it into a standardized two-week-turnaround pillar of the product organization. We generated a 75% increase in projects and attained 94% CSAT, and, dear reader, those are not the metrics that matter most. What matters to me, is that my teams are more informed and feel stronger in their product decisions.
"Yes, but research slows us down. We move fast."
"Yes, but," I hear, "Research slows us down. We move fast."
What you're describing is research, but it's not focused. It's a 40-page slide deck nobody reads or can take action on. Learning is only valuable if it's done at the right time so the brain that needs the info, has it. And fortunately, slow research is a flaw in program design, not research. That's what I'll tackle next week!
This week, the bottom line is: every day you build without user evidence, you are compounding your decisions in an unknown direction. Unless you are doing studies without leading tasks and with participants you didn't handpick, you are moving fast without any data (or, far more likely, moving fast with post-hoc rationalized quant data that tells you what you think you want to hear). So building faster makes building on better data even more valuable! The teams who learn to learn best are the ones who will win. The teams who get the tightest OODA loops, who learn how to ship the right thing fastest.
So take the Five Feature Test today: list the last five features your team shipped. Next to each one, write a sentence about what user evidence showed it was worth building. (We'll revisit this next week!)
And if you can't finish a sentence for three of the five features, you know your user data problem is real, and you're starting to understand it's impacting more than half of what you're shipping.