Rigorous research talks to
the right people.

The most dangerous research participants don't look dangerous, they look familiar.

Why it matters: Recruiting convenient users who are familiar with the product feels good. After all, we want to know what is going right. But the damage shows up later in findings that don't transfer because they are built on power users, not a customer who behaves normally. Building a corpus that integrates the right customer insights is key to building a durable industrial insights moat.

The three most popular participants

A power user – this participant lives in the product. They respond to everything you send, and are an unabashed super-fan of your software. Not only do they know all the ins and outs, they have specific use cases in their business. These use cases are common across industries, but the way they'll have you build the feature suits them, their use case, and their CRM. Everyone else will spend the next six months on workarounds. This is why we talk to expert users to get great ideas, and test these great ideas with our broader base.

Our internal expert – this participant lives in the product. They were integral to building our success and they understand the inner workings not only of the product, but also the business. They are able to explain why products work the way they do. These experts are critical when implementing features, but think too holistically about the product ecosystem, running into thought blockers before the feature ideation has even happened. Instead of testing usability, these experts think through the edge cases, gotcha moments, and can make a feature seem practically impossible. The curse of knowledge is real!

The family friend – this participant lives in our comfort zone. They are trusted, kind, and a person who is just like other people: a good person. They want us to succeed so badly, they'll say anything they think makes us feel successful – even if it's not true. This support is crucial in overcoming the emotional hurdles of product-building, but when we build, we build in reality. Separating the love and care of our community from what is best for us and our product, is an important distinction.

So if we aren't focusing on power users, internal experts, or friends, who the f do we f with?

Representative means contextually situated, not demographically average

Demographics matter, yes, but as an afterthought.

You may be wondering why, when demographics are the most common way of segmenting a population, we are abandoning them for recruiting a population. Great question!

Let's play "would you rather!"

Would you rather... interview someone who:

1. Has ordered products from five competitors in the past month (behaviors)
2. Is 30-35 years old and lives in California (demographics)

Would you rather... interview someone who:

3. Has purchased your product for a person that is not themselves (behaviors)
4. Is 55-75 years old and has $500k+ in retirement savings (demographics)

Behaviors are what your users are doing, and you want to see how your product helps in those different situations.

Think about the situations your users find themselves in – the behaviors you need to observe.

When screening is study design

So what do we do about it? We make like a door and screen.

The screener is really just an operationalization of our research questions and behaviors. If we are looking for people who use a certain feature, we ask about feature usage, frequency, and intent.

This ensures our participants are prepared, both by being users of the product, and also by taking typical actions in the product. If we are talking about a billing feature, recruiting a user who rarely interacts with it would be fruitless.

Skewed screening baises Retreival-Augmented Generation

Unless you've built the corpus around retrieval, your system doesn't know who you recruited.

Having these behaviors documented to index findings against is crucial. And this is especially important if your current corpus is partially built on different types of users who have different core behaviors or tasks! Enabling retrieval against different behaviors and circumstances is exactly what we are aiming for!

Team alignment requires understanding context, situating within that context, and moving forward. To create retrieval systems, we need to add that context in first thing, or it will not exist.

Unlabeled, both power-user findings and behavior-based findings look equally important.

The gap between a product decision that lands and one that doesn't, is having users who behave normally succeed or fail. Ensuring we have a robust dataset of users that represent common behaviors is the first step.

Yes but – My users are hard to find

This, my friend, is the entire purpose of rolling research: to build the muscle of constant customer contact. The first recruits may be hard, but the second is easier and by the tenth, we are humming along like a well-oiled machine.

Tools like Rally help you reach out to your customers, create pools, and capture data on your user base at scale. dscout helps fill in the corners with gen-pop users. With two tools, you've pretty well covered your recruiting needs.

Inconvenient one-time costs like writing screener templates, bringing on paneling tools, and learning to learn are part of the practice.

The good news is this practice compounds across the program, building ground truth rooted in real users.

The bottom line: recruit participants with core behaviors to build a strong base of evidence and enable a robust retrievable research corpus