Kreto
·9 min read

Product market fit survey: the 40% question and what to do with the answer

How to run the product market fit survey: the four questions, who to send it to, how to segment the score and what to do with each answer group.

Cover with the title Product market fit survey: the 40% question and what to do with the answer on a white ground.

A product market fit survey asks active users how they would feel if they could no longer use your product. Picture a founder before a board call: an investor asks "do you have PMF?", and the only honest answer is a feeling. The share who say "very disappointed" is read against a 40% line, but the real value sits in who gave that answer and why.

What a product market fit survey measures

The survey measures how much your users would miss the product, and many teams call it the Sean Ellis test. Sean Ellis described the method in 2009: he asks existing users how they would feel if they could no longer use the product. In his experience, product/market fit requires at least 40% of users to answer "very disappointed."

Ellis's original version had four answer options. Justin Jackson lists them as very disappointed, somewhat disappointed, not disappointed ("it really isn't that useful") and N/A ("I no longer use the product"). Superhuman, whose founder Rahul Vohra documented his version of the survey, used three: very, somewhat and not disappointed.

The 40% line comes from comparison, not theory. Ellis wrote that he defined it "after comparing results across nearly 50 startups," and he called the threshold "a bit arbitrary." Vohra's account in First Round Review says Ellis benchmarked nearly a hundred startups. The two counts differ, likely because the comparison set grew after 2009, but they describe the same line.

Ellis's own summary: "Those that struggle for traction are always under 40%, while most that gain strong traction exceed 40%." Read the line as a pattern that sorted past startups, not a law your company must obey.

Keep the core question word for word. Change the wording, and your score no longer compares with the benchmark. Drop the N/A option only if every recipient passed your recent-use filter.

Who to send the survey to

Send it to users who recently experienced the core of your product. Vohra followed Ellis's recommendation: people who used the product at least twice in the last two weeks. Someone who signed up and left cannot tell you how losing the product would feel.

Define "the core" as one event in your analytics, such as a sent message or a finished report. Then filter out teammates, test accounts, investors and friends. Leave churned users out as well. Their answers belong in exit interviews, not in this score.

The sample can be smaller than most founders expect. Superhuman had between 100 and 200 users to poll at the time. Vohra writes that you start to get directionally correct results around 40 respondents.

Read "directionally" literally. By our calculation, at 40 responses a measured 40% is consistent with anything from about 25% to 55%, at 95% confidence. At 400 responses, the range narrows to roughly 35% to 45%.

Range a 40% score could really sit in, by number of responses: about 25% to 55% at 40 responses, 35% to 45% at 400.
Fewer responses mean a wider range. Illustrative: our calculation at 95% confidence for a measured score of 40%.

Export the qualified list now, and attach plan, role and acquisition channel to each user. You will need those fields to segment the answers later.

The four questions to ask

Superhuman ran four questions in Typeform. The first produces the score, and the other three explain it.

QuestionWhat you read from it
How would you feel if you could no longer use the product? (very disappointed, somewhat disappointed, not disappointed)The score, and the answer group each person belongs to
What type of people do you think would most benefit from the product?How your best users describe your market, often more precisely than your persona doc
What is the main benefit you receive from the product?The one benefit to build and position around
How can we improve the product for you?What holds back users who are close to loving the product

Keep the form that short. Extra questions lower completion, and the people who finish long forms tend to be your most engaged users. Make the three open questions optional, but show them on the same screen as the first.

Copy the four questions into your survey tool and put your product's name where the table says "the product."

How to read the score: segment before you judge it

Never judge the blended score first. Vohra writes that Superhuman's first survey, in the summer of 2017, scored 22%, and the team concluded it had not reached product-market fit.

The team then assigned a persona to each respondent and looked at which personas filled the very disappointed group. Narrowing the market to those personas lifted the score to 33%. Nothing in the product had changed yet, only the group being measured. Within three quarters of work to improve the product, the score reached 58%.

Superhuman's very disappointed share: 22% at first, 33% after segmenting, 58% three quarters later, against the 40% line.
Segmenting moved the score before any product change. The product work then took it past the line. Source: Rahul Vohra, First Round Review.

Segmenting shows who the product already works for. A blended 27% can hide one segment near the line and another that barely values the product. The overall number averages them into a verdict that fits neither.

Cut the answers along lines your business already runs on: plan, role, company size, use case and acquisition channel. Channel matters more than teams expect. Users from one partner or one search theme often share a job, and that job can score far above the average.

Be careful with outside benchmarks. Hiten Shah posed Ellis's question to 731 Slack users in a 2015 open research project, and 51% said they would be very disappointed. The account does not say those respondents were filtered by recent core use, as Superhuman's were. Treat the Slack figure as a reference point, not a target for your qualified sample.

Build a pivot of the very disappointed share by segment before anyone sees the overall number.

What to do with each answer group

Each answer group gets a different job. Superhuman's split, shown below, turns three groups of answers into one roadmap.

Decision map: build on what very disappointed users love, fix what holds back those who share that benefit, pass on the rest.
The split Superhuman used to turn survey answers into a roadmap. Method: Rahul Vohra, First Round Review.

Very disappointed: learn what they love

Start with the main benefit question in this group. Group the answers by theme, and keep the exact words people used. Those words are your positioning, because they describe value your best users already feel.

Positioning work in product marketing starts from the same place: one audience, one main benefit, said in the customer's language. On Peerbie, Kreto positioned the product as an AI-first workspace, part of work the case page credits with $500K in revenue and 150% customer growth.

Draft your next homepage headline from the most common benefit phrase, word for word.

Somewhat disappointed: fix what holds them back

Split this group by main benefit. Superhuman's main benefit was speed, and users who named speed got close attention. Those who named something else were politely disregarded, since the main benefit did not resonate with them.

The improvement answers from the speed group shaped half the roadmap. Their requests included a mobile app, integrations, attachment handling, calendaring, a unified inbox, search and read receipts. The other half went to doubling down on what very disappointed users loved.

Everyone else: read, then pass politely

Vohra's team decided to politely pass over feedback from users who would not be disappointed. Building their requests pulls the product toward people who do not value its main benefit. Read their answers once for surprises, then set them aside.

Rank each roadmap half separately with ICE scoring, so quick fixes do not crowd out the core work.

Below 40%, near 40% and above 40%: what to do next

Use the score in your target segment, not the blended one, to pick the next move.

Target segment scoreWhat it tells youNext move
Clearly below 40%The product is not yet a must-have, even for its best usersKeep spending low and work on the very disappointed share
Near 40%You cannot tell yet, because the sample range likely crosses the lineCollect more qualified responses before you decide
Clearly above 40%Your best segment would miss the productCheck retention, then find the weakest funnel stage

Below the line, Ellis leaves little room. He writes that before product/market fit "it is critical to keep your burn low and focus all resources" on that very disappointed share. In practice, paid acquisition in this phase mostly buys users who answer "not disappointed."

Near the line, the margin of error decides. Keep surveying newly qualified users until the range sits clearly on one side.

Above the line, check the feeling against behavior. Lenny Rachitsky lists the survey as one signal among several, with retention first. He quotes Brian Balfour: "If it flattens off at some point, you have probably found product/market fit for some market or audience." The retention curve guide shows how to plot it by cohort.

Once both signals agree, the constraint moves from the product to the funnel. Kreto's growth marketing work starts at this point: find the weakest stage between first visit and retained user, then fix it.

Write down which row your target segment sits in and the one move that row calls for.

Where the survey falls short

The survey asks people to predict a feeling. Justin Jackson's critique puts it plainly: "it asks customers to guess about their future feelings." He cites Rob Fitzpatrick, author of The Mom Test: "humans are generally bad at forecasting their future behavior."

Jackson also notes that the results "might not be as useful for more mature products." He suggests questions about the past instead, and points to Brian Balfour's three business metrics: top-line growth, retention and meaningful usage.

The score works best as a lens on behavior you can already see. A segment that scores high but churns fast is telling you something the score alone hides.

Add one of Jackson's past-tense questions to the end of your form:

Has the product ever been unavailable?

Did you search for alternatives?

Run the survey as a running metric

One survey is a snapshot. Superhuman constantly surveyed new users to track how the score changed, on a weekly, monthly and quarterly basis. The team made sure never to survey anyone twice, so repeat answers would not throw off the 40% benchmark.

Set the trigger in your product, not your calendar. Send the survey when a user first meets the qualifying rule, and store their segment with the answer. Mark roadmap releases on the chart, so you can see which change moved which segment.

Build the trigger and the chart before your next release ships.

Frequently asked questions

How many responses does a product market fit survey need?

Vohra puts directionally correct results at around 40 respondents. Treat a score near 40% at that size as unresolved, and keep collecting until the range clears the line.

Is the 40% in the Sean Ellis test a hard rule?

No. Ellis himself called the threshold "a bit arbitrary." Use it as a benchmark, and give more weight to the trend in your target segment.

Can I run the survey before launch?

Not in a useful way, because the question needs people who have used the core product recently. Before launch, run problem interviews and test your positioning on sales calls instead.

How often should I run it?

Run it continuously: survey each newly qualified user, and ask each person once. Review the score by segment every week, and judge the direction over a quarter.

What if the very disappointed group is tiny?

Read every answer in it anyway. A handful of users who would truly miss the product is a starting segment. Find more people like them before you widen the product.

Find the stage that loses your segment

You now have a score, a segment and the words that segment uses. Next, find where your funnel loses those users, from first visit to retained account. If you want a second view on that stage, talk to Kreto about where your segment drops out.