How to Analyze Open-Ended Survey Responses
Open-ended questions are where the useful answers are. A rating tells you that satisfaction dropped; the free text tells you it dropped because the queue at 8am got longer. The problem is that 300 paragraphs of free text don't fit on a slide. This guide walks through the method researchers use to turn them into themes you can count, rank, and quote, and it works just as well in a spreadsheet as in any dedicated tool.
Why are open-ended responses hard to analyze?
Every answer is phrased differently. "Too slow," "the wait is ridiculous," and "I gave up and left" are the same complaint, but nothing about the text tells a spreadsheet that. So the job is mostly one of grouping: deciding which answers are saying the same thing, then counting the groups. Everything below is a disciplined way of doing that grouping so the result is consistent rather than a list of the quotes that happened to stick in your head.
Step 1: What should you do before you start tagging?
Read a random sample first, around 30 answers or so, without tagging anything. You're looking for the recurring subjects people raise. Don't start from a list of themes you expect to find: if you decide in advance that the answers will be about price and quality, you'll find price and quality and miss the thing nobody on your team thought to ask about.
If you have more than one open question, analyze each one separately. "What did you like?" and "What would you change?" produce different themes, and pooling them muddles both.
Step 2: How do you build a codebook?
A codebook is a short list of themes, each with a one-line rule for what belongs in it and an example. The rule matters more than the name: it's what keeps you (or a colleague) tagging the 200th answer the same way as the 20th. Here is what one looks like for a cafe that asked "What would make you visit more often?":
| Theme | Belongs here if it mentions | Example answer |
|---|---|---|
| Wait time | Queues, slow service, waiting for an order | "The line at 8am is out the door." |
| Price | Cost, value for money, discounts | "A bit steep for a daily coffee." |
| Seating | Space to sit, tables, crowding, power outlets | "Never anywhere to sit with a laptop." |
| Menu range | Food and drink options, dietary choices | "More plant-based lunch options." |
Keep it to something like five to fifteen themes per question, plus an "other" bucket. Expect to revise it: if "other" grows past about one answer in ten, there's a theme hiding in it.
Step 3: How do you tag (code) each answer?
Go through every answer and give it one or more themes. In a spreadsheet, the simplest layout is one row per answer and one column per theme, with a 1 where the theme applies. A few rules keep this honest:
- Allow more than one theme per answer. "Too expensive and nowhere to sit" is both Price and Seating. Forcing a single theme undercounts whichever you didn't pick.
- Tag what was said, not what you think was meant. If someone writes "it's fine," that is not a complaint about anything in particular.
- Track tone separately. A second column for positive, neutral, or negative lets you see that Menu range is mostly praise while Wait time is mostly complaint, which the counts alone won't show.
- Spot-check with a second person. Have someone else tag 20 answers using your codebook. Where you disagree, the rule is too vague; tighten it.
Step 4: How do you turn themes into findings?
Count how many answers carry each theme and rank them. Report the count and the base together: "62 of 180 answers (34%) mentioned wait time" is a finding; "wait time came up a lot" is an impression. Because answers can carry several themes, the percentages won't add up to 100, and that's correct.
Then pick one or two quotes per theme that say it in a customer's own words. Choose typical ones, not the most dramatic: a quote should illustrate the theme's count, not stand in for it. Finally, check whether themes differ by group. If the Wait time complaints come almost entirely from morning visitors, that's a more useful finding than the overall count. For how to package all this for a decision-maker, see how to present survey results to leadership.
How many open-ended responses do you need?
Fewer than for a rating question. When you're looking for themes rather than precise percentages, 20 to 30 answers per question is often enough to see the main themes start repeating. If new answers keep producing new themes, keep collecting. Once they stop, you've heard most of what there is to hear. For percentages you want to quote with confidence, the usual sample size rules still apply.
How do you write open-ended questions that are easier to analyze?
Ask about one thing at a time. "Any other comments?" produces answers on every subject at once, which are the hardest to code. "What one change would make you visit more often?" produces answers that already share a frame. Pairing a rating with a follow-up ("What is the main reason for your score?") works especially well, because you can read the reasons separately for high and low scorers. More on wording in how to write survey questions that get honest answers.
Frequently asked questions
What is the best way to analyze open-ended survey responses?
Read a sample first, build a short list of themes (a codebook) from what people actually wrote, tag every answer with one or more themes, then count how often each theme appears and pull a representative quote for each. The counts tell you what matters most; the quotes tell you why.
What is coding in qualitative survey analysis?
Coding means tagging each answer with short labels that describe what it is about, such as "price" or "wait time". Once every answer is coded, you can count the codes, which turns free text into something you can rank and compare.
How many themes should a codebook have?
Usually somewhere between five and fifteen for a single question. Fewer and the themes are too broad to act on; many more and each one has too few answers behind it to mean much. Merge small themes into an "other" bucket rather than keeping dozens of tiny ones.
Can AI analyze open-ended survey responses?
Yes, and for anything beyond a few dozen answers it is much faster than doing it by hand. The thing to check is whether the numbers in the result can be traced back to the actual responses. A summary that invents a percentage is worse than no summary, so prefer tools where every figure is computed from the data rather than written freely by the model.
- NPS vs CSAT: Which Survey Metric Should You Use? · What each score actually measures, how they're calculated, and when to run one, the other, or both.
- How to Write Survey Questions That Get Honest Answers · Seven concrete ways question wording quietly biases responses, and how to fix each one.
- How Many Survey Responses Do You Need for Reliable Results? · A practical sample size table by audience size, and what to do when you can't hit the number.
- What Is a Good Survey Response Rate, and How Do You Raise It? · How to calculate response rate, why benchmarks mislead, and the changes that actually get more people to answer.
- Likert Scales: How to Design and Analyze Them · 5 or 7 points, whether to include a neutral midpoint, and how to report the results without averaging away the story.
- How to Present Survey Results to Leadership · How to structure a survey report so the people reading it can make a decision, not just look at charts.
- Sample AI report · What the themes, findings and actions look like on a real 144-response survey
SurveyService reads every free-text answer and returns an executive summary, key findings, sentiment, and recommended actions, in the same language as your survey. Try it free.