Why We Looked Back
At some point after we had been running for about a year, we had accumulated enough customer calls to ask a question that had been nagging at us: if we had been watching our call data more carefully from the beginning, what would we have built differently?
We went through every recorded call from our first 50 customers. Not to evaluate how the calls went or to grade our discovery process. We were looking for one specific thing: product signals that were present in the calls but never made it into the roadmap, and whether those gaps eventually showed up as problems, renewals at risk, or churn.
What we found was not surprising in the sense that we predicted it. It was surprising in the scale of it.
What We Found
The clearest pattern was integration requests. Across our first 50 customers, mentions of a specific data export workflow came up in 11 different calls, spread across 9 distinct accounts, over a period of about four months. None of us had noticed that pattern while it was happening. When we went back and tagged every call transcript for that theme, the frequency was obvious. Four of those nine accounts had either churned or significantly reduced usage by the time we did this analysis.
We are not saying that building the export feature would have definitely retained those four accounts. We do not know that. What we know is that we never had the conversation with those accounts about our roadmap for it, because we did not know the pattern existed. The signal was there. We were not looking at it in a way that made it visible.
The second pattern was more subtle. A segment of customers, roughly a third of our first 50, had a specific workflow that they described in their onboarding calls but never brought up again in subsequent calls. When we looked at what happened to those accounts, the ones who stopped talking about that workflow were, on average, lower-engagement than the ones who kept bringing it up. The hypothesis we formed from that: customers who go quiet about a workflow gap have either found a workaround or have stopped trying to use that part of the product. Both are concerning.
The Prioritization Lessons
Going through 50 calls with a retrospective lens taught us a few things about how we had been making prioritization decisions that we would do differently now.
First, we had been weighting recency too heavily. The feature request from the customer we spoke to last Thursday felt more urgent than the pattern we would have seen if we had looked at the last three months of calls. The loudest recent signal beat the most frequent historical signal, repeatedly. That is a bias that is easy to correct for when you have frequency data in front of you, and nearly impossible to correct for when you do not.
Second, we had been treating each call as an independent data point. We would note a request, add it to a backlog item, and move on. We were not asking how many other accounts had said something similar. We did not have a quick way to answer that question. So the backlog reflected individual requests, not population-level patterns. Some items on the backlog had been mentioned by one account. Others had been mentioned by six. They looked similar in the backlog because we had not been counting.
Third, we had underestimated how much signal was in the language customers used to describe their workarounds. When a customer says "we built a small script to handle this," they are telling you something very specific: they found your product insufficient for that workflow, the need was real enough that they invested time building around it, and they have probably stopped expecting you to fix it. Workaround language in transcripts is one of the clearest indicators of a product gap that has been accepted rather than escalated.
What Changed After This Exercise
We changed three things in how we work after doing this retrospective.
We started tagging every call transcript with a feature-area and friction-type schema within 24 hours of the call. Not a detailed summary, just the tags. Feature area: export, filtering, integrations, reporting, notifications. Friction type: missing functionality, performance, confusing behavior, workflow gap. Two-minute maximum per call.
We started doing a weekly 15-minute review of the previous week's tags against account health. Not a full analysis, just a scan: are any tags accumulating for the same account? Is any feature area showing up across multiple accounts this week? That review surfaces the frequency signal before it becomes a pattern we only notice in retrospect.
We started tracking explicitly when accounts go quiet on a topic they had previously raised. That is a harder practice to build because it requires looking at account history, not just the most recent call. But it has become one of the most useful things we do for catching early churn signal.
The Limits of This Exercise
Looking back at 50 calls is not the same as having a process that surfaces signal in real time. The retrospective was valuable for calibrating our thinking and changing our practices. It was not a substitute for building a system that makes frequency visible on an ongoing basis.
We are also aware that what we learned from our first 50 customers is not perfectly generalizable. Different customer segments surface different signal types. The patterns we found might not apply to a product team working in a different product category or with a different buyer profile. The method, looking at call data longitudinally and counting frequency, transfers. The specific findings are ours.
The exercise also required us to be honest about signals we had heard and consciously deferred. Some of the backlog gaps we found in this retrospective were not invisible. We had heard them and decided not to act. In a few cases, that was the right call, the feature genuinely was not worth the development cost at that time. In other cases, we had made that assessment without knowing that six accounts, not one, would eventually need the same thing. Frequency changes the calculation. We did not have the frequency data when we made those decisions, and we should have.
