The Razzly Inquiry: Qualitative Benchmarks for the Journey's Unscripted Dialogue

User journeys are rarely linear. Even the most carefully scripted flows encounter moments where the user says something unexpected, asks a question the designer didn't anticipate, or takes an action that falls outside the happy path. These unscripted dialogues are where trust is built or broken. Yet most teams lack qualitative benchmarks to evaluate them. We wrote this guide for product managers, conversation designers, and UX researchers who want to move beyond surface-level metrics like completion rate or CSAT and start assessing the actual quality of those critical exchanges.

Over the past few years, we've observed a shift: teams are realizing that scripted dialogues—those carefully written branches—are only as good as their ability to handle the unscripted. The question is no longer whether to script, but how to benchmark the unscripted parts meaningfully. This piece offers a practical framework for that inquiry.

Who Must Choose and By When

The decision to invest in qualitative benchmarks for unscripted dialogue isn't abstract. It arises at specific moments in a product's lifecycle. Typically, the trigger is a pattern of user frustration that doesn't show up in standard metrics. For example, a team might notice that users who reach a certain step in onboarding frequently type "I don't understand" or "what does this mean" into a chat widget. The script handles those phrases with a generic fallback, but the fallback itself generates more confusion. That's the moment to act.

The Decision Window

The window for making this choice is often narrow. If you wait until the next quarterly planning cycle, you risk accumulating negative sentiment that erodes retention. On the other hand, rushing to implement benchmarks without a clear framework can lead to superficial metrics that don't capture real quality. The sweet spot is when you have enough data to see a pattern but not so much that the pattern has become a crisis.

Who owns the decision? In our experience, it's rarely a single role. Product managers own the roadmap, but conversation designers or UX researchers often have the deepest insight into what's broken. The most effective teams form a small working group—three to five people—that meets weekly for a month to define benchmarks, collect sample dialogues, and test their criteria against real interactions. This group should include someone who can implement changes to the script (a developer or conversation designer) and someone who can evaluate the user's emotional response (a UX researcher or support lead).

The timeline matters. We recommend allocating four to six weeks for the initial benchmark definition and validation. That might sound slow, but rushing leads to criteria that look good in a spreadsheet but fail in practice. The output of this phase is a set of qualitative benchmarks that the team can apply consistently to unscripted dialogues across the journey. After that, the benchmarks become part of the regular review cycle—monthly or quarterly, depending on the volume of interactions.

One common mistake is trying to benchmark everything at once. Start with the highest-impact touchpoint: the moment where unscripted dialogue most frequently derails the journey. For many products, that's the first support interaction after a failed transaction, or the onboarding step where users typically get stuck. Focus your working group's energy on that one flow. Once the benchmarks prove useful there, expand to other touchpoints.

The Option Landscape: Three Approaches to Scripting Unscripted Dialogue

Teams have multiple ways to handle unscripted dialogue, and the choice of approach directly affects what benchmarks matter. We've seen three dominant strategies, each with distinct trade-offs. None is universally superior; the right choice depends on your team's capacity, the complexity of user intents, and the tolerance for risk.

Approach 1: Full Scripting with Intent Mapping

This is the traditional approach: map every likely user intent to a scripted response, and use NLP or keyword matching to route the user. The script covers the most common paths, and anything unrecognized falls to a human agent or a polite apology. The strength is predictability—every response is designed, tested, and consistent. The weakness is coverage. Even with dozens of intents, users will find gaps, and the script's fallback becomes a bottleneck. Benchmarks for this approach should focus on fallback rate, escalation quality, and the clarity of the script when it does match.

Approach 2: Generative Response with Guardrails

More teams are experimenting with generative AI to produce responses on the fly, guided by guardrails that prevent harmful or off-brand output. This approach can handle a vast range of unscripted inputs, but it introduces variability. The same user question might get two different but equally valid responses, which can confuse users who repeat themselves. Benchmarks here must assess consistency of tone, factual accuracy, and the system's ability to stay within its guardrails. We've seen teams combine this with a small set of scripted fallbacks for high-risk topics like billing or account security.

Approach 3: Hybrid with Escalation Paths

The hybrid model uses scripting for the most common and most critical intents, and generative responses for low-risk, exploratory dialogue. When the system's confidence drops below a threshold, it escalates to a human agent—but the handoff is scripted to preserve context. This approach balances coverage with control. The benchmark challenge is evaluating the handoff itself: does the user have to repeat themselves? Is the tone consistent? We've found that the hybrid model often performs best for complex journeys, but it requires the most maintenance because the boundary between scripted and generative must be continuously tuned.

Each approach implies different benchmark priorities. The full-scripting team cares about fallback rate and script completeness. The generative team cares about response quality and safety. The hybrid team cares about handoff smoothness and threshold calibration. In the next section, we'll define the criteria that apply across all three, plus the ones that are approach-specific.

Comparison Criteria Readers Should Use

To evaluate unscripted dialogue quality, you need criteria that are specific enough to guide improvement but flexible enough to apply across different contexts. We've developed a set of qualitative benchmarks based on patterns we've observed in dozens of product teams. These are not statistical measures; they're judgment-based assessments that a team can apply consistently after calibration.

Clarity of Response

The most basic benchmark: does the response make sense in context? A clear response acknowledges the user's input, addresses the intent, and uses language the user can understand. We recommend rating each response on a simple 1–3 scale: 1 = confusing or irrelevant, 2 = acceptable but could be clearer, 3 = clear and directly helpful. A benchmark score of 2.5 or higher across a sample of 50 unscripted dialogues is a good target for most teams.

Empathy and Tone

Unscripted dialogues often happen when the user is frustrated or confused. The tone of the response matters as much as the content. A response that is technically correct but cold can escalate frustration. We benchmark for tone by asking: does the response acknowledge the user's emotion? Does it avoid blame? Does it use language that feels human, not robotic? Teams can calibrate by reviewing a set of sample responses together and agreeing on what constitutes an empathetic tone for their brand.

Resolution Efficiency

Does the dialogue move the user toward resolution, or does it create loops? An efficient response answers the question or provides a clear next step. An inefficient response asks clarifying questions that the user already answered, or offers information the user didn't ask for. We benchmark efficiency by tracking the number of turns needed to reach a resolution—or to reach a human handoff. A benchmark of three turns or fewer for common intents is a reasonable starting point, but adjust based on complexity.

Fallback Handling

When the system doesn't understand, how does it respond? A good fallback admits uncertainty, offers options, and doesn't make the user feel foolish. A poor fallback repeats the same apology or sends the user in circles. We benchmark fallback quality by reviewing a sample of fallback interactions and rating them on a scale from 1 (frustrating) to 3 (helpful despite the failure). Aim for an average of 2 or higher.

Consistency Across Sessions

If a user returns to the same dialogue later, do they get a similar experience? Inconsistency can erode trust. We benchmark consistency by comparing responses to the same intent across different sessions or users. This is especially important for generative approaches. A benchmark of 80% consistency (same intent yields responses that are functionally equivalent) is a realistic target.

Approach-Specific Criteria

• For full scripting: Intent coverage (what percentage of user inputs match a known intent) and script freshness (how recently each script was reviewed).
• For generative: Safety violation rate (how often the system produces harmful or off-brand content) and factual accuracy (verified against a knowledge base).
• For hybrid: Handoff satisfaction (does the user feel the handoff was smooth?) and threshold appropriateness (are users being escalated too early or too late?).

These criteria are not exhaustive, but they give teams a starting point. The key is to apply them consistently across a representative sample of dialogues, not just the ones that went well. In the next section, we'll compare these criteria in a structured way.

Trade-Offs Table: Benchmark Priorities by Approach

Different scripting approaches prioritize different benchmarks. The table below maps each approach to the criteria that matter most, along with the trade-offs you'll face.

When to Prioritize One Over Another

If your team has limited resources for maintenance, full scripting with a strong fallback is often the safest bet. The benchmarks are straightforward, and the failure modes are well understood. If your user base is diverse and your product handles many edge cases, generative approaches can scale better—but they require investment in guardrails and monitoring. The hybrid model works well for products with a mix of simple and complex intents, but it demands a team that can continuously refine the boundary.

A common mistake is choosing an approach based on hype rather than fit. We've seen teams adopt generative AI because it's trendy, only to find that their users ask the same three questions repeatedly—questions that could be handled with a simple script. Conversely, teams that stick with full scripting for a complex product often end up with a bloated intent map that's hard to maintain. The decision should be driven by the patterns in your unscripted dialogue, not by what's new.

Composite Scenario: The Onboarding Flow

Consider a typical onboarding flow for a SaaS product. The user is asked to configure a setting they don't understand. They type "what does this do?" into a help widget. In a full-scripting approach, the system might have an intent for "what does this setting do" and provide a canned explanation. If the user then asks "but why would I want to enable it?", the script might not have that intent, and the user gets a fallback. In a generative approach, the system could answer both questions fluidly, but might accidentally provide incorrect information about a third-party integration. In a hybrid approach, the first question is scripted, and the follow-up is generative—but the handoff must be seamless. The benchmarks for this scenario would include clarity of the initial response, empathy if the user is frustrated, and resolution efficiency (how many turns to get the user back on track).

Implementation Path After the Choice

Once you've chosen an approach and defined your benchmarks, the next step is implementation. This is where many teams stall, because qualitative benchmarks require human judgment, which is harder to scale than automated metrics. We recommend a phased approach.

Phase 1: Calibration

Gather a sample of 30–50 unscripted dialogues from your chosen touchpoint. Have your working group independently rate each dialogue against the benchmarks (clarity, empathy, efficiency, etc.). Then compare ratings and discuss disagreements. This calibration process is essential to ensure consistent application. Expect to spend two to three sessions getting aligned. The output is a shared understanding of what each benchmark level looks like in practice.

Phase 2: Baseline Measurement

Once calibrated, apply the benchmarks to a larger sample—at least 200 dialogues, if available. Calculate averages for each benchmark and identify patterns. Which intents score lowest? Which parts of the dialogue consistently fail on empathy? This baseline gives you a starting point for improvement. It also helps you set realistic targets. If your current clarity score is 1.8, aiming for 3.0 overnight is unrealistic. A target of 2.2 in the next quarter is more achievable.

Phase 3: Iterative Improvement

Based on the baseline, prioritize the worst-performing areas. If fallback handling is the biggest weakness, rewrite the fallback scripts or adjust the generative guardrails. If empathy is low, train the team (or the model) on tone. After each change, re-sample and re-benchmark. We recommend a cycle of two weeks: one week to implement changes, one week to collect new dialogues and measure impact. After three cycles, you should see meaningful improvement.

Phase 4: Integration into Regular Review

Once the benchmarks are stable and improvement is consistent, integrate them into your regular review process. This could be a monthly review where the team looks at a random sample of dialogues and scores them. The benchmarks become a leading indicator of user satisfaction, complementing lagging metrics like retention or support ticket volume. Over time, you can refine the benchmarks based on what correlates with user outcomes.

A pitfall to avoid: don't let the benchmarks become a checkbox exercise. If the team stops caring about the qualitative judgment and just assigns scores mechanically, the benchmarks lose their value. Rotate the raters periodically and revisit the calibration if scores drift.

Risks If You Choose Wrong or Skip Steps

The biggest risk is not choosing an approach at all. Teams that ignore unscripted dialogue often see a gradual decline in user trust—users feel unheard, and they churn. But even teams that actively work on it can make mistakes that undermine their efforts.

Risk 1: Over-Optimizing for One Benchmark

If you focus too heavily on clarity, you might produce responses that are clear but cold, sacrificing empathy. If you focus on empathy, you might produce verbose responses that frustrate users who want a quick answer. The benchmarks are a portfolio; no single metric should dominate. We've seen teams celebrate a high clarity score while their empathy score dropped, and users complained about feeling "handled." Balance is key.

Risk 2: Ignoring the Handoff

In hybrid approaches, the handoff to a human agent is a critical moment. If the handoff is clunky—if the user has to repeat themselves, or if the agent doesn't have context—the entire dialogue quality is undermined. We've seen teams spend months perfecting their scripted and generative responses, only to lose users at the handoff. Benchmark the handoff separately, and treat it as a first-class citizen.

Risk 3: Skipping Calibration

Without calibration, benchmark scores are meaningless. Two raters might assign very different scores to the same dialogue, leading to false signals. We've seen teams skip calibration in the interest of speed, only to realize later that their "improvement" was just rater drift. Invest the time upfront; it pays off in reliable data.

Risk 4: Choosing the Wrong Approach for Your Data

If you choose full scripting but your users ask a wide variety of questions, you'll have a high fallback rate and frustrated users. If you choose generative but your product handles sensitive data (like health or finance), you risk safety violations. The approach must fit the nature of your unscripted dialogue. We recommend auditing a sample of dialogues before deciding. Categorize them: how many are simple FAQs? How many are complex, multi-turn issues? How many involve sensitive topics? The distribution should guide your choice.

Risk 5: Not Iterating on Benchmarks

Benchmarks are not static. As your product evolves, user questions change. What was a clear response six months ago might now be outdated. We recommend revisiting your benchmarks every quarter to ensure they still reflect quality. If you notice that scores are consistently high but user satisfaction is flat, the benchmarks may have lost their edge. Update them based on new patterns.

Mini-FAQ: Common Questions About Qualitative Benchmarks

How many dialogues do we need to sample?

There's no magic number, but we've found that 30–50 dialogues per touchpoint is enough for calibration, and 200+ gives a reliable baseline. If your volume is low, you can sample over a longer period. The key is consistency: sample the same way each time.

Can we automate the benchmarking?

Partially. Some aspects, like clarity and empathy, are inherently subjective and benefit from human judgment. However, you can use NLP to flag dialogues that are likely low-quality (e.g., those with repeated fallbacks or long resolution times) and prioritize them for human review. Automating the entire benchmark risks missing nuance.

What if our team is too small for a working group?

Even a team of two can do this. One person rates the dialogues, and the other reviews a subset for consistency. The calibration step is still important, but it can be done with just two people discussing disagreements. If you're a solo practitioner, consider involving a colleague from support or product who interacts with users regularly.

How do we handle dialogues in multiple languages?

Benchmarks should be language-specific. A response that is clear in English might be confusing in Spanish due to translation quality. We recommend having native speakers for each language you support, or at least a translation review step. The same criteria apply, but the calibration must be done per language.

What's the difference between a benchmark and a metric?

A metric is a quantitative measure (e.g., completion rate). A benchmark is a qualitative standard (e.g., clarity score of 2.5). Benchmarks require human judgment to apply, while metrics are often automated. Both are useful, but benchmarks capture aspects of quality that metrics miss. We recommend using both: metrics for scale, benchmarks for depth.

How often should we re-calibrate?

Calibrate at the start of each major review cycle, or whenever you add a new rater. If you notice scores drifting (e.g., one rater consistently gives higher scores than others), re-calibrate immediately. We recommend a brief calibration session every quarter, even if nothing has changed, to reinforce consistency.

Recommendation Recap Without Hype

Qualitative benchmarks for unscripted dialogue are not a silver bullet. They require effort to define, calibrate, and maintain. But for teams that invest in them, they provide a window into the user's experience that raw metrics cannot. Here's a recap of the key actions we recommend:

• Start small. Pick one high-impact touchpoint and define 3–5 benchmarks that matter most for that flow. Don't try to cover everything at once.
• Calibrate before you measure. Gather your working group, rate sample dialogues together, and align on what good looks like. This step is non-negotiable.
• Choose your approach based on data. Audit your unscripted dialogues to understand the distribution of intents and complexity. Let that guide whether you go full scripting, generative, or hybrid.
• Iterate in short cycles. Implement changes, re-sample, and re-benchmark every two weeks. Look for improvement in the weakest areas first.
• Integrate benchmarks into your regular review. Make them a monthly habit. Use them as a leading indicator, not a post-mortem.
• Revisit your benchmarks quarterly. As your product and user base evolve, your benchmarks should too. Don't let them become stale.

The unscripted moments in a user journey are where the relationship with your product is tested. By applying thoughtful qualitative benchmarks, you can turn those moments from points of failure into opportunities to build trust. The work is ongoing, but the payoff is a dialogue that feels less like a script and more like a conversation—one that users actually want to have.