A/B test design for LLM model selection in production

Context

We operate a conversational commerce platform where merchants use an LLM-powered assistant to engage customers and drive sales over messaging channels. The underlying model is a business-relevant lever: a better model can mean more conversions, but it also costs more per message. A new generation of our base model had just become available. On paper it promised stronger reasoning, but "on paper" doesn't pay the bills — before committing to a full migration we needed production evidence of the lift, across real merchants and real conversations.

Problem

A clean A/B on this looks simple until you look closer. Three things made it non-trivial:

• Sparse, heterogeneous conversions: conversions per conversation are rare events, and the highest-volume merchants span different verticals, catalog types, and baseline conversion rates.
• Analysis-unit trap: the natural assignment unit (session, reset after a window without customer reply) is not the natural analysis unit. Two sessions inside the same conversation aren't independent — analyzing at the session level would pseudo-replicate the data and inflate p-values.
• Asymmetric rollout risk: shipping a worse model hurts revenue and merchant trust directly; shipping a better model late is recoverable. The test had to protect the downside.

Methodology

Experimental design:

• Assignment: random 50/50 at conversation-session level via a feature flag, with the full cohort exposed to both variants.
• Unit of analysis: conversation, not session, to avoid pseudo-replication. Conversations that saw both models (mixed) were excluded from analysis.
• Metrics: a sale-mark event as the primary conversion metric (full coverage) and an order-level event as secondary (partial coverage). Average order value by currency as exploratory.
• Statistics: two-proportion z-test for conversion, Mann-Whitney U for AOV (doesn't assume normality on amounts). Conventional significance threshold.
• Window: multi-week production traffic.

Design validation

Before looking at results I verified the test had actually behaved as designed:

• Sticky assignment: the vast majority of conversations stayed on a single model across all their sessions. The rest were excluded from analysis.
• Balanced split: negligible deviation from 50/50 at the aggregate, and every individual merchant within the expected range. No drift.
• Cumulative lift tracking: the lift became significant early in the window and stayed above threshold through the end. Robust result, not a lucky window.

Results

• Double-digit aggregate conversion lift in favor of the candidate model, statistically significant. Similar magnitude on the secondary order-level metric.
• Dramatically larger lift — several times the aggregate — concentrated in merchants using an integrated third-party catalog platform. For merchants on the native catalog, the lift was much smaller and not significant.
• No meaningful change in average order value on any merchant/currency pair. The gain is in sales volume, not ticket size.
• A couple of merchants showed a material negative lift: both on the native catalog, medium volume, with a high baseline conversion rate and heavily customized prompts.

Learnings

• The average hid the real story: the aggregate lift was also a wide spread across merchants, from strongly positive to clearly negative. Segmentation by catalog source explained most of the variance — segmentation by country explained almost none.
• Richer structured catalog data amplifies model capability. Likely mechanism: merchants on an integrated platform handle promotions as structured data; merchants on the native catalog embed promotion rules into free-text prompts, where a more capable model is more sensitive to inconsistency.
• Session-level assignment is fine if you analyze at conversation level. The high sticky rate confirmed the two were near-equivalent in practice, but the methodology choice mattered for p-values — future experiments should move the flag granularity to conversation level.
• Before scaling, cost is the gating decision. A given percentage lift is a different call at 1.5x, 3x, or 10x inference cost. Per-merchant cost tracking is the blocking analysis before any rollout.