Correlation Is Not Conversion: The Problem of Attributing Business Results to AI

Three months after deploying an AI booking assistant across a hospitality client’s website, the stakeholder asks a reasonable question: Did the chatbot increase our bookings?

Your analytics pipeline has a number ready. Users who interacted with the chatbot booked at 11.2%. Users who never opened it booked at 3.1%. That looks like an 8.1 percentage point lift. Ship the slide deck. Schedule the quarterly review.

That number is structurally meaningless. Not approximately wrong. Not slightly optimistic. It conflates who uses chatbots with what chatbots cause, and those are entirely different questions.

1. The Metric Everyone Reports Is Confounded

The standard enterprise pitch slide reads:

“Users who engaged with our AI assistant converted at 3.4x the rate of those who did not.”

This framing treats chatbot interaction as a treatment and compares outcomes across self-selected groups. The problem is immediate: users who voluntarily open a chatbot and type a question about ocean-view suites for December 23–26 are not the same population as users who bounced after skimming the homepage for four seconds.

One group arrived with intent. Dates in mind, credit card nearby, comparison-shopping across three hotel tabs. The other group was killing time on a Tuesday afternoon.

Of course the first group converts more. They would have converted without the chatbot. The chatbot happened to be in their path, but so was the footer, the cookie banner, and the favicon.

This is selection bias. The users who choose to interact with the chatbot and the users who ignore it differ systematically before the interaction begins. Comparing their conversion rates tells you about who opens chatbots. It tells you nothing about what chatbots cause.

2. A Worked Example: Manufacturing a Phantom Lift

Numbers make this concrete. Consider a hotel website receiving 10,000 monthly visitors. Two segments arrive:

• 2,000 high-intent visitors searched specific dates, clicked through from a metasearch engine, and convert at 15% regardless of whether the chatbot exists.
• 8,000 casual browsers arrived from organic or social traffic. They convert at 1%.

High-intent visitors are far more likely to engage with the chatbot. They have specific questions (parking rates, pool hours, cancellation policies). Casual browsers rarely bother.

Now compute conversion rates for each group.

Chatbot users (2,000 total):

Non-users (8,000 total):

The dashboard shows a 3.9x multiplier. Seven percentage points of “impact.” Your slides look phenomenal.

The true causal effect of the chatbot: zero. Within each segment, conversion rates are identical whether or not the user engaged with the assistant. The reported lift is entirely manufactured by the compositional difference between the two groups. High-intent users dominate the chatbot group; casual browsers dominate the non-chatbot group.

This is Simpson’s paradox applied to AI attribution. The aggregate number reverses what the segment-level data actually shows.

3. The Core Problem: You Cannot Observe the Counterfactual

The question “did the chatbot increase bookings?” is really asking: Would this specific user have booked if the chatbot did not exist?

For each visitor $i$, there are two potential outcomes:

• $Y_i(1)$: whether they would book if they interact with the chatbot
• $Y_i(0)$: whether they would book if the chatbot does not exist

The causal effect for that visitor is $\tau_i = Y_i(1) - Y_i(0)$. The number you actually want is the Average Treatment Effect:

Here is the fundamental difficulty. For each visitor, you observe exactly one of these two outcomes. Never both. The counterfactual is permanently unobservable. You cannot rewind time and show the same user the same website without the chatbot and see what they would have done.

The naive dashboard metric computes:

This equals the ATE only if the two groups are identical in every relevant characteristic before the chatbot interaction. They are not. The naive metric conflates the real treatment effect with selection bias:

That second term captures the baseline conversion gap between people who choose to engage with chatbots and people who do not. In the worked example, this term is large and positive. The naive metric overstates the chatbot’s impact by the full magnitude of the selection bias. In many production deployments, the selection bias term is larger than the actual treatment effect.

4. The Cleanest Available Tool: Randomized Holdout Experiments

Randomization eliminates selection bias by construction. Randomly assign 50% of visitors to see the chatbot widget. The other 50% see the website without it. The two groups are, in expectation, identical in intent, traffic source, device, session history, and every other confounder you can and cannot measure.

The conversion difference between groups is an unbiased estimate of the ATE. This is the closest thing to a clean answer that exists.

The Power Problem at Low Conversion Rates

Hotel booking conversions typically sit between 2% and 5%. Detecting a meaningful lift at these base rates requires traffic that most individual properties cannot produce quickly.

The minimum sample size per group for a two-proportion z-test:

import math
from scipy import stats

def required_sample_size(
    baseline_rate: float,
    min_detectable_effect: float,
    alpha: float = 0.05,
    power: float = 0.80,
) -> int:
    """
    Sample size per group for a two-proportion z-test.

    baseline_rate:          conversion rate without chatbot (e.g., 0.032)
    min_detectable_effect:  absolute lift to detect (e.g., 0.008)
    """
    p1 = baseline_rate
    p2 = baseline_rate + min_detectable_effect

    z_a = stats.norm.ppf(1 - alpha / 2)
    z_b = stats.norm.ppf(power)

    num = (z_a + z_b) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))
    den = (p2 - p1) ** 2

    return math.ceil(num / den)

# Hospitality scenario: 3.2% baseline, detect a 0.8pp lift
n = required_sample_size(0.032, 0.008)
print(f"Visitors per group: {n:,}")
print(f"Total experiment:   {2 * n:,}")

At a 3.2% baseline, detecting a 0.8 percentage point lift requires roughly 7,500 visitors per group. That is 15,000 total.

A mid-size hotel at 4,000 monthly visitors needs 2–3 months. A boutique property at 800 visitors per month needs over a year. For a platform serving 3,500 tenants with wildly varying traffic volumes, per-tenant A/B tests are statistically infeasible for the long tail.

Why Teams Skip This Anyway

Even when sample sizes are achievable, controlled experiments face organizational friction:

1. Client resistance: “You want to hide the chatbot from half my visitors? I am paying for this.” Hard to argue against when the client’s contract covers full deployment.
2. Novelty effects: Early chatbot engagement is inflated by curiosity. Users click the new widget because it is new. Measuring a stable effect requires 4–6 weeks minimum for novelty to decay.
3. Seasonality: A two-month test spanning July and August captures peak travel. The same test in January captures off-peak. The treatment effect may genuinely vary across seasons.

None of these invalidate the method. They just make it expensive and politically inconvenient. Which is why most teams skip it and ship the confounded metric instead.

5. Quasi-Experimental Methods When Randomization Fails

If you cannot run a controlled holdout, you can extract weaker (but still informative) estimates from observational data.

5.1 Difference-in-Differences (DiD)

If you are rolling out the chatbot to tenants in waves (a natural pattern when onboarding 3,500 properties), you have a built-in quasi-experiment. Tenants who received the chatbot in wave 1 are the treatment group. Tenants still waiting are the control.

DiD subtracts the control group’s background trend from the treatment group’s observed change:

def difference_in_differences(
    treat_pre: float,
    treat_post: float,
    control_pre: float,
    control_post: float,
) -> dict:
    """
    Two-period, two-group DiD estimator.
    All inputs are conversion rates (0.0 to 1.0).
    """
    treat_change = treat_post - treat_pre
    control_change = control_post - control_pre
    effect = treat_change - control_change

    return {
        "treatment_change": treat_change,
        "control_change": control_change,
        "causal_estimate": effect,
    }

# Wave 1 hotels launched in June. Wave 2 is still waiting.
result = difference_in_differences(
    treat_pre=0.031,    # Wave 1: 3.1% in May (pre-launch)
    treat_post=0.041,   # Wave 1: 4.1% in August (post-launch)
    control_pre=0.029,  # Wave 2: 2.9% in May
    control_post=0.035, # Wave 2: 3.5% in August
)
print(f"Treatment change:    +{result['treatment_change']*100:.1f} pp")
print(f"Control change:      +{result['control_change']*100:.1f} pp")
print(f"Estimated AI effect: +{result['causal_estimate']*100:.1f} pp")
# Treatment change:    +1.0 pp
# Control change:      +0.6 pp
# Estimated AI effect: +0.4 pp

The control group’s conversion also rose 0.6 percentage points (summer travel demand). A naive before/after comparison would claim a 1.0 pp lift. DiD correctly attributes only 0.4 pp to the chatbot.

The critical assumption: Parallel trends. Without the chatbot, both groups would have followed the same booking trajectory. Validate this by checking pre-period data. If the groups were already diverging before launch, DiD gives you nothing.

5.2 Propensity Score Matching

For session-level analysis, match each chatbot user to a statistically similar non-user based on observable covariates: traffic source, device type, landing page, pages viewed before chat, time of day, day of week.

This reduces selection bias on observable characteristics. It cannot address unobserved confounders (purchase urgency, prior brand familiarity, whether the user already has a hotel tab open on another site). Strictly weaker than randomization.

5.3 Interrupted Time Series

For a single tenant, model the pre-launch booking trend, project it forward, and compare projection to actual post-launch bookings. This is the weakest method. Any concurrent change (marketing campaign, competitor closure, a viral review) confounds the estimate. Treat it as a rough sanity check only.

6. What You Can Measure Honestly

When causal attribution is infeasible, report metrics that measure operational effectiveness without overclaiming revenue causation.

Task Completion Rate. What fraction of chatbot conversations ended with the user’s question answered? This measures whether the assistant is functional. It does not measure revenue impact.

Containment Rate. What fraction of queries that would have required a phone call or front desk email were handled entirely by the chatbot? This is a direct cost-savings metric. If your client’s front desk handles 200 calls per week and the chatbot absorbs 120 of them at $0.03 per conversation vs. $4.50 per phone call, the savings are measurable and genuinely attributable. The call either happened or it did not.

Micro-Conversion Events. Track discrete actions within the chatbot session:

• Clicked a booking link provided by the assistant
• Viewed a room detail page after receiving a recommendation
• Added dates to a booking widget within 60 seconds of a chatbot response

These are behavioral engagement signals. They show the assistant influenced the user’s navigation path. They do not prove end-to-end booking attribution.

What to tell the client:

“82% of chatbot conversations reached resolution. The assistant handled 67% of routine inquiry volume that previously required staff, saving an estimated $2,100 per month. Users who received a booking link from the chatbot clicked through 23% of the time. We observe positive correlation between chat engagement and booking completion. Establishing the precise causal revenue impact requires a controlled holdout test, which we recommend running during your next low-season window.”

That separates what was measured from what was inferred. It quantifies cost savings (which are directly attributable: the call either happened or the chatbot handled it) and flags the correlation without calling it causation.

7. The Honest Position

This article does not wrap up with a tidy solution. The attribution problem for AI assistants is genuinely unsolved in most production deployments.

A/B testing solves it in theory. Statistical power requirements, client politics, seasonality, and novelty effects make it impractical for most individual tenants. DiD works when your rollout schedule gives you a natural control group, but the parallel trends assumption is fragile and unverifiable. Propensity matching reduces bias without eliminating it. Everything weaker than randomization carries asterisks.

What remains true regardless of method:

1. The naive chatbot-vs-no-chatbot conversion comparison is not evidence of impact. It is evidence of selection bias. Reporting it as “lift” or “AI-driven revenue” is a measurement error, not a finding.

2. Cost savings from call deflection are genuinely attributable. A phone call that did not happen because the chatbot answered the question is a real, measurable business outcome. This is the cleanest attribution path available for most AI assistant deployments.

3. Sometimes the right answer is “we do not know yet.” Telling a client that you cannot attribute bookings to the chatbot without a controlled experiment is uncomfortable. It is also honest. And honest measurement, even when it produces smaller or less definitive numbers, is more durable than a confounded metric that inflates on the slide deck and collapses under scrutiny.

The question “did the AI increase bookings?” is a causal question. Answering causal questions requires either experimental design or explicit assumptions that you can defend. Dashboards do not produce causal answers. They produce correlations. Knowing the difference is the starting point.