A/B testing

Pages built with the current page design (every account since September 26, 2026) have no A/B variants yet: each ad group gets one version. Republishing a campaign rebuilds its pages in that design and retires their old variants. Funnels built with the earlier page builder got a control variant plus at least one experimental variant on first publish, and the ones not yet rebuilt keep testing as described below. Variants are routed at the Cloudflare edge using cookies so each visitor sees a stable experience.

Funnel A/B variants

Variant recipes are per-category. Examples:

• Urgency variant. Aggressive red accents, "Available Now" banners, sticky mobile CTAs.
• Trust variant. Calmer blue accents, prominent licensing badges, review counts.
• Short-form variant. 3 steps instead of 5 (test for emergency intent).

Emergency hybrid landing pages get their own variant recipes (red accent for higher urgency, shorter response time copy, social proof variant).

The Cloudflare Worker reads the visitor's vba_var_* cookie. New visitors get assigned to a variant based on a deterministic hash. AdsBot-Google and Googlebot always see the control to avoid disrupting ad crawls.

Statistical significance

The system uses a two-proportion z-test at 95% confidence to decide when a variant has reliably out-converted the control. Threshold:

• Minimum 100 visitors per variant
• z-score absolute value greater than 1.96
• Lift in either direction over 5 percentage points relative

Until significance, both variants continue receiving traffic. Variants are tagged in the dashboard as Running, Significant, or Winner.

Winner declaration

Two funnel actions the agent recommends on Pro and Max. Both wait for your approval, including on Max with hands-free on:

• declare_ab_winner. When confidence is over 95% AND each variant has more than 100 visitors, approving it sets is_winner=true on the winning variant, deactivates losers, and copies the winning HTML to the base page. New visitors all get the winning experience.
• pause_losing_variant. When a variant's CVR is under 50% of the winning variant's CVR (z-test significant), approving it pauses that variant without yet declaring a final winner. The remaining variants keep testing.

Bandit Lab (Phase 41/42)

Beyond funnel-level A/B tests, VibeAds has element-level Thompson sampling experiments built for individual funnel elements: sticky CTA copy, hero image, step copy template. They are switched off for now, so no experiments are running. Each element has its own arms with Beta posteriors. At runtime, the edge worker samples from the posteriors and serves the winning arm.

Hierarchical priors (Phase 42) seed new arms with fleet-wide category-level evidence so low-traffic campaigns benefit from what worked across other campaigns in the same category. A plumber funnel with 30 visitors a week inherits ~20% of the plumber-category-wide prior, then quickly outweighs it as it accumulates local data.

Bandit Lab is read-only in the dashboard today. The Tracking tab on the campaign detail page (Max tier) shows live arm statistics: impressions, conversions, posterior probability of being best, locked or running status.