A/B Testing Drip Emails: A Practical Guide

A/B testing drip campaigns is both simpler and more complex than testing one-off emails. Simpler because you have ongoing traffic to test against. More complex because you're testing sequences, not single sends. Here's how to do it right.

Why Test Drip Campaigns?

Drip campaigns run continuously, often for months or years. Small improvements compound significantly:

• 10% improvement in open rate means 10% more people see your message
• 20% improvement in conversion rate could double your revenue from that sequence
• Reducing unsubscribes preserves subscribers for future campaigns

The effort to run a test once pays dividends indefinitely.

What to Test (In Priority Order)

1. Subject Lines

Highest impact, easiest to test. Subject lines determine open rates, which affect everything downstream.

Test variations like:

• Question vs. statement
• Benefit-focused vs. curiosity-driven
• Short (3-5 words) vs. descriptive (8-12 words)
• With vs. without personalization
• Urgency vs. no urgency

2. Send Timing

Test both:

• Time of day: Morning vs. afternoon vs. evening
• Day of week: Tuesday vs. Thursday, weekday vs. weekend
• Sequence timing: 2 days between emails vs. 4 days

3. Email Content

Test one element at a time:

• Long-form vs. short-form copy
• Plain text vs. designed HTML
• Single CTA vs. multiple links
• With images vs. text-only
• Different value propositions or angles

4. Call-to-Action

• Button text ("Start free trial" vs. "Get started")
• Button color and size
• CTA placement (early vs. end of email)
• Single CTA vs. repeated CTA

5. Sequence Structure

Bigger tests, but potentially bigger impact:

• 5 emails vs. 7 emails
• Different email order
• Adding or removing specific emails
• Branching logic vs. linear sequence

How to Run an A/B Test

Step 1: Form a Hypothesis

Don't test randomly. Have a reason:

• "I believe shorter subject lines will increase open rates because our audience checks email on mobile"
• "I believe adding urgency to the trial ending email will increase conversions"
• "I believe reducing sequence frequency will decrease unsubscribes without hurting conversions"

Step 2: Choose Your Metric

Pick one primary metric to determine the winner:

• Subject line tests: Open rate
• Content tests: Click rate or conversion rate
• Sequence tests: Overall conversion rate
• Timing tests: Open rate or conversion rate

Track secondary metrics to ensure you don't win on one metric while losing on another.

Step 3: Calculate Sample Size

You need enough data for statistical significance. Use a sample size calculator:

• Enter your current conversion rate
• Enter the minimum improvement you want to detect
• Enter your desired confidence level (usually 95%)
• The calculator tells you how many subscribers you need per variant

For most drip campaign tests, you'll need 200-1,000 subscribers per variant, depending on your current conversion rates.

Step 4: Randomize Properly

Split your audience randomly at enrollment, not at send time. This ensures each variant gets a representative sample.

Don't split by time ("Everyone this week gets A, next week gets B"). External factors could skew results.

Step 5: Run Until Significant

Resist the temptation to call the winner early. You need:

• Sufficient sample size reached for both variants
• Statistical significance (p-value below 0.05)
• Stable results (not fluctuating day to day)

Most sequence tests need 2-4 weeks to reach significance.

Step 6: Implement and Iterate

Once you have a winner:

• Implement the winning variant for all subscribers
• Document what you learned
• Plan your next test

Testing Drip Sequences vs. Single Emails

The Complexity

Drip sequences add complexity because:

• Subscribers can exit at various points (convert, unsubscribe, complete)
• Early emails affect performance of later emails
• Overall conversion matters more than individual email metrics
• Tests take longer to run (subscribers need to complete the sequence)

Strategies for Sequence Testing

Option 1: Test individual emails. Replace one email at a time. Measure its direct metrics plus impact on overall sequence conversion.

Option 2: Test entire sequences. Run two complete sequence variants in parallel. More definitive results, but requires more traffic and time.

Option 3: Test structural changes. Keep emails the same but test timing, order, or which emails to include.

Common Testing Mistakes

Testing Too Many Things

Change one variable at a time. Testing a new subject line AND new content AND new timing simultaneously means you won't know what caused any improvement.

Stopping Too Early

"Variant B is up 15% after 100 subscribers!" Statistical noise can create temporary winners. Wait for significance.

Ignoring Seasonality

Running a test during Black Friday and applying results year-round. Ensure your test period represents typical conditions.

Testing Trivial Differences

Button color rarely matters as much as the words on the button or the value proposition in the email. Test things that could make meaningful differences.

Not Documenting

Running tests without recording hypotheses, results, and learnings. You'll repeat mistakes and miss patterns.

What Good Testing Looks Like

Example Testing Roadmap

Month 1: Test subject lines for email 1 (high-traffic email)

Month 2: Test send timing for entire sequence

Month 3: Test content approach for key conversion email

Month 4: Test sequence length (5 emails vs. 7)

Each test builds on previous learnings. Document everything.

Example Test Documentation

• Test: Trial expiring subject line
• Hypothesis: Adding specific time frame will increase urgency and opens
• Variants: A: "Your trial ends soon" B: "Your trial ends in 24 hours"
• Metric: Open rate (primary), conversion rate (secondary)
• Sample: 500 per variant
• Duration: 3 weeks
• Result: B won with 34% vs 28% open rate (stat sig at p=0.03). Conversion also higher (8% vs 6%)
• Action: Implemented B. Next test: subject line for email 3.

Tools for A/B Testing

Most drip platforms include A/B testing. Key features to look for:

• Automatic traffic splitting
• Statistical significance calculation
• Automatic winner selection
• Ability to test sequence variations, not just single emails

Sequenzy for Testing

Sequenzy includes A/B testing designed for SaaS drip campaigns. Test subject lines, content variations, and sequence timing with automatic significance calculation. Track impact on trial conversion and MRR, not just opens and clicks.