A/B Testing Drip Emails: A Practical Guide

For most teams that want to A/B test a drip sequence, Drip is the best place to start, because it is built around workflows that enroll people, wait, branch and send, which is exactly the structure a sequence test needs. Its monthly pricing is based on the number of people in your audience, with the lowest tier at $39 a month for 1 to 2,500 people, billed monthly. Before you commit, though, confirm on Drip's pricing page and documentation that split testing is available on the plan you will use, because the pricing page alone does not settle that question. The rest of this guide explains what to test, how to run the test properly, and how to pick a tool for it.

Why test drip campaigns at all?

A drip campaign runs continuously. The same welcome, onboarding or trial sequence can reach thousands of people over many months, so a small change in one email is applied to every future enrollee. That is the main argument for testing: the payoff of a good result is repeated on each new subscriber, while the cost of a test is paid once.

Three kinds of improvement are worth looking for:

• More opens on the first email, which lifts everything that follows because more people see the rest of the sequence.
• More clicks or completions on the email that carries the offer, which turns the same audience into more signups or purchases.
• Fewer unsubscribes, which keeps people on the list for later campaigns.

Keep the arithmetic honest. A change in open rate does not automatically change revenue, and a lift on one email is only worth the effort if the downstream emails still convert. This is why sequence tests need a primary metric that sits at the end of the sequence, not just at the top.

What to test, in priority order

Start where traffic and impact are highest. The order below follows the path a subscriber takes, from the inbox to the click to the sequence as a whole.

1. Subject lines of the first email

Subject lines decide whether the email gets opened at all, so they are the cheapest test with the broadest effect. Compare variants that differ on one dimension:

• A question versus a statement
• A benefit ("Set up your first workflow in 10 minutes") versus a curiosity hook ("The step most new users skip")
• A short subject (three to five words) versus a descriptive one (eight to twelve words)
• With and without the subscriber's first name
• A time-bound phrase versus a neutral one

Open rate is the right primary metric here, but remember that some mail clients load images or pre-fetch content in ways that distort open counts. Pair the open-rate result with clicks before you trust a winner.

2. Send timing

Timing tests fall into three groups, and each needs its own comparison:

• Time of day: morning against afternoon against evening, for the same email in the same sequence.
• Day of week: a weekday against a weekend send, or two weekdays against each other.
• Sequence spacing: the gap between emails, for example two days against four days. Spacing tests take the longest to read, because a subscriber's full path has to play out.

Timing results are often specific to your audience's time zones and habits, so a winner in one list does not transfer cleanly to another.

3. Email content

Change one element at a time, so that a result can be attributed to that element:

• Long-form against short-form copy
• Plain text against a designed template
• A single call to action against several links
• With images against text-only
• Two different value propositions or angles for the same offer

Content tests usually use click rate or conversion rate as the primary metric, because the open is already decided by the subject line.

4. Call to action

• Button wording, such as "Start your trial" against "Get started"
• Button placement, early in the email against after the main argument
• One call to action repeated against a single one at the end

Button colour and size are rarely the variable that moves results. The words on the button and the promise around it usually matter more.

5. Sequence structure

These are the largest tests and the slowest to read:

• A five-email sequence against a seven-email sequence
• A different order of the same emails
• Adding or removing one specific email
• A branching path, where subscribers who click one link get a different follow-up, against a linear path for everyone

Structure tests can produce the biggest difference, but they also need the most traffic and the longest window. Run them after the cheaper tests have settled the subject lines and copy.

How to run an A/B test on a drip sequence

Step 1: Write a hypothesis

A hypothesis names the change, the expected effect and the reason. "Shorter subject lines will raise open rates because most of our readers check email on a phone" is testable. "Let's see what happens if we change the subject" is not. A written hypothesis also stops you from re-running the same idea six months later without knowing it was already tried.

Step 2: Choose one primary metric

Pick a single metric that decides the winner before the test starts:

• Subject line tests: open rate
• Content and call-to-action tests: click rate or conversion rate
• Sequence tests: conversion rate at the end of the sequence
• Timing tests: open rate or conversion rate, chosen in advance

Also track one or two secondary metrics, such as unsubscribes and complaints. A variant that wins on opens but doubles unsubscribes is not a winner.

Step 3: Size the test

The number of subscribers you need depends on three inputs: your current conversion rate for the metric, the smallest improvement you care about, and the confidence level you want, commonly 95%. Use a sample-size calculator with those three inputs. The practical rule that follows from the maths is that the smaller the lift you want to detect, the more subscribers each variant needs. Detecting a large change in a low-volume sequence is usually possible; detecting a one-point change in a rarely converting email may not be worth the wait.

If the calculator says a test needs more people than you will enrol in a reasonable time, that is useful information. Test a bigger change, test an earlier point in the funnel where more people reach the email, or accept that the test will be directional rather than conclusive.

Step 4: Randomize at enrollment

Split people randomly when they join the sequence, not when each email is sent. Each new subscriber is assigned to variant A or variant B and stays in that branch for the whole sequence. This gives both variants a comparable mix of sign-up sources, devices and time zones.

Avoid splitting by calendar period, such as "everyone who joins this week gets A and next week gets B". Seasonal events, a product launch or a press mention will then fall into one variant only.

Step 5: Run until the result is stable

Do not call a winner early. Before you read the result, check that:

• Both variants have reached the sample size from your calculation
• The difference is statistically significant at your chosen threshold, usually a p-value below 0.05
• The gap has stayed roughly the same for several days rather than swinging with each batch of enrollments

Sequence tests often need several weeks, because the last conversion in a long sequence can come well after the first email is sent. Plan the window around the slowest step in the sequence, not the first.

Step 6: Implement, then iterate

When a variant wins, roll it out to all new enrollees and write down what changed and by how much. Then decide on the next test. Ideally each test builds on the one before it, for example by moving from a winning subject line to a test of the first email's body.

Testing single emails against whole sequences

Drip sequences add complications that one-off sends do not have. Subscribers exit at different points: some convert, some unsubscribe, and some simply finish. Early emails also influence how people respond to later ones, so a change at step two may show up as a difference at step five. Individual email metrics can therefore mislead you if you read them alone.

There are three practical ways to set up the test.

• Test one email in place. Replace a single email and measure its own metrics plus the conversion rate at the end of the sequence. This is the most common and least disruptive option.
• Test two complete sequences. Run two full versions in parallel. The result is the most definitive, but it needs the most traffic and the longest window.
• Test structure without changing copy. Keep the same emails and change only their timing, order or inclusion. This isolates the effect of the flow itself.

Whichever you choose, write down in advance which of these questions you are answering, so that the result is read against the right metric.

Choosing a tool for the test

You need three things from a platform for drip testing: a way to enroll people into workflows, a way to split traffic between variants, and reporting that shows the outcome at the end of each branch. The table below compares the tools this page covers. Only Drip is an email platform; the others support the test from the side, for landing pages, data movement or documentation.

Drip

Drip suits teams whose drip sequences are the core of their lifecycle email, because its plans include up to 50 workflows and the ability to build dynamic segments that decide who enters which branch. For testing, that structure matters: you need to be able to route half of the new enrollees into variant A and half into B, and keep them there. Most of Drip's listed tiers include unlimited email sends, so sending the variants does not add a per-send charge at those levels.

Its limitations are clear from the pricing page. There is no ongoing free plan, only a 14-day trial that needs no card. The price tiers are set by the number of people, and the listed tiers are billed monthly. The top tiers on the page, such as 37,501 to 40,000 people, include a set number of sends (480,000 for that tier), so check how many emails your sequences will send before choosing a tier. The pricing page does not show whether split testing is included at each plan, so confirm it in Drip's documentation before you design a test around it.

Price: from $39 a month for 1 to 2,500 people, billed monthly. Pricing page: drip.com/pricing.

Convert

Convert is a website experimentation platform. It is a good fit when your email test is only half the story and the page the subscriber lands on is the next thing to optimize. The plans are priced on monthly tested users (MTUs), so the cost depends on how many visitors are included in experiments each month, not on the number of email subscribers.

It falls short for inbox testing. It does not send or manage drip sequences, so you will still need your email platform to run the emails themselves. The listed Growth plan is $399 a month, or $299 a month billed annually, for 100,000 monthly tested users. The Pro plan is $599 a month, or $420 a month billed annually, and adds more segments and advanced features. A 15-day free trial is available on Growth and Pro. Pricing page: convert.com/pricing.

Make

Make is an automation platform that moves data between apps. It does not run emails or A/B tests itself, but it can be useful when your results live in different places. For example, you can send sign-ups by variant from your email platform into a spreadsheet, then compare conversion at the end of each branch without copying numbers by hand.

Its free plan includes 1,000 credits a month with no time limit, but scenarios run at most every 15 minutes, with a maximum of two active scenarios. Each module action in a scenario uses one credit, so count your steps before you build a reporting flow. The Core plan is $10.59 a month, or $9 a month billed annually, for 10,000 credits, and the Pro plan is $18.82 a month, or $16 a month billed annually, for the same credit allowance. Pricing page: make.com/en/pricing.

Linear

Linear is an issue tracker. It has nothing to do with sending email, but teams that run many tests often lose track of what they tried. Using Linear as the test register solves that: each test becomes an issue with the hypothesis, variants, primary metric, sample size, start and end dates, and result. The free plan covers unlimited members, two teams and 250 issues. Basic is $10 per user a month, billed yearly, and includes five teams and unlimited issues. For a team of ten on Basic, that is $1,200 a year. Pricing page: linear.app/pricing.

What the cost looks like at realistic list sizes

Drip is the only email platform in this comparison, so the table below shows its cost at several list sizes. The price is taken from the tier that covers each size on Drip's pricing page, billed monthly. The annual figures are the monthly price multiplied by 12, so they show what a year of drip testing costs before any discounts that might be negotiated.

Two practical points follow from the table. First, the monthly cost steps up with each band of people, so a list that sits just above a band boundary pays for the whole next band. Second, a test does not add to the bill on tiers that include unlimited sends, which makes the cost of running variants easy to estimate. Keep the sends cap in mind for the largest band.

Setup checklist for a split test

Before you switch on a test, go through this list. Most false results come from one of these items being skipped.

• Entry event. Write down the exact event that enrolls a person into the sequence, and check that it fires once per person.
• Random assignment. Confirm that the split happens at enrollment and that a person who re-enrolls is kept in the variant they first received, or is excluded from the test.
• Suppression. Exclude people who have already bought, replied, unsubscribed or entered a competing sequence, so that the variants compare like with like.
• Primary metric and window. Name the single conversion that decides the winner, and the number of days after enrollment you will wait for it.
• Sample target. Record the number of people per variant from your calculator before you start.
• Stop rule. Decide in advance when you will read the result, and do not read it earlier.
• Test register. Log the hypothesis, variants, dates, sample and result, in a tracker such as Linear or a shared document.

Common testing mistakes

Changing too many things at once

If the new subject line, the new body copy and the new send time all go live together, a winning result tells you nothing about which change caused it. Change one variable per test. When you need to test several at once, use a structured multivariate design and a much larger sample.

Stopping too early

Early in a test, one variant often leads by a wide margin purely by chance. Read the result only after the sample target is met and the gap has stayed stable. A number that looks dramatic after a hundred subscribers may disappear by the end of the window.

Ignoring the calendar

A test that runs through a holiday sale, a product launch or a slow week will reflect those conditions. Either run the test across a typical period, or note the event in your register and avoid applying the result to a different season without a second test.

Testing trivial differences

Small visual changes rarely move a sequence in a measurable way, and they consume the same test capacity as a bigger idea. Put your effort into the offer, the first sentence of each email, and the sequence structure.

Not documenting the result

A test you cannot find later is a test you will repeat. Record the hypothesis, the variants, the metric, the sample size, the dates and the outcome, even when the variant lost. Losing results are often the most useful, because they rule out an idea for good.

Documenting a test: a template

The register works best as the same short record for every test. Fill it in before the test starts and complete the result once the stop rule is met:

• Test: the name of the email or sequence
• Hypothesis: the change, the expected effect and the reason
• Variants: A (control) and B (challenger), with the exact wording or setting of each
• Primary metric: one metric, plus the secondary metrics you will watch
• Sample target: people per variant, from the calculator
• Window: start date, end date and days after enrollment
• Result: the observed value for each variant and whether the difference met your significance threshold
• Action: what you rolled out, or why you kept the control, and the next test to run

How to choose the right setup

The decision depends on what you are testing and what you already have in place. The table below gives a starting point.

If you are unsure which test to run first, start with the first email's subject line on the sequence that gets the most enrollments. It is the cheapest test to set up, it reads fastest, and a winner there improves every email that follows.

Frequently asked questions

Which email tool should I use for A/B testing a drip sequence?

Drip is the most direct fit among the tools on this page, because its workflows and dynamic segments route people into branches. Its plans start at $39 a month for 1 to 2,500 people, billed monthly. Confirm on Drip's pricing page and documentation that split testing is included on the plan you choose.

How many subscribers do I need per variant?

It depends on your current conversion rate, the smallest improvement you want to detect and your confidence level. Enter those three values into a sample-size calculator. Smaller expected lifts need larger groups, so a test for a big change on a low-volume list is usually more practical than a test for a small change.

How long should a drip test run?

Run it until both variants reach the sample target and the result is stable. Sequence tests often take several weeks, because the final conversion can happen well after the first email. Plan the window around the slowest step in the sequence.

Should I split people by date or at enrollment?

Split at enrollment, randomly, and keep each person in the variant they received. Splitting by date means one variant may get a holiday, a launch or a quiet week, which can distort the result.

Can I test the send time of each email separately?

Yes. Test time of day, day of week and the spacing between emails as separate experiments, each with its own primary metric. Timing results are often specific to your audience, so treat a winner as a finding about your list rather than a universal rule.

What if the test shows no clear winner?

A test with no significant difference is still a result. Record it in your register, keep the control, and check whether the test was too small to detect the change you expected. A null result is often the cue to test a bigger change rather than a smaller one.

Do I need a separate tool for landing page tests?

Not always. Many teams run their first tests on the email alone. If your sequence sends people to a page you want to optimize, a website experimentation tool such as Convert can test that page, while your email platform handles the sequence itself.

How do I avoid repeating tests I have already run?

Keep one register for all tests, with the hypothesis, variants and outcome for each. Search it before you write a new hypothesis. Linear, a spreadsheet or a shared document all work, as long as every test is logged in the same format.