A/B testing is still the cheapest reliable way to learn what makes your subscribers click, buy or unsubscribe. It only works if you change one thing at a time and judge the result by a metric you can trust. That metric is no longer the open rate. Since Apple Mail Privacy Protection launched in 2021, Apple Mail preloads images and inflates opens, so clicks, conversions and revenue are the safer ways to pick a winner. Every variant you send also has to meet the Gmail and Yahoo bulk sender rules from 2024 (SPF, DKIM, DMARC, one-click unsubscribe, a spam rate under 0.3%) and Microsoft’s Outlook.com rules from May 2025.
Why testing deserves a place in every campaign
It sounds obvious. But when deadlines are tight, even experienced teams cut testing time. Testing is a critical step, both when you plan a new campaign and when you improve one that is already running, and it should not be skipped.
It matters most when you work toward measurable goals and business objectives. A test turns “we think the shorter subject line works better” into a number you can report, repeat and build on. Each test is cheap. The value comes from running them over and over: small, proven gains in click rate or revenue per email add up across every send in a year.
There is a fair counter-argument. On a small list a test can’t detect anything but a big difference, and a badly run test can be worse than none because it gives you false confidence. The answer is to test less often and test bigger changes, not to stop testing.
The basic terms, borrowed from school science
A/B testing is the same controlled experiment you learned in elementary school science. Three terms do most of the work:
- Control: the version you already send, or your current best performer. Each new variation is measured against it.
- Variable: the one element you deliberately change between version A and version B.
- Constant: everything you keep exactly the same in both versions, so any difference in results can only come from the variable.
The old version of this post used Merriam-Webster’s general dictionary definitions. The plain testing meanings above are more useful when you plan an email test.
Email variables you can test
Any element a subscriber sees or experiences can be a variable. The classic list still holds. A few items now come with caveats:
- Subject line: the most popular test. Judge it by clicks or conversions, not opens, because Apple Mail Privacy Protection inflates opens.
- From line: the sender name and address. Keep the sending domain the same in both versions so authentication and reputation stay constant.
- Content: the body copy and how it is structured.
- Creative: images, layout and design. Check both versions in dark mode and on mobile, since many people now read email that way.
- Color: buttons, backgrounds and accents.
- Number of links: fewer links can focus clicks. More links can catch more interests.
- CTA (calls to action): wording, placement, button versus text link.
- Copy length: short and scannable versus long and detailed.
- Delivery type: HTML versus plain text. The old option of sending a separate text version to some readers is history now. Modern ESPs send multipart HTML with a plain-text part, so the real test today is a designed email against a simple, text-style one.
- Price: different price points or ways of showing the price.
- Offer: percentage off versus a fixed amount, free shipping, a gift with purchase.
- Frequency: how often you send. This can’t be settled in one campaign. It needs a holdout group tracked over weeks, with unsubscribes and spam complaints watched as closely as revenue.
- Time of day: send time. Mailchimp and most other ESPs offer it as a built-in test variable.
How to run a clean A/B test
Pick one variable, keep everything else constant, and decide in advance how you will choose the winner. Isolating the variable is what makes the result sound. If you change the subject line and the hero image together, you can’t tell which one moved the numbers.
- Choose one variable from the list above and write down what you expect to happen.
- Hold every other variable constant: same audience segment, same send time (unless time is the variable), same sending domain, same offer.
- Choose the winning metric before you send. Use click rate for content and CTA tests, conversion rate or revenue for price and offer tests, and click rate rather than open rate for subject lines.
- Split the audience randomly. Your ESP handles this. Mailchimp, for example, assigns recipients to combinations at random.
- Let the test run long enough. Mailchimp recommends waiting at least 4 hours before it sends the winning version to the rest of the list. Klaviyo can end a campaign test automatically once the winning metric reaches statistical significance.
- Record the result: the variable, the versions, sample sizes, the metric and whether the difference was significant. Over time this log tells you more than any single test.
When you plan a campaign, write down how you will test each variable and how you will track the rise or fall in results it causes. That plan is what turns one-off tests into a program.
What today’s ESPs let you test
Most platforms build in single-variable A/B tests. The two below are widely used examples, taken from their help centers:
| Platform | Variables in a standard A/B test | How the winner is picked | Multivariate option |
|---|---|---|---|
| Mailchimp | One variable (subject line, From name, content or send time), up to 3 variations | Click rate, open rate, total revenue (with a connected store) or manual choice | Up to 8 combinations, on higher-tier plans (Premium, per Mailchimp’s feature page) |
| Klaviyo | Subject line, content, sender and send time in campaigns | Open rate or click rate for campaigns, with an option to end the test once results are statistically significant | Several variations per test. Results are labeled statistically significant, promising, not statistically significant or inconclusive |
A worked example on sample size
Small differences need big lists. Say you send two CTA versions to 2,000 subscribers each. Version A gets a 2.0% click rate (40 clicks) and version B gets 2.5% (50 clicks). That gap looks like a 25% lift, but a standard two-proportion test gives a z-score of about 1.07, well short of the 1.96 needed for 95% confidence. It could easily be chance.
To reliably detect a jump from 2.0% to 2.5% at 95% confidence and 80% power, you need roughly 14,000 recipients per version. If your list is smaller, test bolder changes, such as a new offer or a completely different layout, where the gap is big enough to show up. Or repeat the same test over several sends and combine the results.
Keep both versions compliant
A test version that breaks the rules can damage the reputation of the whole domain. Since February 2024, Google’s email sender guidelines and Yahoo’s Sender Hub have required bulk senders to authenticate with SPF, DKIM and DMARC, support one-click unsubscribe on marketing mail, honor unsubscribes within two days and keep spam complaint rates below 0.3%. Google recommends staying under 0.1%. Microsoft began enforcing similar authentication rules for domains sending more than 5,000 emails a day to Outlook.com, Hotmail.com and Live.com addresses on 5 May 2025.
Watch complaints per variant, not just clicks. An aggressive subject line that wins on clicks but doubles spam reports is a loss. The FTC’s CAN-SPAM rules apply to every version too: honest subject lines and From lines, a physical postal address, and a working opt-out honored within 10 business days.
Multivariate testing
Multivariate testing changes several variables at once and measures every combination, so you can see which elements work best together. The catch is the audience it needs. Testing two subject lines against two CTAs makes four combinations. Each one gets a quarter of your sample, and you need enough sends in every cell to reach significance.
Mailchimp caps multivariate tests at 8 combinations built from its four variables. For most senders the better order is to find the strongest single variables with A/B tests first, then use multivariate tests to fine-tune how the winners combine. If your list isn’t large enough to run a clean A/B test on clicks, it isn’t large enough for multivariate testing either.
The habit that matters most is simple: start a test log this week, run one single-variable test per campaign and judge it on clicks or revenue. Within a quarter you’ll have results you can act on.
FAQ
Can I still A/B test subject lines using open rate?
Not reliably. Apple Mail Privacy Protection preloads images for many Apple Mail users, which records “opens” whether or not anyone read the message. Choose subject line winners by click rate, conversions or revenue. If your ESP can filter out machine opens, treat the adjusted open rate as a rough signal only.
How big should my test groups be?
It depends on your baseline rate and the size of the lift you want to detect. To spot a change from a 2.0% to a 2.5% click rate with confidence, you need about 14,000 recipients per version. Smaller lists should test bigger, bolder changes so the difference is large enough to measure.
How long should an email A/B test run?
Long enough for most of the engagement to come in. Mailchimp recommends at least 4 hours before it sends the winning version. For revenue or conversion tests, 24 hours or more gives a fairer picture, especially if your subscribers live in different time zones.
Should I test one variable or several at once?
Test one variable at a time unless your list is very large. A single-variable A/B test tells you exactly what caused the difference. Multivariate tests split your audience into many small groups and need far more recipients to reach significance.
Do test emails have to follow Gmail and Yahoo sender rules?
Yes. Every version you send counts toward your domain’s reputation and spam rate. Both versions need SPF, DKIM and DMARC authentication and a one-click unsubscribe, and your complaint rate should stay below 0.3% in Google Postmaster Tools.
What should I test first?
Start with what drives your main goal. For most marketers that is the offer, the CTA or the subject line, judged by clicks or revenue. Leave small details like button color until you have proven which big elements work.
Sources
- Google Workspace Admin Help: Email sender guidelines FAQ
- Yahoo Sender Hub: Sender best practices
- Microsoft Tech Community: Outlook’s new requirements for high-volume senders (April 2025)
- Federal Trade Commission: CAN-SPAM Act, A Compliance Guide for Business
- Mailchimp Help Center: About A/B Tests and About Multivariate Tests
- Klaviyo Help Center: Understanding statistical significance in Klaviyo campaigns