Here’s an in-depth guide to A/B testing for email campaigns that will improve everything from conversion rates to your sales department’s self-esteem.
What is A/B Testing?
A/B testing (also called split testing) is an email marketing strategy that “tests” two different versions of the same thing by altering a single component to see which of the two versions — Version A or Version B — is preferred by the audience based on metrics like click rate, click-through rate, open rate, etc. Common variations in email A/B testing include:
Subject line
Preview text
CTA
CTA button
Body copy
Design elements
Images
Timing
The Benefits of A/B Testing
But what does it yield? The benefits of this type of testing are many, and include:
It reveals what your subscribers like or want.
It’s a form of targeted market research.
It’s inexpensive and takes little time when marketing automation software is used.
It leads to more conversions.
It can help you hone your messaging and overall marketing strategies.
A/B Testing 101: The Fundamentals
A/B testing can be as simple or as complex as you have time and resources. However, there are still some fundamentals that apply to almost every situation, including:
Analyze the metrics and data you already have. If you’ve been engaging in email marketing, then you already have some data to pore over. What does it show you? What’s working? What converts? Where do people bounce? Are there any overlaps in the emails that get opened and clicked on?
Identify your goal(s). What’s the problem you’re trying to solve? That’s the goal — or KPI — to work toward. Too few people opening your emails? Too few clicking on the call-to-action button? Want to find out if a new pricing structure garners more clicks? Whatever it is, define it, and work from there.
Develop a hypothesis. What do you think could fix your problem and help you reach your goal? Should you send the email at a different time? Is the subject line not engaging? Is the email too long? Does your CTA button look weird on mobile? Be specific, and make sure your hypothesis can be addressed in a clean and clear way.
Set up metrics and tracking events. Depending on what you’re testing, you need to make sure you can accurately measure it.
Test your hypothesis. Here’s where you need to set up a control email — ideally something that isn’t different from what you’ve been doing — and a variation email that tests your hypothesis. Send them out according to your hypothesis.
Analyze, optimize, and go again. How did your hypothesis fare? What new information did you learn? If a clear winner emerged among your A/B testing ideas, then move forward with it. If there wasn’t a clear winner, you may need to re-hypothesize and go again.
Common Mistakes in A/B Testing
A/B testing requires a certain restraint. While many email marketing automation apps have guardrails in place, mistakes can be made. Here are some of the more common ones you need to be sure to avoid:
Not taking enough time. Testing takes time. A good rule of thumb is to let a split test run for at least one to two weeks before analyzing its data.
Testing too many variables at once. Yes, some email marketing tools offer multivariate testing, but too many variables can muddy your data’s waters, making it difficult to address your real problem(s) and reach your goal(s).
Misunderstanding statistical significance. In A/B testing, your marketing efforts are only as good as your math. It can be tempting to see statistical significance where there is none, because when you tested two different subject lines, this one here had a higher open rate. To put it plain and simple, statistical significance calculates the mathematical probability that a difference in variable performance was meaningful and not due to chance. It requires a statistically significant sample size, but that’s just the tip of the iceberg. Here’s a deeper dive into statistical significance. But don’t worry, a lot of marketing automation software can calculate it for you.
Not continuing to test. Just because you’ve solved a problem or two by A/B testing email subject lines doesn’t mean it’s time to rest on your laurels. Continuous experimentation and tweaks based on A/B test results will drive improvement, especially since there’s almost no end to what can be tested.
Core Elements to A/B Test in Email Campaigns
It really is possible to A/B test almost everything in your email marketing campaigns, but there are some core elements to start with.
Subject Lines
Split testing subject lines is one of the most important tests you can make in your email marketing campaigns. Why? Because a good email subject line gets opened more often than a mediocre one.
If you need to boost your email open rates, then you need to focus on the subject line. Just be sure the subject line faithfully represents the email content. Nothing ruins a good subject line like email body copy that doesn’t deliver on what was promised.
Call-to-Action (CTA)
Testing your call-to-action can take many, many different formats. Where the CTA lands in the email copy can be something you test. CTA button color can be something you test. CTA language can be a test.
If you’re trying to get your click rate up, you need to spend some time on this one. Just remember not to test too many different elements at once.
Design Formats
There’s a lot that goes into an email’s design that can impact how users respond, including visuals and images, body copy, font, colors, emoji use, lack of emoji use, and more. When you’re trying to tweak your design format to see if you can improve some of your email marketing metrics, stick to the fundamentals: develop a hypothesis and don’t test too many versions of an email at once.
Then, once you’ve got a winning version, keep testing. Design preferences change over time, and different audience segments likely will respond differently to different design elements, too.
Timing
One of the white whales of email A/B testing is timing. While there are general rules around the best times and day(s) of the week to send an email, your brand and audience are unique. You’ll have to conduct your own split testing to see what works best, depending on your goals.
Additional Pro Tips for Conducting Effective A/B Tests
Now that we’ve laid the groundwork for conducting effective A/B tests, here are a few more pro tips to improve your campaign even more:
Pay attention to target audience. Email marketing can really help you tap into certain segments of your customers who aren’t as engaged as you’d like. Use A/B testing to find out what elements can be tweaked to drive everything from social media signups to purchases.
Know which metrics matter and act accordingly. There are a lot of metrics to track in email marketing, but only you and your marketing team can decide which ones to focus on.
Validate your email lists. Keep your bounce rates and unsubscribe rates low by validating and cleaning your email lists regularly.
Leveraging Continuous Experimentation to Build Better Campaigns
One important distinction between A/B testing and other marketing techniques and tools is that A/B testing works best when it’s an ongoing process. Tech, marketing theory, consumer behavior, trends, policies — everything is in flux, and marketing well means relating well to all of it.
Committing to A/B testing in order to make small, incremental improvements over time can really pay off, because you can respond to changes in consumer behavior that don’t even exist yet.
Why Email List Hygiene Is a Must for Successful A/B Testing
Successful A/B testing has to have accurate email lists, or the data you’re generating will be off, rendering the tests invalid — and you might not even know it. After all, what’s the point of sending two versions of your email if 5% of them don’t even end up in active inboxes?
Email list hygiene — validation, real-time verification, and ongoing, automated email list cleaning — can keep your email A/B testing valid. Without it, you risk getting spam complaints and earning a bounce rate that makes overall deliverability more difficult.
Practical Tools & Resources for A/B Testing
Whether you’re an e-commerce startup with a lot of funding or a nonprofit with limited resources, practical tools and resources to run solid A/B tests abound. Here are some of the best:
ActiveCampaign. This platform lets you run multivariate tests and offers customizable metrics.
Mailchimp. A classic in the email marketing landscape, Mailchimp allows for three A/B variations to test, while also providing detailed analytics and reporting options.
Klayvio. Easy to use, Klayvio offers basic split testing with reporting and analytics.
HubSpot. This CMS offers excellent email marketing analytics and plenty of options for A/B testing.
NeverBounce. Essential service to ensure email lists are clean and up-to-date.
How NeverBounce Can Help You Build Winning Email Campaigns
Don’t shortchange your A/B testing efforts. Ensuring email list hygiene is excellent before you test is the only way to ensure accurate and actionable testing results.
NeverBounce offers bulk email list cleaning, real-time email verification for new signups, and automated list cleaning so you can have confidence that when you A/B test, the results will be accurate.