How to A/B Test Cold Email Campaigns (Full Guide)

One of the single most important factors when running a cold email campaign is A/B testing.

A/B testing is important because over time it can take a cold email campaign that receives a 5% response rate and convert it into a cold email campaign that returns a 20% response rate. That's a 4x increase in responses for the same exact effort.

In order to make sure you get it right, we'll be outlining our own internal method for A/B testing cold email campaigns in this article. By the end of it, you'll know everything needed to run successful A/B tests and improve the results of your cold emails.

Let's dive into it!

What to Know Before A/B Testing Cold Emails

Before we get into the exact process of A/B testing cold email campaigns, there are a few important tips we need to highlight. Each tip aims at improving the quality of your A/B tests so you can avoid using bad data to make decisions.

Let's run through what you need to keep in mind before you start A/B testing:

Tip #1) Use the subject line to optimize open rates.

It's somewhat self-explanatory, but just in case you're not aware, I wanted to reiterate that subject lines have a direct impact on open rates. So, if you're testing subject lines, make sure you're optimizing for the open rate and not the response rate.

Tip #2) Use the email body to optimize response rates.

Unlike subject lines, the email body has a direct impact on response rates. If you’re A/B testing email bodies, be sure you're optimizing for response rates and not open rates.

It is possible that an email body has a negative impact on open rates, but this is generally due to spam trigger words or the use of too much HTML. However, assuming your email follows our guidelines on writing cold emails, the email body shouldn't impact the open rate all that much.

Tip #3) Start the A/B test at the same time.

There are tons of different factors that can impact the results of an A/B test. A few examples are holidays, weekends, news, and even a decrease in a domain’s sender reputation.

All these factors can hinder or improve a cold email campaign. So, if you were to run two tests and send one at the beginning of December and run the second at the end of December near Christmas, of course, the first test would yield a better result.

Even though this is an oversimplified example, it does a great job of clarifying why it’s important to start an A/B test at the exact same point in time.

Tip #4) Always test with the exact same audience.

We’re constantly talking about the effectiveness of your ideal customer profile. Well, targeting your ideal customer will yield better results than targeting a not-so-ideal customer. And for this reason, it’s extremely important to run A/B tests against the same exact audience.

A good example would be targeting startups versus enterprise organizations. Startups are known to be easier to sell to while enterprise organizations are typically more difficult.

For this reason, if you were to A/B test startups against enterprise organizations, you would think the cold email campaign targeting startups was the better performing campaign. But, in reality, this would only be the case because it's simply easier to get responses from startups.

So, if you're going to run an A/B test, make sure you build one audience and test your cold email campaigns against that audience.

Tip #5) Test one aspect of a cold email at a time.

It's also important to test just one aspect of a cold email at a time. This means testing simply the body or the subject line.

If you're testing both the subject and the body, it's going to be hard to determine which aspect of the email is actually responsible for the increase in the response rate.

However, if you were to test two cold emails with the exact same body but two different subject lines, you would know that any change in the open rate would be a direct result of the subject line.

In my experience, it's generally best to find a subject line that yields a high open rate and as soon as you are receiving an open rate of 50% or more, move on to testing the email body and optimize for the response rate.

How to A/B Test Cold Emails

Now let's get into how to A/B test cold email campaigns. In this guide, we’ll be using LeadLoft to walk you through our process.

Let's dive in!

Step #1) Build “Playbook A”

The first step to running an A/B test is to build your first cold email campaign. Within LeadLoft these are called Playbooks.

So we'll hop into LeadLoft and click “Create Playbook” and start building our first Playbook.

You can add as many or as few email steps as you'd like but in this example, we'll add just two.

Here's what Playbook A looks like:

Step #2) Decide on what you are going to test.

Now we need to decide on what we're going to test. In this example, we're going to test the subject line.

LeadLoft allows you to send emails in the same thread so we'll go ahead and set the first email’s subject line to “Quick question” and we'll leave the second email’s subject blank so it sends in the same thread.

Here's what my email looks like in Playbook A:

Step #3) Duplicate Playbook A & Rename it to “Playbook B”

Now that we have the first cold email campaign set up, let’s build test B. We can make this really easy by simply duplicating Playbook A and renaming it.

If you click the top right “more” button, you get a series of options, duplicate being one of them.

So go ahead and click that.

Let's go ahead and rename this Playbook to ”Playbook B”

Step #4) Update Playbook B

Now that we have both Playbook A and Playbook B created, let's update Playbook B's subject lines so we can run our test.

To do this, simply open up the first email and update the subject line to the subject line you would like to test.

For Playbook B, we’ll be testing the subject line ”{{FirstName}}, need revenue?”

This is what that looks like:

And there we go, both A/B tests are set up and ready to begin testing.

Step #5) Build a Narrow List of Prospects

Just in case you forgot, it's extremely important that you test with one narrow audience at a time. We like to use LinkedIn Sales Navigator to build a narrow set of profiles to ensure we're testing a very similar audience, but you can also use LinkedIn’s Advanced Search. This reduces the odds of the A/B test producing misleading results.

For this example, we'll use the below search criteria:

Job Title: Founder
Location: San Francisco California
Headcount: 11 to 50
Keyword: AI

Here's what the search filters look like on LinkedIn Sales Navigator:

Step #6) Enroll Prospects & Run The Test

Before we can run the test we'll need a way to save the prospects. We’ll be using the LeadLoft Prospector to find contact info and enroll these prospects in Playbook A and Playbook B.

Here are the settings we’ll use to find and engage prospects with Playbook A:

And here are the settings we’ll use to find and engage prospects with Playbook B:

Now all we have to do is click the bright blue save button to send the Prospects to the LeadLoft dashboard, find their contact info, and enroll them in the selected Playbook.

Let's save 20 for each playbook to run our test.

After you've selected and enrolled 20 prospects in Playbook A and Playbook B, this is what your LeadLoft dashboard will look like. You can see there are 40 leads that we’re now engaging.

Now all that is left is to sit back, relax, and await the results.

Step#7) Pick a Winner

We've now let the campaigns run their course and all prospects have received both emails.

Now let’s review the Playbooks and pick a winner:

The Winner is Playbook A with a 75% open rate.

This means that the subject line ”Quick question” yielded 19% higher open rate than “{{FirstName}}, need revenue?”.

Even after I’ve picked a winner, I like to archive the losing Playbook and repeat Steps #3 through #7.

This is to ensure that I’m continuously testing my cold email campaigns and never get satisfied with the current results. At the end of the day, there's always room for improvement, especially when it comes to the body of the email.

Wrapping Up

If you're running cold email campaigns, A/B testing is just something that you should be doing. It doesn't take very much effort and the impact it can have on your bottom line is massive.

Getting a 20% response rate might sound difficult, but after three or four A/B tests, you might just be there.

If you found this helpful but still need some additional help setting up your cold emails, feel free to reach out to our team. We’re obsessed with cold email and are always more than happy to help out our users.