A/B testing cold emails and outreach messages

To A/B test outreach, change one thing, split a similar list at random, send both versions at the same time and compare real replies once the whole sequence has run. At a 3% reply rate you need roughly 750 sends per version to see a doubling reliably; at 10% about 200. Smaller lists can still guide you, but do not call a win from twenty sends.

Everybody agrees you should test your outreach. Few do it in a way that teaches them anything. The usual version is: send version A to Monday's list and version B to Tuesday's, notice B did better, adopt B. That test has changed the list, the day and the wording at once, and it ran on too few people to mean anything. This guide shows a plain method that a small team can follow, what to test in what order, and how to read the answer.

Why test at all?

Because reply rates differ widely and your audience is not the average. Instantly's 2026 report says top campaigns keep emails under 80 words and A/B test new messaging weekly, and lists frequent testing among the factors that separate top campaigns. That is the vendor's reading of its platform data, so take it as direction. The reason to test is not to chase a trick. It is to replace opinions in your team with numbers from your own list.

What should you test, in what order?

Not everything is worth a test. Rank by how much each thing can move.

Our own data supports that order. In the Sellinger case study, cutting reply time from 19 to 2 minutes did not lift meetings: the list was the bigger lever. In the Lamothermic LinkedIn campaign, follow ups lifted the reply rate from 17.2% to 30.9%, with 7 negative first replies across 1,699 follow up recipients. See the Lamothermic case study and the Sellinger case study.

How do you run a clean test?

Five rules.

1. One change. If version B has a new subject, a new ask and a new length, you cannot say which did the work.
2. Random split. Shuffle the list, then give every other person to B. Do not split by alphabet, company size or date added.
3. Same time. Send both at once, so the day and week affect them equally.
4. Same audience. The two groups must be the same kind of people.
5. Full sequence. Wait until the follow ups have run, otherwise you compare first touches only. In the Shelly installer programme the first email alone drew 3.3% and the whole sequence 13.6%, so ending the test early would have changed the answer.

How many sends do you need?

More than you think. A standard two proportion calculation at 95% confidence and 80% power gives:

These are textbook estimates, not guarantees, and they assume a clean random split. The meaning is simple: large differences show up in a few hundred sends, and small differences need thousands. If your list is 300 people, you cannot detect a one point change. You can still learn from a large difference, and you can read the replies to see why.

So choose tests that could produce a big effect. Rewording a sign off will never show up in a 300 person test. Changing the offer might.

What do you measure?

Count, per version:

• Replies that say something. Not auto replies, and not just "thanks".
• Positive replies. Interest, a question about price, a forward to a colleague.
• Refusals and opt outs. A version that gets more replies and many more opt outs may be worse.
• Booked meetings, if the list is large enough.

Avoid opens. Backlinko's study of 12 million emails reports on response rate, and its authors note their findings may be correlational, not causal. That is a good habit to adopt: be cautious about causes.

Use the free reply rate calculator to keep the arithmetic consistent, and count a reply the same way in both versions.

What can you learn from a small test?

Even a 100 person split is useful if you read the replies. Look for:

• Echoed phrases. What words does the person repeat from your message?
• Confusion. Did they ask what you meant? Fix the clarity.
• The wrong person. Many "not me" replies mean the list or the role is off.
• Forwards. A message worth forwarding is a good message.

The Uberwood Agency campaign found that follow ups added 43% more replies when they brought a new angle. That is a finding about the content of a follow up, and you can reproduce it by testing a follow up that adds a fact against one that repeats the first. See the case study.

What are the usual mistakes?

• Stopping early when one version leads after twenty sends.
• Testing five things in one round.
• Changing the list during the test, which breaks the comparison.
• Declaring victory on opens, which may reflect privacy features, not interest.
• Never retesting. A winner on one segment may lose on the next. Audiences and seasons change.
• Keeping a "winner" that gets more refusals.

Can an agent help?

Yes, within limits. Sellinger's config optimization proposes changes to a campaign, such as custom directions, the first message, the number and interval of follow ups and flow sequences. By default it runs daily, makes at most three changes per cycle, requires a minimum confidence of 0.7 and does not auto approve, and it excludes product information and inclusion and exclusion rules from changes. A person reviews each proposal. It is a way to get ideas from your own data, not a replacement for a clean test.

A worked example

Suppose you write to 800 heads of operations at logistics firms. Your question: does a first line about a specific trigger beat a general opening?

• Version A: "I saw you opened a depot in Plovdiv, so I wondered how you handle returns."
• Version B: "I help logistics firms handle returns more efficiently. Is that a topic for you?"

You shuffle the 800 and send 400 of each, the same day, with the same follow ups. After two weeks, A drew 36 replies (9%) and B drew 20 (5%). Is that real? With 400 per version it is borderline, so you read the replies. A's replies mention the depot and ask follow up questions. B's are mostly "not now". You keep A, and the next test applies the same idea to a second segment, where you can see whether the lesson holds. The numbers here are invented to show the method, not a result.

Two lessons. A borderline number plus a qualitative read is a decision. And a test is a question you carry to the next segment, not a permanent verdict.

A one page test plan

• [ ] Hypothesis: "A question first line gets more positive replies than a statement."
• [ ] One change only, written down.
• [ ] A random split of a similar list, at least 200 per version.
• [ ] Both versions sent in the same week.
• [ ] Result read after the full sequence, with refusals counted.
• [ ] Decision: keep, drop or retest on another segment.