A/B test a campaign

An A/B test sends two variants to a sample of your audience, waits, then sends whichever performed better to everyone remaining. It is the cheapest way to stop guessing about subject lines.

1. Choose what varies

Enable A/B testing on the campaign and set up two variants. Change one thing between them — usually the subject line. If you change the subject, the sender and the first paragraph all at once, a win tells you nothing about which change caused it.

2. Set the sample size

The sample is the portion of your audience used to decide, split evenly between the variants. Around 20% is a sensible default: large enough to be informative, small enough that most of your list still receives the winner.

On a small list, consider not testing at all. A test on 200 people produces a difference well inside the range of random noise, and acting on it is worse than not testing, because it feels like evidence.

3. Set the decision window

The decision window is how long Solstral waits before picking a winner. Most opens arrive in the first few hours, but a window that is too short over-weights whoever happens to be at their desk. A few hours is usually right; overnight is reasonable for a business audience.

4. Let it send

The sample goes out first. When the window closes, the variant with the higher open rate is sent automatically to the rest of the audience. No one has to be watching at the right moment.

5. Read the result honestly

Both variants stay visible in the campaign statistics, so you learn something either way. Treat a small gap as a tie rather than a discovery — if variant A wins 22% to 21%, you have not learned that A is better, you have learned that the subject line was not the thing holding this campaign back.

Consistent, repeated wins across several campaigns are worth generalising from. A single result is not.