If you want to optimize the messages you send, A/B testing is key. But what should you test?
Marketers have always had opinions about what their audience wants. A/B testing is the practice of testing those opinions rather than just acting on them—so every message you send is smarter than the one before it.
By sending two versions of a message to different segments and measuring which performs better (on CTR, conversion rate, or revenue per send), you replace gut instinct with evidence. The gains can be significant: Attentive customers that test send time alone see an average 8% CTR lift and 33% CVR lift.
In this article, we’ll cover best practices for SMS and email A/B testing, real examples with measurable outcomes, and concise answers to the questions teams ask most:
- How many subscribers you need
- How long to run a test
- When A/B testing is the right tool vs. multivariate testing
A/B testing best practices
Before you start running A/B tests, let’s review a few best practices from our team—having worked with more than 8,000 brands:
- Test one variable at a time. Keep copy, design, and offer identical across variants—only the element you’re testing should differ. Testing two variables simultaneously makes it impossible to know which change drove the result.
- Start small. Begin with high-impact, low-effort elements—subject lines, CTAs, or send times—before testing full redesigns.
- Prioritize revenue-generators. Run tests on your highest-frequency sends: welcome series flows, abandoned cart messages, and promotional campaigns—these deliver the largest sample sizes and fastest learnings.
- Hit minimum sample size. Minimum 3,000 subscribers per variant for SMS. The larger the test group, the more reliable your results.
- Write a hypothesis first. State what you’re testing and what result you expect before you launch—this keeps results interpretable and prevents post-hoc rationalization.
- Run tests more than once. A single result isn’t conclusive. Rerun key tests across campaigns and time periods to confirm the finding holds.
- Test up to 30 variations. With Attentive, you can run up to 30 variations in a single A/B test campaign—useful when testing send times or copy across multiple segments.
- Use auto-winner tests. Set a test window (e.g., 20% of your list for two hours), define your winning metric (e.g., CTR), and automatically send the best-performing variant to your remaining recipients.
- Wait for statistical significance. Don’t declare a winner until results are statistically significant. See “How long should you run an A/B test?” section.
What is the difference between A/B testing and multivariate testing?
Quick answer: A/B testing compares two versions of one element. Multivariate testing tests multiple elements and their combinations at the same time. A/B testing works for any list size. Multivariate testing requires significantly more traffic to reach reliable results.
| A/B Testing | Multivariate Testing | |
|---|---|---|
What it tests | One variable, two or more versions | Multiple variables and their combinations |
List/traffic requirement | 3,000+ subscribers per variant | 5-10x more than A/B |
Speed to results | Faster | Slower |
Best for | Testing a single change | Finding the best-performing combination |
Practical threshold | Any size | 100,000+ monthly sessions or very large lists |
How long should you run an A/B test?
Quick answer: Run every test for at least two full weeks, regardless of how promising early results look. One week is rarely enough to account for day-of-week variation in subscriber behavior.
A few rules to follow:
- Set a fixed end date before the test starts and commit to it
- Do not stop a test early because one variant is ahead
- Make sure you have hit your minimum sample size (3,000 subscribers per variant for SMS) before calling a winner
- Re-run important tests across multiple campaigns to confirm the finding holds
Stopping a test too early, even at high confidence, inflates false-positive rates significantly. A result that looks decisive after three days may reverse by day 10.
What are the most common A/B testing mistakes?
Quick answer: Testing too many variables at once, ending tests early, and using sample sizes too small to produce reliable results.
- Testing multiple variables at once. If your copy, image, and CTA all differ between variants, there is no way to know which change drove the result. One variable per test, always.
- Stopping tests too early. The most common and costly mistake. Set a fixed end date before you launch and do not change it based on early trends.
- Using too small a sample. A test with a few hundred recipients may show a clear winner that reverses at scale. Minimum 3,000 subscribers per variant for SMS.
- Not rerunning tests. Consumer behavior shifts. A result from one campaign in one season may not hold across others. Rerun your most important tests before treating findings as evergreen.
- Confusing statistical significance with business impact. A result can be statistically significant and still not be worth acting on. Always ask: is this lift large enough to matter to our business?
- Starting with low-traffic campaigns. Tests on low-volume sends take longer to reach significance and produce learnings that may not generalize. Start with your highest-frequency sends.
What is the difference between statistical significance and practical business impact?
Quick answer: Statistical significance tells you a result is unlikely to be due to chance. Practical business impact tells you whether the result is large enough to act on. You need both before declaring a winner.
Before you launch any test, define two thresholds:
- Significance threshold: Typically 95%. This is your reliability floor.
- Minimum detectable effect (MDE): The smallest lift that would be worth acting on for your business.
A test that reaches 95% confidence with a 0.2% conversion lift is a real result. Whether it justifies a program change depends entirely on what that 0.2% means in revenue terms for your brand.
Now that we’ve talked about what goes into A/B testing in SMS and email, let’s look at some examples.
Types of A/B tests to run on SMS
Here are some A/B testing ideas and examples to help you get started:
SMS vs. MMS
Incorporating multimedia is a fun way to add color to your text messages.
- Images
- GIFs
- Video
- Audio
But maybe your customers are more likely to engage with text-only messages (i.e., SMS vs. MMS). This is one of the first and easiest A/B tests you can run to see which type of message works better for your audience.
Remember to keep the copy the same for each variation, so you’re only comparing the effectiveness of including an image vs. not including one.
Image type and content
If you’ve determined that your subscribers are more likely to engage with messages that include visuals, the next step is to test different types of images to see which ones convert best. For example, you could include:
- a product close-up in one variant,
- an image of a model in another variant, and
- a lifestyle image in a third variant—using the same copy in each one.
You can also run A/B tests to see if using static images or GIFs and audio or video have more of an impact on click-through and conversion rates. Or, if adding text over an image (e.g., to reinforce a limited-time sale or offer) performs better than the same image without the text added.
Emoji usage
Emojis can be another great way to add personality to your text messages. While we typically recommend using emojis sparingly—and only when they add value to the message—every audience is different, so it’s important to test what works best for yours.
To see if emojis are effective in your SMS campaigns, start by running an A/B test comparing messages with and without emojis. If you find that customers respond positively to messages with emojis, then you can run additional tests to figure out the ideal number of emojis to use per message and which emojis get more engagement.
Format and length
Keeping your text message copy between 75-115 characters (or 3-4 lines long) is a good rule of thumb, but you can run tests to see if your subscribers prefer shorter or longer messages. Try to keep the copy generally the same for both versions, using the shorter message copy as the foundation for your longer message, so you can accurately compare them.
Experiment with how you structure your messages, too. Consider this:
- Do your subscribers prefer when you use no line breaks or many?
- Does it make a difference when you capitalize certain words (e.g., FREE, SALE) or keep the whole message in sentence case?
- Test these elements one at a time to fine-tune the format and length of your text messages.
Link placement and destination
Every text message you send should include a tracking link to your website, but where you place the link can affect how many people click it. Run tests to see if placing links near the top of your message, in the middle of your copy, or at the bottom of your message drives more conversions.
You should also experiment with directing subscribers to different pages on your website. For example, when you launch a new collection, does sending people to the full collection page or to the specific product page influence more purchases? You can do a similar test when promoting new arrivals. Try creating one variation where you direct people to your homepage and one where you link directly to your new arrivals page.
Call to action
Ending your text messages with a clear call to action can help encourage subscribers to click through and shop immediately. Within your CTA, you could test the same words but see how a CTA in all caps performs.
You can test different short-and-sweet options (e.g., “shop now,” “click here,” or “ends soon") and capitalizations (e.g., “Shop now” vs. “SHOP NOW”). Or, see if something more playful or descriptive works better (e.g., “What are you waiting for?” vs. “Shop now”).
Incentives
Our research shows discounts are the No. 1 incentive, but what comes after matters more than most brands realize. Beyond the discount itself, 65% of shoppers say free or faster shipping motivates a repeat purchase, 59% say the same about rewards points, and 52% cite free gifts. Meanwhile, 85% of shoppers are more likely to purchase after receiving a price-drop alert on something they already want.
The only way to know what moves your audience is to test it. Run A/B tests across your campaigns and triggered messages—pitting a straight discount against a free shipping threshold, a loyalty perk, or early access to new drops. Then do the same in your sign-up units: test discount-based offers against non-promotional incentives to see what converts more browsers into subscribers.
Copy variations
In some cases, you might want to experiment with different copywriting approaches or positioning. For example, highlighting the benefits of specific items vs. using a fun play on words to garner interest, or leaning into FOMO to promote limited-stock items vs. positioning them as “back-in-stock” while supplies last.
Just remember to keep as many elements as possible the same in both versions—like the link and the image—since the copy is the variable you’re testing.
Tone of voice
For some brands, sending text-only messages that get straight to the point may work better (e.g., “We’re having a sale. Shop now!”). For others, it might be more effective to use a casual tone of voice that reflects the personal nature of text messaging (e.g.,”Hi friend! Our new collection has your name written all over it. Treat yourself to something new.”).
One Attentive customer that tested their brand voice in SMS found a 15x ROI and their CEO and Co-Founder said, “Test every aspect of your tone to fine tune it and understand what resonates with your subscribers.”
Play around with different tones or styles until you find something that resonates with your subscribers and aligns with your brand voice. Or, test Brand Voice AI that generates brand-specific copy with models trained on your past, top-performing messages.
This is also an area where you can play around with dynamic variables, such as the {firstName} macro, to see if your subscribers respond well to being addressed directly.
Send times or wait times
When’s the best time to send an SMS campaign? Your audience will have specific preferences, but a good place to start is by sending the same message at different times (i.e., in the morning, afternoon, and at night) and on different days of the week to see when your subscribers are most active. If you find that more people tend to open and engage with your messages at night, then you can get more granular and test the same message at hourly intervals (e.g., 5pm, 6pm, 7pm, 8pm). We’ve found if you want to optimize your revenue per send, weekdays are slightly better than weekends.
Keep in mind: Under the TCPA and related state laws, you can’t send text messages during “quiet hours.” Attentive's default and recommended Quiet Hours are 8pm to 12pm EST. If you use Attentive, we also recommend using our time zone-based message sending feature, which allows you to schedule and send messages based on a subscriber's local time zone.
You should also test the timing of your triggered messages to see how different wait times impact performance. For example, A/B test your abandoned cart reminders to send after one hour vs. three hours to see which one leads to more completed purchases. Or, experiment with sending post-purchase messages after 14 days vs. 30 days to figure out the best time to nudge recent shoppers to come back and buy again.
Pro tip: Optimize send by each subscriber. Attentive’s Send Time AI operates at the individual subscriber level. It analyzes factors such as time zone, past interactions, and purchasing habits to identify the perfect time to send a message to each subscriber.
Types of A/B tests to run on email
Many A/B tests that work on SMS apply equally well to email: images, copy variations, emojis, personalization, and CTAs are all fair game across both channels. The main difference is how subscribers engage with each. Emails allow for more expansive content and visuals, and recipients typically read them on their own schedule. SMS is more immediate and concise, which shapes how audiences respond to each type of message.
Timing matters differently across channels too. For email, we recommend sending campaigns earlier in the morning (between 8 am and 11 am) and/or later in the evening (7 pm onward). For great click-throughs, send emails between noon and 3 pm. For SMS, send time is one of the highest-impact variables you can test, since messages compete for attention in a much more immediate way.
Attentive customers can A/B test both SMS and email in the same campaign. The Autowinner feature automatically sends the better-performing variant to your remaining recipients based on whichever winning metric you choose: click-through rate, conversion rate, total revenue, or unsubscribe rate.
Now let’s cover some tests to run in your email programs.
Subject lines
.png)
Your subject line is often the first thing subscribers see in their inbox—and it can make or break your open rates. By creating two variations of your subject line, you can gather data on which version performs better with your target audience. This could involve testing different lengths, using emojis, personalizing with the recipient's name, or experimenting with various tones and wording.
For example, you might compare a straightforward subject line like "Exclusive Offer: 20% Off Your Next Purchase" with a more playful one like "🎁 Unwrap Your 20% Discount Inside! 🎁".
We’ve found that medium-length subject lines (between 25-35 characters) drive the most opens, followed by short ones (fewer than 25 characters) for trigger-based emails.
Email content
[image-modal-trigger-3] [data-bg="#FFD60B"]
[image-modal-trigger-3]
[image-modal-source-3]
.png)
[image-modal-source-3]
[image-modal-trigger-4] [data-bg="#FFD60B"]
[image-modal-trigger-4]
[image-modal-source-4]
[image-modal-source-4]
Your subject line gets subscribers to open your email, but your email content is what gets them to engage. You can test various aspects of your email’s content, such as personalization, offers, social proof, sizing, or placement of images, images vs. videos, and messaging. We’ll drill down further into some of these specific tests below.
Email design
[image-modal-trigger-5] [data-bg="#FFD60B"]
[image-modal-trigger-5]
[image-modal-source-5]
[image-modal-source-5]
[image-modal-trigger-6] [data-bg="#FFD60B"]
[image-modal-trigger-6]
[image-modal-source-6]
[image-modal-source-6]
A/B testing the design of your email marketing campaigns can significantly impact user engagement and overall performance. This could involve testing different color schemes, fonts, layouts, image placements, or even the use of interactive elements like GIFs or buttons.
Testing these variables can provide valuable insights into what visually resonates with your audience. It's crucial to isolate one variable at a time for clear results and to ensure that each version is aligned with your brand's overall aesthetic to maintain consistency.
CTAs
[image-modal-trigger-7] [data-bg="#FFD60B"]
[image-modal-trigger-7]
[image-modal-source-7]
[image-modal-source-7]
[image-modal-trigger-8] [data-bg="#FFD60B"]
[image-modal-trigger-8]
[image-modal-source-8]
.png)
[image-modal-source-8]
By A/B testing your CTAs, you can determine which approach motivates your audience to take action. This could involve testing different button colors, hyperlinked text versus buttons, varying the CTA copy (e.g., "Shop Now" vs. "Discover More"), experimenting with placement within the email, as well as the number of CTAs within an email.
These tests can reveal which language resonates most with your audience, and which type of CTA impacts click-through rates. By comparing the performance of different CTAs, you can gain valuable insights into what motivates your audience to engage and convert. These insights can then be used on other marketing channels to drive even better results.
Send time
A/B testing the send time of your email marketing campaigns can dramatically influence open rates and engagement. This could involve testing different times of the day, such as morning versus evening, or different days of the week, like weekdays versus weekends. Testing your email send time ensures your emails are seen and engaged with at the most effective moments.
Curious about the best time to send your messages during the busiest shopping holiday of the year? Check out our guide: When is the Best Time to Send SMS Marketing and Email During BFCM.
Imagery
[image-modal-trigger-9] [data-bg="#FFD60B"]
[image-modal-trigger-9]
[image-modal-source-9]
.png)
[image-modal-source-9]
[image-modal-trigger-10] [data-bg="#FFD60B"]
[image-modal-trigger-10]
[image-modal-source-10]
[image-modal-source-10]
They say a picture is worth a thousand words, but we prefer pictures that are worth a thousand clicks. Imagery is a specific subset of content you can test in your email campaigns. This could involve testing various image styles, such as lifestyle photos versus product shots, or even comparing static images against animated GIFs.
Frequently asked questions
1. What is A/B testing?
A/B testing is a controlled experiment where you split your audience into two or more groups, show each group a different version of a message or campaign element, and measure which version performs better against a defined metric—such as click-through rate, conversion rate, or revenue per send.
2. What's the difference between A/B testing and multivariate testing?
A/B testing tests one variable at a time. Multivariate testing (MVT) tests multiple variables and their combinations simultaneously. A/B testing is faster and requires far less traffic—multivariate testing requires 5-10x more volume and is only practical for very large lists or high-traffic pages. See the full comparison above.
3. How many subscribers do you need for an A/B test?
A minimum of 3,000 subscribers per variant for SMS. For email, requirements depend on your baseline open and click rates and the size of the lift you're trying to detect. More is always better—larger samples produce more reliable, repeatable results.
4. What metrics should you track?
Track the metric that matches your test goal:
- Open rate → for subject line tests
- Click-through rate (CTR) → for copy, CTA, and design tests
- Conversion rate (CVR) → for offer and landing page tests
- Revenue per send → for holistic campaign performance
- Opt-out rate → to monitor subscriber health alongside performance metrics
5. How long should an A/B test run?
A minimum of two full weeks, plus long enough to reach your target sample size. Never stop a test early because results look promising—early stopping can inflate false-positive rates to 30–40%+.
Keep testing
A/B testing should be an ongoing part of your SMS and email marketing strategy versus a one-time initiative. Even when you find something that works, keep testing. Consumer behavior evolves, and the message that converts today may underperform in six months. Brands seeing the biggest cumulative gains run multiple tests, learn fast, and iterate continuously.
Check out our Resource Hub for more tools, guides, and marketing tips.
