During the PTC/Asymmetric campaign check-in, a key strategic question arose: should the outbound campaign launch to the full target list at once, or roll out in staged subsets to allow for iterative A/B testing? The team decided to launch to the full list simultaneously, with optimization deferred to the post-launch phase.
Related context: [1] | [2] | [3]
Launch to the full list at once. No staged rollout or subset testing prior to launch.
The target list is small enough that splitting it into test cohorts would produce statistically insignificant data — not enough signal to justify withholding any portion of the list from the initial send.
"Because this list is such a small set to begin with, if you were to break it even more into those subsets, we would get some data to optimize, but we wouldn't probably get enough data to really make a difference."
— Karly Oykhman
The threshold Karly referenced: staged rollouts are appropriate for lists in the range of several thousand contacts. Below that, the tradeoff doesn't hold.
Individual contacts are enrolled in different email cadences, so the campaign is inherently distributed across time at the contact level — even if all contacts are activated simultaneously. This provides natural pacing without requiring artificial list segmentation.
The team's plan is to run the full campaign, collect response data across the whole list, and then make optimization decisions (copy changes, CTA adjustments, sequence modifications) informed by real performance. Additional lists can be pulled for subsequent waves once learnings are in hand.
For small, highly targeted outbound lists (roughly under 1,000–2,000 contacts), staged A/B testing is often counterproductive. The statistical noise from small subsets can lead to false conclusions or simply delay launch without meaningful learning. A better approach is to launch fully, treat the first wave as a learning run, and optimize before pulling the next list.
This is especially true when:
- Traffic is driven by direct outreach (email, LinkedIn) rather than paid search
- Contacts are already segmented by sequence or persona
- The goal is pipeline generation, not conversion rate optimization at scale