Have you ever launched a campaign with a thrilling result out the gate, only to watch your numbers tank for a few days after that?
Have you ever shut off an ad after a particularly bad day?
Let’s talk about the importance of Statistical significance.
If you’re running an A/B test to optimize a landing page, do you shut it off after 100 visitors have come and one variable is a CLEAR winner? Ab so lutely… NOT!
You really want 10k visitors. And you want them within a two week period.
You see, within that 10k user base are many, many different people. When you close your eyes, can you envision 10,000 people? I cannot. And I’ve been trying for five years.
The fact is that our puny human brains fry out when we start thinking in this number range. So you need to understand… RESULTS WITHOUT STATISTICAL SIGNIFICANCE MEAN JACK SHIT.
The real point here is don’t let day to day performance thrill or spook you. Say you’re running an ad to 1,000,000 audience size and can get 10k reach per diem. You can’t imagine 10k people. One million is ten thousand ten thousands. Stop trying to imagine it you fucking idiot. You’re like me, a peanut brained human. Just trust in the fact that within a 10k sample size is enough variability to get an okay test result. But when you multiple THAT sample size by another 10k, your hypothesis will truly be put to the test. Look for profit over time, not short bursts of incredible performance. If anything, let a short burst of amazing performance steel you for the inevitable day of NO PERFORMANCE. Take the Buddhist approach. Allow the ups and downs to wash over you.
I see a lot of people posting about performance issues, or a few days of no conversions. Just because you have conversions one day does not guarantee them the next day. You are marketing to huge swathes of people. If your campaign doesn’t work at a statistically significant level, then OK, it’s time to revisit your creative and audience. But if you don’t let the performance fail long enough, you won’t ever really learn what DOESN’T work. With perseverance, a little risk tolerance, and letting your failures become confirmed duds, you’ll have bigger success in the long run. (year over month).