What a p-value actually means for your A/B tests
A p-value is closer to a weirdness score than a probability that you are right. Here is how to read one, why the 0.05 threshold is arbitrary, and how to sanity-check sample size before you act on an A/B test.