Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The biggest issue with those three hypotheses is one of them, the noticing the button, almost certainly isn't being tested. But, how the test goes will inform how people think about that hypothesis.


Rate of traffic on the checkout page, divided by overall traffic.

We see a lot of ghosts in A/B testing because we are loosey goosey about our denominators. Mathematicians apparently hate it when we do that.


That doesn't test noticing the button, that tests clicking the button. If the color changes it is possible that fewer people notice it but are more likely to click in a way that increases total traffic. Or more people notice it but are less likely to click in a way that reduces traffic.


Good observation that the noticing doesn’t get tested.

Would there be any benefit from knowing the notice rate though? After all, the intended outcome is increased sales by clicking.


This is what I was driving at in my original comment - the intermediary steps are not of interest (from the POV of the hypothesis/overall experiment), so why mention them at all.


Probably not, but then that hypothesis should not be part of the experiment.




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: