Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> The wrong question to ask is: Is A better than B, given some confidence threshold?

That's a perfectly fine question so long as there are three acceptable answers: Yes, No and Unclear.



Suppose A is significantly better than the control. And B is not significantly better than the control. Does that imply A is significantly better than B? No. Statistical differences do not the follow the same logic as normal differences.

It feels so pointless to bin probabilities into yes, no, and unclear. We're throwing away information with value when we compress the problem this way. And I think it leads to more misinterpretations.


In A/B testing, either A or B is the control.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: