> people can decide they don't want to associate with you or me for reasons we don't find justifiable

Unless your decision involves a protected group, in which case this is a little more nuanced.

> I can throw a party

Professional organizations putting together conferences do not get the same liberties as individuals organizing family events.