Even if Signal's server can't do it, what's to prevent a client from using a really large list of contacts? (Say, millions of contacts captured via a security leak?)
Assuming OWS doesn't rate limit, all that would do is let you tell what phone #s use signal. Which is slightly sensitive info, but radically less so than "who is communicating with whom", which is what the older method could leak.
But to avoid minimizing it, there is a scenario I've heard where it matters: crazy ex-boyfriend discovers you're using a new messaging service, just because he still has your phone number in his contacts.
Ideally, nobody should know you're using a new messaging service unless you've given explicit consent to share that info with them.
