They do pattern matching against lots of pieces of information. It could be as simple as a local utility company selling their customer list with phone number and address attached, then a retail website has your address from when you bought something, now your phone number gets linked.
It doesn't matter if you don't give your phone number to many companies, it only takes one.
It doesn't matter if you don't give your phone number to many companies, it only takes one.