this sounds insane. but im not familiar with all the concepts. it seems that now instead of alice and bob exposing their real keys, you have node a, b, c and d exposing theirs during bob handshake... which will lead to sociopath nodes that wont be part of this, but rely on nodes that accepts it for them.
also, if the message is encrypted, why not use a "torrent" of all the messages being shared and discarded after a while (no need for offline messages anyway) and each node decrypts its own as they please but relay all of them in the same stream? no way to know who could decode it. and it was delivered to everyone. much like a bitcoin transaction, but without the metadata.
it would probably be vulnerable to a dos by flooding it with bogus, expensive to try to read, requests... but will be the most reliable and private you can get
But you don't have to actually know them. If they posted their Tox ID on 4chan, which happens all the time, and they accept my friend request, I would have their IP address, correct?
Ip addresses change. Sometimes they don't even work for P2P. Without ICE or something its hard to even know your public IP address programmatically. So often some kind of proxy or server is necessary for most people. True decentralized connections are a sketchy proposition.