They do leave you and go into the relay. The swap happens on finished contracts.

The stealing occurs in the relays not in the contract.

I think you have a misunderstanding of how the protocol works. At no point do your assets "go into the relay." The relay only takes your signed orders (not the actual assets), and the tokens literally do not leave your wallet until the time the trade is executed. No trades can be executed unless they have a signature generated by your private key.