The problem was that SSH linked to libsystemd. Libsystemd linked to xz. So the library got pulled in transitvly. Here are two recent commits that fix the problem by reducing dependencies:

https://salsa.debian.org/ssh-team/openssh/-/commit/cc5f37cb8...

https://github.com/systemd/systemd/pull/31550