Its about 20-40% depending on the GPU, from my tests.
And only very recent builds of torch 2.1 (with dynamic input) work properly, and it still doesn't like certain input changes, or augmentations like controlnet.
AIT is the most usable compiled implementation I have personally tested, but SHARK (running IREE/MLIR/Vulkan) and torch-mlir are said to be very good.
Hidet is promising but doesn't really work yet. TVM doesn't have a complete implementation outside of the WebGPU demo.
And only very recent builds of torch 2.1 (with dynamic input) work properly, and it still doesn't like certain input changes, or augmentations like controlnet.
AIT is the most usable compiled implementation I have personally tested, but SHARK (running IREE/MLIR/Vulkan) and torch-mlir are said to be very good.
Hidet is promising but doesn't really work yet. TVM doesn't have a complete implementation outside of the WebGPU demo.