I don’t think it’s impossible if certain roads/highways or lanes become marked as autonomous only. Just as pedestrians and bicycles are banned from using high-speed roads, manually-operated cars could be banned from using certain roads/lanes, removing a lot of variables that manually-operated cars bring.
How is the ability to leave at your own time, have your own dedicated cabin, and ability to stop nearly anywhere, even in the same ballpark as a Tram...?
I wish governments would consider the death and injury toll of manually driving, and fastlane (pun intended) infrastructure for self driving cars, such as specific roads and lanes.