On first few AMD's original AMD64 microarchitectures, 16 bit segments (both vm86 and 16b protected mode) were not working when the CPU was in long mode. The fact that they do on many modern CPUs is one of the Intel's extensions to AMD64. Also this is the reason why 64b windows do not support NTVDM and 16b windows applications, while the current hardware can support that. Another of these backward compatibility Intel's extensions concern whether x87-style FPU instructions are valid in long mode 64b code segment.

But you can leave long mode with a couple instructions. Was that not possible at some point? A good old triple fault should always work, shouldn't it?

This discussion is interesting, I had read earlier that long mode was a point of no return, I guess that's not true after all?

I guess no v86 mode still makes it a dealbreaker for NTVDM type applications. Nobody wants to go back to a windows 95 style OS where a 16bit application runs unprotected and can stomp over anything and bring the machine to a halt. (For example, in win95 you could run debug.com and overwrite the first blocks of physical memory with zeros, which would immediately crash everything since that's where interrupt vectors like the IRQ timer is stored)

Would it be possible to bounce from 64bit long mode to 16bit real mode with some trusted code that bounces on to a mini kernel in 32bit protected mode that hosts v86 16bit applications, and then back again?

I heard that too, but then I actually looked at the manual, and it's very simple to both enter and exit long mode.

As far as I know it's possible to do all that and have a very-mixed OS.

The easiest way to host 16 bit applications now is to use the virtualisation extensions to run a VM in real mode.

AFAIK that is not true as far as I know.

You can jump into real mode from long mode, but you can't run real mode programs while inside long mode.