Interesting idea. I've the bash built-in "pushd" and "popd" for a long time for similar reasons, but those effectively limit you to treating the history as a stack rather than an arbitrary list.
yeah, it can be annoying that the directory stack sequence keeps changing, which means that sometimes you can't easily re-run commands from your history if you've pushd since you last ran the command
e.g., a command like "tar cfz bak.tgz ~3 ~2/subdirectory ~1" would mean something else once you've "pushd +3".
I always find it hard to undo my sequence changes to get the stack in the same order as before, so usually end up typing "dirs -v" a lot (or its alias) and renumbering my ~N references.
I frequent systems where tcsh, for historical reasons, is the default shell, and for all its faults it does have some nice pushd settings that make directory browsing with 'cd +<number>' and 'dirs' very convenient.