Hmm yeah that is interesting.
After reading some more, I suspect you might be able to replicate the Panotour way by doing the same or similar to what you suggested: adding a partial video sphere as a distorted hotspot and lining it up with the pano perfectly, but I'm out of my depth here. I'm not a photographer, just a web developer.
Interested to learn more.