if the panorama was projected on a sphere that problem wouldn't occur
That's not correct, a sphere would be even more worse. You could test that by using a <sphere> image.
To avoid too strong distortions the geometry behind the pano-image should match roughly the content when moving out of the pano-projection center.
I'm already planing to add a depth-map support for distorting the pano-sphere-geometry, but then the main problem will be how to generate that depth-map...
Best regards,
Klaus