Totally weird. Unless you set height and width to 100% in a style element inside the pano div tag (inline), iphone_fullscreen_swipe won't work. If you set these in a stylesheet instead, the div will size as expected but iphone_fullscreen_swipe doesn't work. I can't see anything in the javascript that would cause this so I suspect it has to do with the oniphonefullscreen event not firing.
Also, I found that the viewport meta tag must include "viewport-fit=cover" although that makes a little more sense.
Thoughts?