krpano 1.19 - Pre Release 1 - WebVR / MobileVR / Stereoscopic 3D Support (Oculus Rift, Google Cardboard, GearVR, ...)

First prepare your system / device for VR

• Desktop / Oculus Rift
  For using the Oculus Rift DK1 or DK2 with krpano, it would be necessary to use a web browser that supports WebVR. WebVR itself is a new and still experimental browser API and not included in the normal browsers versions yet. There are special builds of Chrome and Firefox with WebVR support available here:
  • Chrome - Chromium WebVR Builds
  • Firefox - MozVR Downloads
    
  For the best possible experience the Chromium WebVR version for Windows would be recommended. This version supports the Oculus Rift Direct-Mode with 75hz rendering, time-warp and low-persistence and this provides a performance and latency like native Oculus Rift applications.
  
• Mobile devices with VR-Headsets (Cardboard, GearVR, VR ONE, ...)
  Basically any Android, iOS or Windows-Phone mobile device with a VR-Headsets would work but there are a few requirements:
  • The mobile device / browser needs to support WebGL.
  • The device needs to have accelerometer and gyroscope sensors.
  • And the device should be pretty powerful / high-end and have a good large screen. The faster the device, the larger the screen and the higher the resolution the better it is for VR.
  Additionally for Android there is also a special Chrome version with WebVR support available: Chromium WebVR Builds (ChromeShell_WebVR.apk). This version provides a slightly better sensor tracking and optionally also the usage of the Android Cardboard API for the screen setup and the lens distortion.
  

krpano VR Demo

• The demo includes normal panos, stereoscopic 3d panos and a panoramic video.
• The navigation is done via 3d hotspots by looking at them for some time.
• For mobile VR usage there is a setup menu (webvr.xml) included to change interactively the IPD (interpupillary distance), the screen size (if the detecting wasn't possible) and the lens parameters of the used VR-headset (including some presets).
  
  Here screenshots of that menu on different devices with different VR-headsets:
  
• Note - this example has a FAKE VR support enabled to allow testing VR also on desktop and tablet systems which normally wouldn't have VR support.
• A reduced example (fewer panos, shorter lower-res video) is included in the download package (in the folder 'viewer/examples/webvr/').
• Testing video: krpano 1.19 with an Oculus Rift DK2

Technical Details

WebVR Plugin

• The krpano VR support will be available via the new WebVR Plugin.
• Just including the WebVR plugin and a 'VR button' would be already enough to add VR support to any pano or tour.
• The WebVR plugin handles everything related to VR support. On desktop system it detects and provides the WebVR API to krpano and on mobile devices it does the device detection, the lens distortion and the sensor tracking.
• To be able to provide and good and accurate head-tracking also on mobile devices the WebVR plugin has a new and very advanced accelerometer and gyroscope sensor tracking. The plugin uses the newer 'devicemotion' browser events for getting almost raw sensor data from the system. The accelerometer will be used for stabilizing/leveling the tilt and roll of the device (by using the gravity acceleration) and the gyroscope for intermediate relative device rotations and the horizontal panning. Together with interpolation and extrapolation of the sensor data (the sensor events and the rendering are happening at different time intervals) it is possible to get a fast, responsive, accurate and smooth movement. Addtionally this works almost equally the same on all devices/browsers - Android Chrome/Opera, Android Firefox, iOS and Windows Phone.
  
  ⇒ Gyro2 Plugin
  • The new sensor tracking is additionally also available as seperate new Gyro 2 Plugin (but with different sensor interpolation/extrapolation settings by default than the WebVR plugin).
  • Here a side-by-side comparison of the old and new gyro plugin: Gyro vs Gyro2.
• A very important aspect of the VR support for mobile devices is the correct physically-sized rendering. The distance between the eyes (the IPD) and the size of the VR-headset and its lens is given and fixed, the rendering itself must adapt to this and be independent of the screen size and the resolution of the device. But as there is no Web API available for getting the physical screen size of a mobile device, the WebVR plugin needs to do a device detection to find the correct size. The different iPhone devices can be detected easily by their properties, but on Android or Windows Phone it is necessary to know the size from somewhere else. The WebVR plugin therefore has an internal list of several device user-agents and their screen sizes (the list can be extended on request). When the device and screen size will be unknown, the WebVR plugin will call the onunknowndevice event - in this case the user should be asked for the screen size of the device.
  
  Here different sized devices with the same sized rendering:
  
• To avoid that the user needs to set his IPD, the VR headset settings (and if necessary the device screen) every time again when reloading the pano or when viewing an other pano or other page, the WebVR plugin provides the possibility to store (and load) the VR related settings locally on the device inside the browsers HTML5 localstorage. This will work also cross-domain for other webpages. That means after changing and saving the settings once, they will be automatically available on all further panos/pages (regardless if on the krpano page or your own one).
• Some tested VR headsets:
  
  • krpano with an Oculus Rift DK2
  • How to use krpano with the GearVR

Stereoscopic 3D Support

• Stereoscopic image support (input side):
  • Cube:

    <image ... stereo="true" stereolabels="1|2">

     <cube url="pano_%s_%t.jpg" />

    </image>

    • The

      %t

      placeholder will resolve to the "stereolabels" values.
    • The right image will be only loaded when stereoscopic rendering is enabled.
  • Sphere/Cylinder: (image or video)

    <image ... stereo="true" stereoformat="TB">

     <sphere url="plugin:video" />

    </image>

    • The "stereoformat" settings defines the left/right frame-packing:
    • stereoformat="TB" - Top/Bottom (the default)
    • stereoformat="SBS" - Side-By-Side
• Stereoscopic rendering support (output side):
  • The stereoscopic rendering will be either enabled automatically when using the WebVR Plugin and calling the enterVR() action ...
  • ... or by manually changing these new settings:

    display.stereo

    display.stereooverlap

    • The "display.stereo" settings can be true or false.
    • The "display.stereooverlap" settings defines the left/right overlapping (in the range 0.0 to 1.0 - 0.0=no overlapping, 1.0=full overlapping).
  • Pre-Release Notes:
    • In stereo rendering mode only distorted image hotspots will be available (no non-distorted hotspots, no textfield hotspots, no polygonal hotspots).
    • In stereo rendering mode the hotspots are only reacting to onover/onout events (no ondown, onup, onclick by mouse or touch).

3D Hotspots

• For navigation / interaction inside a VR pano / tour the usage of hotspots would be the best choose. Especially on mobile devices as there is no other input method like a keyboard or mouse available there.
• To provide more possibilities here, krpano supports a new

  depth

  setting for hotspots to set the 'stereoscopic 3d depth' of the hotspot (default=1000).
• Additionally the hotspots can be moved/translated freely in 3d space by the new

  tx,ty,tz

  hotspot settings.
• For rotation in 3d space the already available

  rx,ry,rz

  hotspot settings can be used.

VR Cursor

• The WebVR Plugin provides the possibility to use any custom hotspot as VR Cursor.
• That cursor hotspot will be displayed automatically above all other hotspots at the center of the screen and act as cursor / pointer. The styling of the hotspot itself (like some custom animations) can be setup and controlled with the normal hotspot attributes and krpano possibilities.
  
• The stereoscopic 3D depth of the cursor will be automatically adjusted to match the 3D depth of other hotspot when hovering them. This makes it eaiser to focus hotspots with different depths.
• When the cursor will be over other hotspots, the onover and onout events of that hotspot itself and additionally also the vr_cursor_onover, vr_cursor_onout events will be called. This could be used to implement an automatic clicking when hovering a hotspots for some time. This way existing tours with hotspot navigation could be used without the need for updating/adapting.

webvr.xml

• To make the usage of VR easier, a lot of the above described functionality is packed into a single xml file. With it would be only necessary to include that webvr.xml file for adding more advanced/complete VR support to a pano or tour.
• The webvr.xml provides:
  • An automatic detection if WebVR or MobileVR is available.
  • An EnterVR / ExitVR button.
  • For mobile VR - a 'Setup VR' button with a setup / settings screen for changing several device / VR related settings. Including lens distortion presets for several different VR headsets.
  • The automatic loading and saving of the VR settings.
  • A VR cursor with an automatic time-out-based clicking and a timing animation.
  • And an automatic hiding of all non-VR screen elements when entering the VR mode.

Why a Pre-Release?

• There are also several non-VR related features that are planed for the final 1.19 version that aren't ready yet.
• The distorted hotspot rendering via WebGL (currently only enabled in Stereo-rendering mode) isn't fully implemented yet (alignment, click/touch events, event flow, children hotspots, ...).
• The lens distortion for mobile VR will be currently simulated by the krpano internal fisheye distortion. This has the advantage of being a fast one-pass rendering, but also the disadvantage of limited correction possibilities (e.g. no chromatic aberration correction). For further versions there are plans to add also an additional post-processing distortion/chromatic-aberration correction step.
• The design/layout of the default templates still need to be adapted for VR support.
• There is no tools/droplet support for handling the input of stereoscopic images yet. The usage of stereoscopic images need to be set-up manually in the xml at the moment.
• The krpano webpage and the documentation aren't updated for VR yet.
• But beside these still-in-work things, the pre-release version itself should be stable and ready to use.

Download

• krpano 1.19 Prerelease

Hi,

Thanks!

Quote

How would be a good way to enable zoom in and out? I tried orbulus and they have zoom while tilting head left or right, i think that works quite well. Have your tried other methods?

I'm not sure about zooming in VR at the moment - with VR headsets like Cardboard which will be hold already like a binoculars zooming might be okay, but with other hands-free headsets with a larger field of view I think the zooming might be more distracting than helpful. In the way Orbulus did it by tilting you might already zoom unintentionally when looking over the shoulder...

But using a 'binoculars' button/spot to switch into a special zoom mode with a zooming HUD and a vignette (smaller fov) might be eventually a possibility...

Quote

Please forgive me if this is a rookie question, but I'm still new to KRpano. What would be the easiest way to update an existing tour (built in 1.18) to add this new VR functionality?

1. First update the older viewer plugin files,
   e.g. this way:
   
   
   1. Start the krpano Tools
   2. Select the Update Tool tab
   3. Drop in your whole pano/tour folder and press Update
2. Then add (manually copy) the new VR related files - the 'webvr.js', 'webvr.xml' and 'webvr_cursor_80x80_17f.png' files from the plugins folder.
3. And finally add
   
   Code

   <include url="%SWFPATH%/plugins/webvr.xml" />

   to your tour/pano xml file.

Then you would already have added VR support.

When using a desktop browser with WebVR support or a capable mobile device, there will be now a 'EnterVR' button for entering the VR mode. For a better integration into your pano/tour UI, you could also use custom buttons - see and edit the webvr.xml for details.

While this would be basically already enough for VR support, it would be still necessary to think about the tour navigation. There are no thumbs or other buttons to switch the panos in this case. When having hotspots in your tour, they would already work automatically, but it might be good to add also these additional loadscene flags - NOPREVIEW|KEEPVIEW|KEEPMOVING for a better experience. The NOPREVIEW flag to start the blending only then the image itself is fully loaded, KEEPVIEW - to keep the current viewing settings and KEEPMOVING to let the user still look around during blending to the next pano. See also the included VR demo for an example about this.

Quote

On my Gear VR + Note 4 it's still a bit buggy with visible latency.
What would be causing this? The FAKE VR support?

No, the FAKE VR support mode is only to allow testing on desktop and tablets.
Can you please describe what you mean with bit buggy? The term 'buggy' indicates an error, but I'm currently not aware about one...

Which browser were you using for testing? The Samsung browser, Chrome (which version?) or the special Chrome WebVR build?

The Samsung browser is based on a older Chrome version where the gyro/accelerometer sensor events were only updated at a 20hz rate and with this the tracking will be more inaccurate of course. But with Chrome version 40 (or higher) that rate will be 60hz and I think the special Chrome WebVR build is internally using the maximum rate, which should be 200hz. With these browsers the latency should as good as currently technically possible on the web - and even be comparable to native Cardboard apps.

Or do you mean/compare it with native GearVR applications? Getting the same latency as there is technically not possible today - the GearVR hardware sensors are not available to external applications like browsers and what's even more important - the 'low persistence' screen mode is also only exclusive to native GearVR apps.

Maybe Samsung will open the GearVR one day for external applications, then WebVR browsers should be able to use it as native. To see what performance and latency would be possible with the right interfaces, try the Chrome WebVR version on Windows with an Oculus Rift DK2.

Best regards,
Klaus

Hi,

Quote from Ibrendie

Buggy isnt the right work, but what I mean is that if I look around its not "stable", its like the image isnt fluid. It even makes me a bit sick :p

That might be also related to the frame-rate - it's not very stable on the Note 4 at the moment, sometimes it's around 60fps but sometimes also only around 45fps. The hotspot rendering also has a big influence on the framerate - without hotspots (like when inside a pano) it should be more fluid than when in the 'intro scene' with the pano selection hotspots. But I'm still optimizing and the next release will be already better here.

Quote from Jann Lipka

I can see hotspot arrows made with tour editor ( in non vr mode ) but hotspots itself are not visible in VR mode .

Only 'distorted' hotspots are currently visible in VR mode.

Change in the vtourskin.xml the 'skin_hotspotstyle' <style> to this (the distorted setting and the loadscene flags are changed there):

<style name="skin_hotspotstyle" url="vtourskin_hotspot.png" scale="0.5" edge="top" oy="0" distorted="true"
       tooltip=""
       onclick="if(linkedscene, skin_hidetooltips(); tween(scale,0.25,0.5); tween(oy,-20,0.5); tween(alpha,0,0.5); looktohotspot(); loadscene(get(linkedscene),null,NOPREVIEW|MERGE|KEEPVIEW|KEEPMOVING,get(skin_settings.loadscene_blend)); skin_updatescroll(); );"
       onloaded="if(skin_settings.tooltips_hotspots, if(linkedscene, copy(tooltip,scene[get(linkedscene)].title); loadstyle(skin_tooltips); ));"
       />

Best regards,
Klaus