About the depthmap support in krpano...
(available since krpano 1.20)

Topics:

• What are Depthmaps?
• Why Depthmaps?
• How to build Depthmaps?
• Depthmap 3D-Modeling: everpano 3D
• Technical Details / More Information
• Depthmap Examples / Demo Tours for testing

What are Depthmaps?

Depthmaps are typically images where the pixel color (typically grayscale) defines the distance (or 'depth') to a certain surface.

Here as example a spherical pano image plus a related spherical depthmap image - the darker the gray-tone in the depthmap image the farther away the surface:

Why Depthmaps?

Knowing the depth of the pixels makes it possible to generate new viewpoints from an image. That means viewing the image from a different camera position than it was originally captured.

And that allows doing many nice things:

• Generating synthetic Stereo-views with depth perception.
  • Very helpful and very important for good VR viewing.
  • Can be used also for Anaglyph (red/cyan glasses) stereo-viewing.
  • No need for shooting and stitching stereo panos.
  • Depth perception even when looking up or down (that's not possible using normal-stereo pano images).
• 6DOF Movement
  • Six Degree of Freedoms: additionally to the normal rotation in a pano, now also a translation is possible.
  • Positional head-tracking for VR.
  • Perspective changes, e.g. viewing the pano from 'outside' ('Dollhouse'-effect).
  • Walking inside the pano a bit (depending on the image content).
• 3D-Transitions
  • Doing 3D-translations during the pano-blending.
• Measurements
  • By having the depth / distance information and a reference size, it will be possible to do real measurements in the pano image.

The new generated views from other perspectives might be not perfect (depending on the actual image contents), parts that were previously occluded will be 'wrong' of course, but the effect itself, especially the depth perception in VR will be very convincingly.

Here a quick video preview for what depthmaps can be used: Try this example yourself here: Little Temple of Abu Simbel (also included in the download package)

How to build Depthmaps?

There are several different possibilities how to generate depthmaps:

• With specialized software from different perspective images, e.g. from stereo images.
  • But can be difficult, content-dependent and today still very error-prone.
  • Might get better in future with better software.
  • There are already some (expensive) cameras available that can generate panoramic-depthmap images automatically.
• By Laser-Scanning / LIDAR
  • Rare and probably expensive (sorry, no real own experiences here).
• Drawing by hand
  • For 2D images this is possible, but for panos almost impossible (or at least very difficult).
• With 3D-Modeling
  • Can be done easy and quickly with the right software.
  • Can be done for existing panos!
  • Because of 3D geometry flat surfaces can be rendered very accurately (depthmap-images have limitations here).
  • The 3D model itself do not even need to be very accurate for a good 3D effect.
  • The storage needed for 3D models is typically very low, so they are quick to load.
  • So that will be the focused method here now.

Depthmap 3D-Modeling: everpano 3D

There is a new special krpano-based software available for modeling depthmaps:


The everpano 3D software is developed by Jordi Vallverdú (a long-term and experienced krpano user) and allows a quick and easy modeling of 3D-geometry for panoramic images. Additionally the software can link several panos and build automatically whole 3D-tours with nice 3D-transitions effects.

Here an introduction video of the everpano 3D software for how to draw the geometry:
For more tutorial videos please see here on the everpano 3D website.

The software is available for purchase here on the krpano site and also on the everpano site. The depthmap feature requires a krpano license upgrade. That upgrade is available separately or also bundled with a everpano 3D license.

Some background information:

• Originally the depthmap support was in development only as 'very special' feature and with several limitations - e.g. only spherical-pano images, only single-res and only depthmap-images were possible. This because that was relatively simple to implement and depthmap-images were generally not easily available or easy to made, so the usage cases would have been very limited anyway.
• But then by chance Jordi Vallverdú contacted me that about the krpano depthmap support in the next release and that he is developing a tool for drawing depthmaps.
• The depthmaps generated by his tool were showing great results and a lot of potential. With it everyone would be able to build depthmaps and 3D-panos. And depthmaps could be added to all existing panos - no need for special pano-shooting or special hardware!
• So I've started to improve the depthmap support in many ways to make it better and more generally usable. It can be used now also with cubical-images, with multires-images, use depthmap-images or even directly a 3D-geometry, link panos with 3D-translation transitions, have different rendering modes, support VR-viewing with positional-tracking, has special hotspot-support and many things more.
• That was originally not planned and caused a lot of internal changes and therefore was one of the reasons why the development of this release took so long. But I hope it was worth the waiting time.

Technical Details / More Information

The krpano viewer supports two sources for the depth information:

• Depthmap-Images
  • Typically grayscale images where the gray-tone defines the depth:
    
    
    
    
  • As there is no standard that defines what gray-tone related to which depth, there are additional settings available to customize the depthmap-interpretation for getting the correct offsets and scales.
  • The depthmap images don't need to have the same resolution as the pano image, resolutions like 1024x512 should be already enough. More would be waste of resources.
  • The browsers are supporting only a 8-bit color/grayscale resolution, even when the input image might be a 16-bit PNG image. So the depth accuracy is limited to 256 (=2^8) different depth-levels. For some cases, this limitation can become problematic and cause visible 'ripples' when viewing the pano from other viewpoints.
• 3D-Models
  • Additionally krpano can also use 3D-models / 3D-geometries for getting the depth for a panoramic image:
    
    

    

    
  • 3D-models don't have the limitations of depthmap-images.
  • They are very accurate and typically also faster to load (because the files are smaller).
  • krpano supports 3D-models in the STL file-format (ASCII format only),
  • and in a simple, easy to parse and fast to load special DEPTH file-format.
  • The 3D-geometry will be either used to generate an internal depthmap image (but with a high-quality 24bit depth resolution) or used directly and the panoramic image projected onto it. Depending on the usage-case and the image content the one or the other mode will be better.

For movements inside depth-mapped panos these variables are available:

• view.tx / view.ty / view.tz
  • Move the viewer /camera position.
  • Coordinate system:
    • X: left(-) / right(+)
    • Y: up(-) / down(+)
    • Z: in(+) / out(-)
• view.ox / view.oy / view.oz
  • Add an additional offset, but after doing the viewer looking rotation.
  • Could be used for 'outside' / 'dollhouse' views.
• image.ox / image.oy / image.oz
  • Move the position of the pano-image itself.
  • Can be used for tours with depthmap panos.
  • Use the ox/oy/oz to define the absolute/world positon of a pano, and for moving to an other pano, tween the view tx/ty/tz to them.
  • The pano will be viewed from its center when the view.tx/ty/tz and the image.ox/oy/oz are the same.

To make to easier to place hotspots in such 3D-world build by depth-mapped panos these new hotspot settings are available:

• hotspot[name].tx / hotspot[name].ty / hotspot[name].tz
  • Move the hotspot in 3D space.
  • When using these setting it can make be helpful to set the hotspot[name].depth setting to 0 to avoid a combined translation.
• hotspot[name].torigin
  • Defines the origin of the hotspot position: should it be absolutely positioned or be relative to the current pano or relative to the current view.
• hotspot[name].depthbuffer
  • Define if the depth of the pano and the depth of the hotspot should be 'merged' during rendering.

Depthmap Examples / Demo Tours for testing

Some examples showing the possibilities when using depthmaps:
  Little Temple of Abu Simbel

• A full-featured demo tour using depthmaps.
• It demonstrates how to view a pano dynamically from new perspectives, e.g. to show wall friezes without a perspective distortion.
• Additionally the depthmaps allow nice 3D-transitions.
• The tour offers automatic- and manually-generated hotspots for navigation.
• The depthmaps were generated by the everpano 3D tool, but the tour itself manually designed and coded.
• A little helper tool for finding 3D viewpoint and hotspot positions is also included.

  Gravina Apartment - everpano 3D tour with navigator plugin

• A whole tour fully generated by the everpano 3D tool.
• For navigation in the tour the everpano Navigator plugin is used.
• That plugin allows easy navigating by automatically finding the pano next to the point where the user had clicked - and when there is no pano near, a nice 3D-zooming-animation indicates that.
• Here for comparison and more testing the same tour with hotspot navigation: Gravina Apartment - Example 2

  Examples of Depthmap Images

• Here a set of example panos with automatically depthmaps generated.
• The depthmap images here were either by created the camera itself or by rendering software or by stereo-to-depth software.

  Depthmap-Image Helper Tool

• A small helper tool for finding the correct offset and scale settings for depthmap images.
• The values for these settings depends how the depthmap-image was created and mapped.

More Depthmap Demos at everpano.com:  

Hi,

Quoted from "igor.socha"

Now the first question: is it possible to create 3D Depthmaps also for uneven scenes (without any straight lines etc.) such as caves, mines, or other underground places. And what about drone scenes?

Technically yes, e.g. create points at different heights and then connect them by polygons.
Maybe see 'Lesson 7 - Getting more complex' here:
https://everpano.com/crash_course.php

Here some examples with partially uneven parts:
https://everpano.s3.eu-central-1.amazona…urch/index.html
https://everpano.s3.eu-central-1.amazona…rosa/index.html

Quoted from "dreidesign"

I got a question regarding the depth map. How can you use a depth map with a 6x1 rendered cubestrip? Do you convert the rendering with the normal "make droplet" and also the depth map as a "second pano" and then link the url to that folder? Or how does that work?

The depthmap feature is supported with cube, multires-cube and spherical images.
Here an overview of the supported image-types by krpano:
https://krpano.com/docu/xml/#image

When you have a 6x1 cubestrip you could use that on the krpano MAKE VTOUR droplets (they will convert it automatically to separated cube images) and then use the everpano 3D to load that tour.

Or build the tour and add the <depthmap> element manually in the xml:
https://krpano.com/docu/xml/#image.depthmap

Quoted from "dreidesign"

Is there anymore information on how a proper .stl has to look like? I'm trying for hours different exports of a simple room modeled in 3dsmax and I cant get it working. Its seems that the floor height is totally off, but I dont know how to fix that. I played with the scale parameter but nothing works.

The stl would be need to centered at the position where the pano camera is.
About scale - a unit of 100 in the stl relates to 100cm in krpano.

The basic idea is that the stl file itself should be made in a way that it would fit the pano image, but krpano also offers several (undocumented) additional settings for customizing the stl loading.

Here the available settings:
But finding the correct settings manually will be probably difficult...

Maybe make a separate post with your example, then we can try finding the correct settings for it there.

Quoted from "jeromebg"

Is this the right place for Everpano support ?

Good question - I will make a separate sub-forum for everpano 3D questions.

Best regards,
Klaus

Quoted from "dreidesign"

Thx I got that. But the "real" question was which kind of "format" the depthmap has to be. I tried to convert the depthmap to a cube, but then its not getting recognized. So it seems that the pano itself can have "cube, multi-res or spherical" but the depthmap as an image has to be spherical, is that correct?

Right, the depthmap-image need to be spherical.

Quoted from "dreidesign"

That makes sense. I tried it again, setting the pivot of the geometry to exact camera position x/y/z. But the issue still persists, the bottom of the room seems to be at the height of the camera. I tried setting the pivot of the geometry to the floor but that didnt change a thing.

Sorry, but I would need more information - please make a separate post with your example for analyzing it.

Quoted from "ily"

are there any demo version for everpano?

Yes, you can download them here:
https://everpano.com/downloads