krpano Depthmap Support


  • About the depthmap support in krpano...
    (available since krpano 1.20)


    Topics:

    What are Depthmaps?

    Depthmaps are typically images where the pixel color (typically grayscale) defines the distance (or 'depth') to a certain surface.

    Here as example a spherical pano image plus a related spherical depthmap image - the darker the gray-tone in the depthmap image the farther away the surface:


    Why Depthmaps?

    Knowing the depth of the pixels makes it possible to generate new viewpoints from an image. That means viewing the image from a different camera position than it was originally captured.

    And that allows doing many nice things:
    • Generating synthetic Stereo-views with depth perception.
      • Very helpful and very important for good VR viewing.
      • Can be used also for Anaglyph (red/cyan glasses) stereo-viewing.
      • No need for shooting and stitching stereo panos.
      • Depth perception even when looking up or down (that's not possible using normal-stereo pano images).
    • 6DOF Movement
      • Six Degree of Freedoms: additionally to the normal rotation in a pano, now also a translation is possible.
      • Positional head-tracking for VR.
      • Perspective changes, e.g. viewing the pano from 'outside' ('Dollhouse'-effect).
      • Walking inside the pano a bit (depending on the image content).
    • 3D-Transitions
      • Doing 3D-translations during the pano-blending.
    • Measurements
      • By having the depth / distance information and a reference size, it will be possible to do real measurements in the pano image.
    The new generated views from other perspectives might be not perfect (depending on the actual image contents), parts that were previously occluded will be 'wrong' of course, but the effect itself, especially the depth perception in VR will be very convincingly.

    Here a quick video preview for what depthmaps can be used:
    Try this example yourself here: Little Temple of Abu Simbel (also included in the download package)

    How to build Depthmaps?

    There are several different possibilities how to generate depthmaps:
    • With specialized software from different perspective images, e.g. from stereo images.
      • But can be difficult, content-dependent and today still very error-prone.
      • Might get better in future with better software.
      • There are already some (expensive) cameras available that can generate panoramic-depthmap images automatically.
    • By Laser-Scanning / LIDAR
      • Rare and probably expensive (sorry, no real own experiences here).
    • Drawing by hand
      • For 2D images this is possible, but for panos almost impossible (or at least very difficult).
    • With 3D-Modeling
      • Can be done easy and quickly with the right software.
      • Can be done for existing panos!
      • Because of 3D geometry flat surfaces can be rendered very accurately (depthmap-images have limitations here).
      • The 3D model itself do not even need to be very accurate for a good 3D effect.
      • The storage needed for 3D models is typically very low, so they are quick to load.
      • So that will be the focused method here now.

    Depthmap 3D-Modeling: everpano 3D

    There is a new special krpano-based software available for modeling depthmaps:
    everpano 3D

    The everpano 3D software is developed by Jordi Vallverdú (a long-term and experienced krpano user) and allows a quick and easy modeling of 3D-geometry for panoramic images. Additionally the software can link several panos and build automatically whole 3D-tours with nice 3D-transitions effects.

    Here an introduction video of the everpano 3D software for how to draw the geometry:
    For more tutorial videos please see here on the everpano 3D website.

    The software is available for purchase here on the krpano site and also on the everpano site. The depthmap feature requires a krpano license upgrade. That upgrade is available separately or also bundled with a everpano 3D license.

    Some background information:
    • Originally the depthmap support was in development only as 'very special' feature and with several limitations - e.g. only spherical-pano images, only single-res and only depthmap-images were possible. This because that was relatively simple to implement and depthmap-images were generally not easily available or easy to made, so the usage cases would have been very limited anyway.
    • But then by chance Jordi Vallverdú contacted me that about the krpano depthmap support in the next release and that he is developing a tool for drawing depthmaps.
    • The depthmaps generated by his tool were showing great results and a lot of potential. With it everyone would be able to build depthmaps and 3D-panos. And depthmaps could be added to all existing panos - no need for special pano-shooting or special hardware!
    • So I've started to improve the depthmap support in many ways to make it better and more generally usable. It can be used now also with cubical-images, with multires-images, use depthmap-images or even directly a 3D-geometry, link panos with 3D-translation transitions, have different rendering modes, support VR-viewing with positional-tracking, has special hotspot-support and many things more.
    • That was originally not planned and caused a lot of internal changes and therefore was one of the reasons why the development of this release took so long. But I hope it was worth the waiting time.

    Technical Details / More Information

    The krpano viewer supports two sources for the depth information:
    • Depthmap-Images
      • Typically grayscale images where the gray-tone defines the depth:

      • As there is no standard that defines what gray-tone related to which depth, there are additional settings available to customize the depthmap-interpretation for getting the correct offsets and scales.
      • The depthmap images don't need to have the same resolution as the pano image, resolutions like 1024x512 should be already enough. More would be waste of resources.
      • The browsers are supporting only a 8-bit color/grayscale resolution, even when the input image might be a 16-bit PNG image. So the depth accuracy is limited to 256 (=2^8) different depth-levels. For some cases, this limitation can become problematic and cause visible 'ripples' when viewing the pano from other viewpoints.
    • 3D-Models
      • Additionally krpano can also use 3D-models / 3D-geometries for getting the depth for a panoramic image:
      • 3D-models don't have the limitations of depthmap-images.
      • They are very accurate and typically also faster to load (because the files are smaller).
      • krpano supports 3D-models in the STL file-format (ASCII format only),
      • and in a simple, easy to parse and fast to load special DEPTH file-format.
      • The 3D-geometry will be either used to generate an internal depthmap image (but with a high-quality 24bit depth resolution) or used directly and the panoramic image projected onto it. Depending on the usage-case and the image content the one or the other mode will be better.
    For movements inside depth-mapped panos these variables are available:
    • view.tx / view.ty / view.tz
      • Move the viewer /camera position.
      • Coordinate system:
        • X: left(-) / right(+)
        • Y: up(-) / down(+)
        • Z: in(+) / out(-)
    • view.ox / view.oy / view.oz
      • Add an additional offset, but after doing the viewer looking rotation.
      • Could be used for 'outside' / 'dollhouse' views.
    • image.ox / image.oy / image.oz
      • Move the position of the pano-image itself.
      • Can be used for tours with depthmap panos.
      • Use the ox/oy/oz to define the absolute/world positon of a pano, and for moving to an other pano, tween the view tx/ty/tz to them.
      • The pano will be viewed from its center when the view.tx/ty/tz and the image.ox/oy/oz are the same.
    To make to easier to place hotspots in such 3D-world build by depth-mapped panos these new hotspot settings are available:

    Depthmap Examples / Demo Tours for testing

    Some examples showing the possibilities when using depthmaps:
    Little Temple of Abu Simbel
    • A full-featured demo tour using depthmaps.
    • It demonstrates how to view a pano dynamically from new perspectives, e.g. to show wall friezes without a perspective distortion.
    • Additionally the depthmaps allow nice 3D-transitions.
    • The tour offers automatic- and manually-generated hotspots for navigation.
    • The depthmaps were generated by the everpano 3D tool, but the tour itself manually designed and coded.
    • A little helper tool for finding 3D viewpoint and hotspot positions is also included.
    Gravina Apartment - everpano 3D tour with navigator plugin
    • A whole tour fully generated by the everpano 3D tool.
    • For navigation in the tour the everpano Navigator plugin is used.
    • That plugin allows easy navigating by automatically finding the pano next to the point where the user had clicked - and when there is no pano near, a nice 3D-zooming-animation indicates that.
    • Here for comparison and more testing the same tour with hotspot navigation: Gravina Apartment - Example 2
    Examples of Depthmap Images
    • Here a set of example panos with automatically depthmaps generated.
    • The depthmap images here were either by created the camera itself or by rendering software or by stereo-to-depth software.
    Depthmap-Image Helper Tool
    • A small helper tool for finding the correct offset and scale settings for depthmap images.
    • The values for these settings depends how the depthmap-image was created and mapped.

    More Depthmap Demos at everpano.com:


  • Hi Jordi,
    first, let me thank you too for your excelent input to krpano comunity!
    Now the first question: is it possible to create 3D Depthmaps also for uneven scenes (without any straight lines etc.) such as caves, mines, or other underground places. And what about drone scenes?
    Thanks!

    Best regards,
    Igor Socha

  • Thank you very much for the release.

    I got a question regarding the depth map. How can you use a depth map with a 6x1 rendered cubestrip? Do you convert the rendering with the normal "make droplet" and also the depth map as a "second pano" and then link the url to that folder? Or how does that work?

  • Downloading the everpano 3D software from the everpano website does not work for me, I have tried different computers and browsers. Was someone else successful or does anyone know where to get the software? Best regards, Lothar

  • Hi,

    I was lucky to have the chance to play a bit with the depthmap tools already.
    Jordi really did a great job and good support on me, with my 1000 questions and suggestions.
    Sorry Jordi *wink*

    I'll show you my first depthmap attempt i made some weeks ago.
    https://www.virtualtuur.com/krpano/120/depth/poohouse/

    simple, but i immediately understood the power of this great feature.

    Tuur *thumbsup*

  • Is there anymore information on how a proper .stl has to look like? I'm trying for hours different exports of a simple room modeled in 3dsmax and I cant get it working. Its seems that the floor height is totally off, but I dont know how to fix that. I played with the scale parameter but nothing works.

    Only way which I got working is the normal depth map, but only as a spherical pano.

  • Is this the right place for Everpano support ?
    Congrats on what can be done with Krpano !!! Nice Ui
    I don't know if it's bugs or me but tried severals times with same issues :
    I first link the panos with at least 2 references by panos, then add a cube in the first one, but in the other I only get a vertical pane instead ? so when I generate a preview I can only see the part of the vertical pane, all the rest is black...
    Undo & Redo seems not to work either ?
    Maybe adding numbers to cubes or any other shapes could be cool because on some backgrounds colors are not really evident to recognize ?
    And last thing, a spherical and cylindrical drawing tool would be a great addition !
    Thx !

  • Hi,

    Now the first question: is it possible to create 3D Depthmaps also for uneven scenes (without any straight lines etc.) such as caves, mines, or other underground places. And what about drone scenes?

    Technically yes, e.g. create points at different heights and then connect them by polygons.
    Maybe see 'Lesson 7 - Getting more complex' here:
    https://everpano.com/crash_course.php

    Here some examples with partially uneven parts:
    https://everpano.s3.eu-central-1.amazonaws.com/3d/church/index.html
    https://everpano.s3.eu-central-1.amazonaws.com/3d/vallferosa/index.html


    I got a question regarding the depth map. How can you use a depth map with a 6x1 rendered cubestrip? Do you convert the rendering with the normal "make droplet" and also the depth map as a "second pano" and then link the url to that folder? Or how does that work?

    The depthmap feature is supported with cube, multires-cube and spherical images.
    Here an overview of the supported image-types by krpano:
    https://krpano.com/docu/xml/#image

    When you have a 6x1 cubestrip you could use that on the krpano MAKE VTOUR droplets (they will convert it automatically to separated cube images) and then use the everpano 3D to load that tour.

    Or build the tour and add the <depthmap> element manually in the xml:
    https://krpano.com/docu/xml/#image.depthmap


    Is there anymore information on how a proper .stl has to look like? I'm trying for hours different exports of a simple room modeled in 3dsmax and I cant get it working. Its seems that the floor height is totally off, but I dont know how to fix that. I played with the scale parameter but nothing works.

    The stl would be need to centered at the position where the pano camera is.
    About scale - a unit of 100 in the stl relates to 100cm in krpano.

    The basic idea is that the stl file itself should be made in a way that it would fit the pano image, but krpano also offers several (undocumented) additional settings for customizing the stl loading.

    Here the available settings:

    Code
    <depthmap url="model.stl" stlaxis="zxy" stlrotate="0" stlscale="1,1,1" stlmove="0,0,0" stlcull="front" />

    But finding the correct settings manually will be probably difficult...

    Maybe make a separate post with your example, then we can try finding the correct settings for it there.


    Is this the right place for Everpano support ?

    Good question - I will make a separate sub-forum for everpano 3D questions.

    Best regards,
    Klaus

  • Hi all,

    Thanks for your kind words !

    is it possible to create 3D Depthmaps also for uneven scenes

    As you can see all the tools are about to draw straight lines, but one of the main concepts here would be to try to simplify shapes. Then you can get something like this :

    You can check the project here (using AWSDQE you can walk around)


    what about drone scenes

    Here you can do a bit the same. Here it's an example of an aerial view :

    and the project here


    Downloading the everpano 3D software


    I was having some issues sending emails that's why some users could not download, but that should be solved now.
    There's also a new build, which can handle special characters for chinese alphabets

    I first link the panos with at least 2 references by panos


    You should just be using one reference per pano pair. you can check this video, where a simple model is being build.


    when I generate a preview I can only see the part of the vertical pane


    in the preview mode you can toogle depthmap render mode with key 2 & 3 (2 = depthmap and 3 = 3dmodel)

    Undo & Redo seems not to work


    That's right, I'm already working with the implementation of such feature, currently this undo/redo would work when filling inputs in the software. Hopefully soon you can do ctrl+z.

    Maybe adding numbers to cubes or any other shapes could be cool because on some backgrounds colors are not really evident to recognize

    There's a shortcut for that. If you press shift+a then the polygons get highligthed, that's also very useful to check if the triangulation is correct
    You can find more info in the documentation here (8.- Check triangulation)

    a spherical and cylindrical drawing tool would be a great addition


    Definitely, they are already implemented, but when you want to edit a cylinder that has 20 divisions, that became very difficult, so for working with such geometry I need to first implement more advanced manipulation tools.

    I was lucky to have the chance to play a bit with the depthmap tools already.


    Thanks Tuur, you did report important points !!

    Thanks to Klaus for such a release, all the implementations that krpano 1.20 has to make that works in every single device, with such fluency are just impressive !

    Kind regards

  • Thanks a lot for your answers and nice examples!

    Btw. for now, everpano doesn´t support more floors - so what would you suggest to do if i have 2 floors?
    just do it as two separate projects and link the scenes normally in krpano?

    Best regards,
    Igor Socha

  • Can I request a walkthrough video on how to efficiently model a typical residential property - i.e. a room with sofas, chairs, tables, various decorative objects etc.

    Also the best way to model a typical L shaped open plan space that might have dining/living/kitchen spaces all together?

    Thanks
    GK


  • The depthmap feature is supported with cube, multires-cube and spherical images.
    Here an overview of the supported image-types by krpano:
    https://krpano.com/docu/xml/#image

    When you have a 6x1 cubestrip you could use that on the krpano MAKE VTOUR droplets (they will convert it automatically to separated cube images) and then use the everpano 3D to load that tour.

    Or build the tour and add the element manually in the xml:
    https://krpano.com/docu/xml/#image.depthmap


    Thx I got that. But the "real" question was which kind of "format" the depthmap has to be. I tried to convert the depthmap to a cube, but then its not getting recognized. So it seems that the pano itself can have "cube, multi-res or spherical" but the depthmap as an image has to be spherical, is that correct?


    The stl would be need to centered at the position where the pano camera is.
    About scale - a unit of 100 in the stl relates to 100cm in krpano.

    The basic idea is that the stl file itself should be made in a way that it would fit the pano image, but krpano also offers several (undocumented) additional settings for customizing the stl loading.

    Here the available settings:

    Code

    But finding the correct settings manually will be probably difficult...

    Maybe make a separate post with your example, then we can try finding the correct settings for it there.

    That makes sense. I tried it again, setting the pivot of the geometry to exact camera position x/y/z. But the issue still persists, the bottom of the room seems to be at the height of the camera. I tried setting the pivot of the geometry to the floor but that didnt change a thing.

    The not documented additional settings, doesnt seem to have any effect when I change them.

  • Thx I got that. But the "real" question was which kind of "format" the depthmap has to be. I tried to convert the depthmap to a cube, but then its not getting recognized. So it seems that the pano itself can have "cube, multi-res or spherical" but the depthmap as an image has to be spherical, is that correct?

    Right, the depthmap-image need to be spherical.


    That makes sense. I tried it again, setting the pivot of the geometry to exact camera position x/y/z. But the issue still persists, the bottom of the room seems to be at the height of the camera. I tried setting the pivot of the geometry to the floor but that didnt change a thing.

    Sorry, but I would need more information - please make a separate post with your example for analyzing it.

  • Hi *thumbup*

    I have just finished creating my first depth map in Photoshop *w00t* . I was working on it for about 20 minutes and the effect, in my opinion, is overwhelming. It turns out that making such a map is not so hard at all (I'm sure it will be easier with the next ones).

    At Oculus Quest and GearVR the impressions are delightful *squint* . New possibilities of Krpano are amazing.
    Just Photoshop and time.

    Link (best in For Oculus or GearVR)

    https://cms4vr.com/examples/1.20/cliff/


    My depth map looks like this.

    If you want, I can record a tutorial film, like I do in Photoshop. *thumbup*


    Regards
    Piotr

    p.s.Hi Klaus, I don't know if this is the right place for examples for depthmap with new Krpano. If not, please move this thread to the right place.

  • Very excited for this release. Thanks for all the hard work *g* . I have been looking into depth map generating python scripts with not much luck in the results.

    If you want, I can record a tutorial film, like I do in Photoshop.

    Piotr, that looks great *thumbsup* . I would greatly appreciate your Photoshop tutorial, thanks *g*

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!