About the depthmap support in krpano...
(available since krpano 1.20)
Topics:
- What are Depthmaps?
- Why Depthmaps?
- How to build Depthmaps?
- Depthmap 3D-Modeling: everpano 3D
- Technical Details / More Information
- Depthmap Examples / Demo Tours for testing
What are Depthmaps?
Depthmaps are typically images where the pixel color (typically grayscale) defines the distance (or 'depth') to a certain surface.Here as example a spherical pano image plus a related spherical depthmap image - the darker the gray-tone in the depthmap image the farther away the surface:
Why Depthmaps?
Knowing the depth of the pixels makes it possible to generate new viewpoints from an image. That means viewing the image from a different camera position than it was originally captured.And that allows doing many nice things:
- Generating synthetic Stereo-views with depth perception.
- Very helpful and very important for good VR viewing.
- Can be used also for Anaglyph (red/cyan glasses) stereo-viewing.
- No need for shooting and stitching stereo panos.
- Depth perception even when looking up or down (that's not possible using normal-stereo pano images).
- 6DOF Movement
- Six Degree of Freedoms: additionally to the normal rotation in a pano, now also a translation is possible.
- Positional head-tracking for VR.
- Perspective changes, e.g. viewing the pano from 'outside' ('Dollhouse'-effect).
- Walking inside the pano a bit (depending on the image content).
- 3D-Transitions
- Doing 3D-translations during the pano-blending.
- Measurements
- By having the depth / distance information and a reference size, it will be possible to do real measurements in the pano image.
Here a quick video preview for what depthmaps can be used:
Try this example yourself here: Little Temple of Abu Simbel (also included in the download package)How to build Depthmaps?
There are several different possibilities how to generate depthmaps:- With specialized software from different perspective images, e.g. from stereo images.
- But can be difficult, content-dependent and today still very error-prone.
- Might get better in future with better software.
- There are already some (expensive) cameras available that can generate panoramic-depthmap images automatically.
- By Laser-Scanning / LIDAR
- Rare and probably expensive (sorry, no real own experiences here).
- Drawing by hand
- For 2D images this is possible, but for panos almost impossible (or at least very difficult).
- With 3D-Modeling
- Can be done easy and quickly with the right software.
- Can be done for existing panos!
- Because of 3D geometry flat surfaces can be rendered very accurately (depthmap-images have limitations here).
- The 3D model itself do not even need to be very accurate for a good 3D effect.
- The storage needed for 3D models is typically very low, so they are quick to load.
- So that will be the focused method here now.
Depthmap 3D-Modeling: everpano 3D
There is a new special krpano-based software available for modeling depthmaps:
The everpano 3D software is developed by Jordi Vallverdú (a long-term and experienced krpano user) and allows a quick and easy modeling of 3D-geometry for panoramic images. Additionally the software can link several panos and build automatically whole 3D-tours with nice 3D-transitions effects.
Here an introduction video of the everpano 3D software for how to draw the geometry:
The software is available for purchase here on the krpano site and also on the everpano site. The depthmap feature requires a krpano license upgrade. That upgrade is available separately or also bundled with a everpano 3D license.
Some background information:
- Originally the depthmap support was in development only as 'very special' feature and with several limitations - e.g. only spherical-pano images, only single-res and only depthmap-images were possible. This because that was relatively simple to implement and depthmap-images were generally not easily available or easy to made, so the usage cases would have been very limited anyway.
- But then by chance Jordi Vallverdú contacted me that about the krpano depthmap support in the next release and that he is developing a tool for drawing depthmaps.
- The depthmaps generated by his tool were showing great results and a lot of potential. With it everyone would be able to build depthmaps and 3D-panos. And depthmaps could be added to all existing panos - no need for special pano-shooting or special hardware!
- So I've started to improve the depthmap support in many ways to make it better and more generally usable. It can be used now also with cubical-images, with multires-images, use depthmap-images or even directly a 3D-geometry, link panos with 3D-translation transitions, have different rendering modes, support VR-viewing with positional-tracking, has special hotspot-support and many things more.
- That was originally not planned and caused a lot of internal changes and therefore was one of the reasons why the development of this release took so long. But I hope it was worth the waiting time.
Technical Details / More Information
The krpano viewer supports two sources for the depth information:- Depthmap-Images
- Typically grayscale images where the gray-tone defines the depth:
- As there is no standard that defines what gray-tone related to which depth, there are additional settings available to customize the depthmap-interpretation for getting the correct offsets and scales.
- The depthmap images don't need to have the same resolution as the pano image, resolutions like 1024x512 should be already enough. More would be waste of resources.
- The browsers are supporting only a 8-bit color/grayscale resolution, even when the input image might be a 16-bit PNG image. So the depth accuracy is limited to 256 (=2^8) different depth-levels. For some cases, this limitation can become problematic and cause visible 'ripples' when viewing the pano from other viewpoints.
- Typically grayscale images where the gray-tone defines the depth:
- 3D-Models
- Additionally krpano can also use 3D-models / 3D-geometries for getting the depth for a panoramic image:
- 3D-models don't have the limitations of depthmap-images.
- They are very accurate and typically also faster to load (because the files are smaller).
- krpano supports 3D-models in the STL file-format (ASCII format only),
- and in a simple, easy to parse and fast to load special DEPTH file-format.
- The 3D-geometry will be either used to generate an internal depthmap image (but with a high-quality 24bit depth resolution) or used directly and the panoramic image projected onto it. Depending on the usage-case and the image content the one or the other mode will be better.
- Additionally krpano can also use 3D-models / 3D-geometries for getting the depth for a panoramic image:
- view.tx / view.ty / view.tz
- Move the viewer /camera position.
- Coordinate system:
- X: left(-) / right(+)
- Y: up(-) / down(+)
- Z: in(+) / out(-)
- view.ox / view.oy / view.oz
- Add an additional offset, but after doing the viewer looking rotation.
- Could be used for 'outside' / 'dollhouse' views.
- image.ox / image.oy / image.oz
- Move the position of the pano-image itself.
- Can be used for tours with depthmap panos.
- Use the ox/oy/oz to define the absolute/world positon of a pano, and for moving to an other pano, tween the view tx/ty/tz to them.
- The pano will be viewed from its center when the view.tx/ty/tz and the image.ox/oy/oz are the same.
- hotspot[name].tx / hotspot[name].ty / hotspot[name].tz
- Move the hotspot in 3D space.
- When using these setting it can make be helpful to set the hotspot[name].depth setting to 0 to avoid a combined translation.
- hotspot[name].torigin
- Defines the origin of the hotspot position: should it be absolutely positioned or be relative to the current pano or relative to the current view.
- hotspot[name].depthbuffer
- Define if the depth of the pano and the depth of the hotspot should be 'merged' during rendering.
Depthmap Examples / Demo Tours for testing
Some examples showing the possibilities when using depthmaps:
- A full-featured demo tour using depthmaps.
- It demonstrates how to view a pano dynamically from new perspectives, e.g. to show wall friezes without a perspective distortion.
- Additionally the depthmaps allow nice 3D-transitions.
- The tour offers automatic- and manually-generated hotspots for navigation.
- The depthmaps were generated by the everpano 3D tool, but the tour itself manually designed and coded.
- A little helper tool for finding 3D viewpoint and hotspot positions is also included.
- A whole tour fully generated by the everpano 3D tool.
- For navigation in the tour the everpano Navigator plugin is used.
- That plugin allows easy navigating by automatically finding the pano next to the point where the user had clicked - and when there is no pano near, a nice 3D-zooming-animation indicates that.
- Here for comparison and more testing the same tour with hotspot navigation: Gravina Apartment - Example 2
- Here a set of example panos with automatically depthmaps generated.
- The depthmap images here were either by created the camera itself or by rendering software or by stereo-to-depth software.
- A small helper tool for finding the correct offset and scale settings for depthmap images.
- The values for these settings depends how the depthmap-image was created and mapped.
More Depthmap Demos at everpano.com:
. I was working on it for about 20 minutes and the effect, in my opinion, is overwhelming. It turns out that making such a map is not so hard at all (I'm sure it will be easier with the next ones).
. New possibilities of Krpano are amazing. 
. I have been looking into depth map generating python scripts with not much luck in the results.