Import Gaussian splat using three.js

kme · January 13, 2024 at 12:25 PM

Hi All,

Last year, I experimented with importing three.js objects into krpano (Import GLB in krpano using threejs helper). More recently, I played around with gaussian splatting and noticed that there was a three.js based implementation (https://github.com/mkkellogg/GaussianSplats3D) to display gaussian splats in a web browser. so I had the idea to update the threejs helper plugin so it could also display gaussian splats.

What I had to do to accomplish this:

* upgrade to the latest version of three.js: previously, I used version 0.147 as this was the latest version that had a non-modular approach to integrate three.js into other applications. With some direction of klaus (RE: Using ES Modules in krpano plugin) I managed to import three.js into the plugin and use the latest version.

* upgrade to webgl2: I kept running into WebGL errors. After some investigation, it seemed that by default, krpano uses the stanadard WebGL implementation, and the libraries I was using required WebGL2. But ofc, krpano provides a way to upgrade to WebGL2 by using https://krpano.com/docu/embedpano/#webglsettings in the embed options!

After these 2 hurdles were taken, extending the existing helper to accomodate gaussian splats was pretty straight forward. A quick demo scene can be found here:

sketchfab tutorial

I'm really looking forward to a better integration for krpano and other WebGL renderers (like three.js) so we can extend the power of krpano even further.

Have a nice weekend!
Kristof

Tuur · January 13, 2024 at 1:22 PM

Nice and interesting!

Tuur

klaus.krpano · January 15, 2024 at 8:30 AM Official Post

Great work!

And yes - in krpano 1.22 and there are new explicit rendering and hit-testing APIs (and with them things like stereo and VR support are working almost automatically) - and also WebGL2 is enabled by default .

Best regards,
Klaus

indexofrefraction · January 15, 2024 at 11:11 AM

amazing work, kme !

on my intel macbook pro from 2020, the example runs a bit slow.

can the splats density be reduced ?

lastromantik · January 20, 2024 at 6:23 AM

Quote from klaus.krpano

and also WebGL2 is enabled by default

What will the transition to webgl2?

klaus.krpano · January 20, 2024 at 9:53 AM Official Post

Quote from lastromantik

What will the transition to webgl2?

There is no transition, all keeps working the same.

When WebGL2 is available, krpano will use it by default, otherwise WebGL1 will be automatically used.

The main reason why WebGL2 will be enabled by default is ThreeJS.
ThreeJS really needs WebGL2, without WebGL2 is lacks mipmapping (because it enforces sRGB textures) and that can look ugly in some cases.

For krpano the main advantage of WebGL2 will be the possibility to use mipmapping also on non-power-of-two images (but that's avoids the need to upscale unfitting images, but that's only a small internal detail).

Best regards,
Klaus

lastromantik · January 20, 2024 at 10:45 AM

Okay, then why krpano Three.js, If krpano does not use his?)

There is a demo version of the tour where he uses it, but here he did not use the model as a basis. And if you use the model as the basis of the tour, then imported using Three.js Notdo not intersect in a common depth buffer. Or did something change and now it's resolved and they exist in the same depth buffer?

klaus.krpano · January 20, 2024 at 1:33 PM Official Post

There will be a threejs plugin in the next release.
It allows fully integrated rendering (shared rendering and depthbuffer) and interaction (hit-testing).

lastromantik · January 20, 2024 at 3:37 PM

Oh wow!!! This is already cool!) That is, new opportunities open up in importing models with animation and managing it?))))

lastromantik · January 20, 2024 at 3:38 PM

Damn, what a very anticipated release of krpano for me !!!!

Import Gaussian splat using three.js

Participate now!

Similar Threads

Spatial videos an 3d gaussian splatting

Users Viewing This Thread