Gaussian Splatting: The Next Big Breakthrough in 3D Graphics
Gaussian Splatting: The Next Big Breakthrough in 3D Graphics
Forget NeRFs - Why Gaussian Splatting Is a Game Changer. Plus, Amazing Use Cases It Unlocks.
🪄 Creative Technology Digest
Your weekly dose on the future of creation & computing. Topics spanning creativity, technology and culture.
3D Gaussian Splatting - Explained!
🚀 Email Highlights
🌟 Technological Leap in Reality Capture: Discussing the evolution in reality capture tech, bringing photorealistic 3D model creation to consumer hardware.
📹 Deep Dive into Gaussian Splatting: An 8-minute video explaining Gaussian Splatting in detail and covering 5 reasons why they’re superior to NeRFs.
🛠️ Practical Applications & Tools: explore applications like memory capture, reality-bending effects, and 3D reskinning.
📚Learn & Engage: Interview with Singularity Hub on advances in 3D capture including applications for historical conservation.
Read Time: ~4 minutes
🍇 Juicy Topic For The Week: Advances in Reality Capture
Amidst the hype around AI image and video generation is another key vertical getting a huge boost: The Art & Science of Reality Capture
While reality capture isn’t new per se, the level of democratization has been staggering. The costs of sensors, AI-driven techniques like NeRFs and gaussian splatting, and computing power have dropped dramatically, making photorealistic 3D model creation accessible on consumer hardware.
3D capture is moving so fast - I scanned & animated this completely on an iPhone.
Last summer you'd need to wrangle COLMAP, Instant NGP, and FFmpeg to make NeRFs.
Now you can do it all inside Luma AI's mobile app. Capture anything and reframe infinitely in post!
Thread 🧵
And baby is it moving fast! Remember NeRFs? Only a couple years old, and along comes Gaussian Splatting — which is faster, editable, and readily integratable. We're talking 100 fps rendering performance on a modern GPU 🤯
3D scanning and rendering is moving so fast - got my splats up and running and I'm mind blown getting ~100fps for this complex 3D scene ⬇️ 🤯
1. WAY faster than NeRF: For comparison, NeRFs would takes around 10 seconds per frame (!) Instead I'm zipping around with FPV controls… twitter.com/i/web/status/1…
It's a big step towards democratizing photorealistic 3D content creation.
🤯 But Wait… What the heck is Gaussian Splatting?!
While I did a TL;DR post on this, ya’ll asked for a more thorough explanation. So by popular demand, here’s an 8 minute deep dive! This video covers all the core concepts you need to understand Gaussian Splatting, and why it is one of the most promising advancement in computer vision and graphics.
In this 8 minute video, we’ll get into:
00:00 Gaussian Splatting Overview
00:17 Why Is Gaussian Splatting Awesome?
02:35 What's Happening Under The Hood?
04:00 Step-By-Step: Building a Gaussian Splatted Scene
06:27 Conclusion: The Future of 3D Graphics
🎥 Creation Corner:
It's time for the visual umami 😎 Reality capture is about to turn up to 11. ✨Truly remarkable to witness the pace of progress in this timeless quest to connect the physical and digital worlds.
1/ Memory capture: perhaps the killer consumer use case for 3D/4D capture? Unsurprising that Apple is leaning into this “spatial media” direction with 3D photos/videos for their Vision Pro headset.
i'm convinced the killer use case for 3d reconstruction tech is memory capture
my parents retired earlier this year and i have immortalized their home forever more
photo scanning is legit the most future proof medium we have access to today
scan all the spaces/places/things
2/ Reality bending effects: Given how close gaussian splatting is to a 3D point cloud representation (another benefit over NeRFs!), creators like Ruben are adapting their shaders to work their reality bending magic on 3D captures.
Holy crap! @Ruben_Fro needs to stop blowing my mind with these epic shader effects for Radiance Fields created with Gaussian Splatting (no I’m never calling it a “gsplat” lol)
3/ Clean Animated Reveals: Everyone is absolutely obsessed with Luma AI’s loading animation. Just take a look at this capture of downtown Dallas @ night. PolyCam needs to step its game up in this regard!
Welcome to the jungle 🌴 🏞️
Love how this 3D scan turned out — immaculate trees, vegetation, and that little waterfall is just 😍
⚙️ Processed with the new Luma AI update that allows you to create radiance fields with Gaussian Splatting
4/ Reskinning 3D captures: After getting the new DJI Mavic Mini 4, I just had to put it to the test to make a 3D “dronie” — combined with a healthy dose of generative AI.
drones are fun 🔁
so are glitchy transitions reskinning realty 🪄
⚙️workflow: dji mini pro 4 --> luma ai gaussian splat --> kaiber ai video transform --> premiere pro edit
5/ Dynamic 3D scenes: Early research is showing promising results bringing VR-ready performance. A big step up from the rather GTA 3 looking “videogrammetry” pipelines of yesteryear.
4D reality capture is making major strides. You probably saw everyone raving about gaussian splatting and debating whether it's a (Ne)RF or not... lol
Well how about making them dynamic? This new AI paper creates dynamic 3D scenes with characters and objects that you can attach… twitter.com/i/web/status/1…
🌶️ Hot take: Big step forward for static capture with 3d scanning upstarts and incumbents alike benefitting, but volumetric video creation still remains elusive. Perhaps Apple will nail it for the Vision Pro?
The tradeoff with the current SOTA research is you still need a darn synchronized multi-camera array. But if you have that you get some killer results - such as this 4K volumetric video ripping at 80 fps.
4k volumetric video @ 80fps on a 4090 🔥 Of course capture rn necessitates a synced multi-camera array; but the fact that we’re moving past uncanny GTA looking “videogrammetry” to photorealistic 4D radiance fields means distribution won’t be a problem :)
Behold! AR gaussian splatting on iOS with Metal and ARKit 📱
Check out this sweet demo by @laanlabs featuring volumetric video by @8Infinite8.
Wonder how such approaches compare to what Apple is doing with their spatial media format for Vision Pro 🤔
While I love the performance and quality of the approaches above - I would love to see simpler, more forgiving forms of 4D capture. It will be interesting to see how far Apple can take things with stereo magnification off the iPhone 15 Pros.
So this prediction turned out to be true — spatial video is gonna be a ton of fun. Stereo magnification is very possible with such a small baseline.
Layer in 4K 60 FPS ProRes Log recording to a USB-C SSD? 🤯 I’m definitely buying an iPhone 15 Pro!
So, how can you make your own splats? 🤳🏽📷
If you want the easiest onramp, I’d recommend playing with Luma AI and Polycam. Both companies have already implemented gaussian splatting generation and viewing, and are swiftly rolling out improvements (e.g. floater removal).
Folks that know their way around command line might enjoy digging into the the OG Inria repo, though note that unlike Luma and Polycam it’s for non-commercial use only unless you get a license from Inria.
🤠 Dallas, Texas visualized in glorious 3D with @LumaLabsAI’s brand new, commercial-ready implementation of Gaussian Splatting. Also fitting, as I’m in Dallas for an AI panel at #VidSummit2023.
First @Polycam3D now Luma. This has to be fastest implementation of research to… twitter.com/i/web/status/1…
Floaters be gone! 🪄
The new @Polycam3D update not only improves quality (fewer floaters and improved rendition of backgrounds), you can now upload videos directly AND use these gaussian splats commercially.
Want to edit your splats? Check out Spline and PlayCanvas:
We are adding Gaussian Splatting to Spline!
⚡️Import splats in .ply format
✂️Crop and adjust the splats
🍭Mix splats with 3D objects
🍻Export + embed into websitesDemo: my.spline.design/girlstudio-8b6…
Learn how it works: docs.spline.design/e17b7c105ef043…
Incredible progress building a high quality #GaussianSplatting pipeline:
🤳 Capture with @LumaLabsAI/@Polycam3D
✂️ Edit in the @playcanvas SuperSplat tool
👀 Render in the PlayCanvas Splat Viewer
Background replaced with 16K HDR equirect. Specular on leather is 🔥!
Links in 🧵 twitter.com/i/web/status/1…
🌱 Long-form Deep Dive:
📈 Interview With Singularity Hub - I enjoyed sharing my thoughts on 3D reality capture tech with Singularity University’s publication.
A Revolution in Computer Graphics Is Bringing 3D Reality Capture to the Masses
🎙️ Podcast with Bad Decision Studios - Timely conversation regarding the Apple Vision Pro, NeRFs and a bunch of other fun stuff. We’re all high energy, so watching this legit feels like drinking 3 shots of espresso 😛
On our latest episode, we talked to @bilawalsidhu about how Artificial Intelligence is spreading into different sectors, and how you can actually make the most of this change.
We also covered Apple Vision Pro, NeRFs & Photogrammetry and etc.
Watch it now here… twitter.com/i/web/status/1…
💌 Stay in Touch:
Got feedback, questions, or just want to chat? Reply to this email or catch me on social media.
Cheers,
Bilawal Sidhu