I knew you wanted the code. I didn’t think this would happen.
Since I open sourced God’s Eye View, the repo hit #1 on GitHub Trending. Chris Messina, the creator of the Twitter hashtag, put it on Product Hunt. Then Brendan Eich, the creator of JavaScript -- yes, the language I used to code the damn thing -- jumped into the conversation.
And so many of you have been installing it, making videos, and building on top of it. I’m honestly overwhelmed by the support. Thank you.
The repo is now at roughly 38,200 stars and 7,700 forks.
But the same two questions keep coming up: how do I run the damn thing if I’m not a coder, and can you give me a proper walkthrough?
So I made a 30 minute deep dive, which has astoundingly crossed 2.3 million views in a week. Check it out below:
Are you already set up? Jump straight to the walkthrough (2:08), voice control (7:47), camera tours (19:55), or the Nepal flood reconstruction (24:18).
Get a feel for it
We are building the window through which we see the world - an audit trail for physical reality ready for you to inspect.
For the open source version that means pulling together the most generous data providers to give you a remarkably complete picture of what’s happening on the planet at any given moment. With it you can:
Look at all those vessels. The coverage in Europe and the US is remarkably dense.
A landing from the perspective of DAL 3146. And while I did this the voice AI informed me that Delta actually started as a crop dusting company (TIL!)
Satellite fire detections via FIRMS. I used voice mode to navigate the Ukraine front line and oil refineries across the globe.
Setup Instructions
The last post covers getting started. Notably, here’s an official one-click installer (Pinokio 8.2+) and the GitHub repo. Thank you to the Pinokio team for fixing a bug we found. You can now start without API keys and add them for more features - further reducing friction :)
And for everyone asking me for a link they can just open — we’re building an official hosted God’s Eye View at Halfpixel. More soon; sign up for the waitlist and get notified.
Build on it
You’re not limited to the data layers I put in. In the video, I show how you can bring reports, photos, and footage into the same map to reconstruct the Nepal floods.
Layer in Vantor and Planet disaster response imagery and suddenly you know the before and after. Geo reference and visualize where those clips were filmed - and suddenly you get a very different sense of the scale and scope of what happened along the valley.
Whether you’re trying to report, help, understand what’s going on, or marvel at the interconnected nature of our planet - there’s a lot you can build on this foundation.
Oh and important. If you make something worth sharing, send a pull request so we can all play with it. Or open an issue for a layer or feature you want. I’m extremely grateful for the amazing issues and PRs so far - you’re really driving our roadmap, and we don’t take the opportunity lightly.
Hurricane replay
Now look, God’s Eye View is our open foundation. It takes esoteric fields that rarely talk to each other and brings it all together on a 3D globe. If you push the time dimension hard enough, suddenly you’re building a time machine and a crystal ball at the same time.
And the potential is remarkable. I used our new WorldFX platform to build this situation monitor whose sole job it is to track the trio of powerful hurricanes fueled by a generational Super El Nino.
You can watch Marie, Karina and Lowell slowly move westward across the Pacific Ocean, including Lowell rising to Category 5.
The replay uses real geostationary satellite imagery + ECMWF winds + official NHC tracks of where the storm was and where it might go.
And it absolutely rips on my machine. This is where people are going to be shocked at the performance jump we can achieve from v1 to v2.
Miami crash reconstruction
Recently, we all witnessed a tragedy that was the prime air crash. Open source intelligence flooded social media, and I used the tools we’ve been building to fuse it all together.
Essentially creating a 4D reconstruction of the Amazon Prime Air cargo plane crash at Miami International Airport.
You can see the plane overrun the end of Runway 30, cross the airport perimeter, and come to rest roughly 1,300 feet beyond the paved surface. Built it all using public flight data, eyewitness footage, and ATC excerpts.
Precise cause is still TBD - because importantly, we don’t pretend to know the truth about everything. Our job is to fuse public and commercial sources to get you as close to the ground truth as possible - and let you form your own understanding based on the situation at hand.
I also created a 3D scan of the crash site using newly released NTSB footage. You can explore the 3d gaussian splat, site photos, and replay the landing with flight tracking data and ATC audio. Made with COLMAP, LichtFeld Studio, God’s Eye View and Astra.
Roberto Nickson put this first in his roundup of things people built with Astra. Astra helped put the interactive site together :)
Astra slid into the party
Sharif Shameem shared Astra’s recreation of the Palace of Fine Arts in Blender. This blows my mind.
Turning real world 3d scans with messy topology into procedural 3d scenes (modeled like a human artist would have) has historically been a tall order. But Astra seems to have cracked it - or at least made serious strides.
MSFT flight simulator does a basic version of this using 2d satellite imagery and a building grammar system - but it only looks decent from the air, and quickly falls apart when you get to the ground.
If this technique scales, you could get GTA level 3d detail at street level for the whole world. Just give coarse 3D scans and ground level photos as reference. The implications are nuts for everything from gaming to defense.
And to think this was a whole IARPA program just a few years ago (see CORE3D and WRIVA - both fun rabbit holes in how the same tech for virtual production and gaming is incredibly relevant to defense and intelligence)
Then I tried it on my parents’ old home
Of course I wanted to push it further. Asked Astra to take an old 3d scan of my parents drawing room and model it from scratch in Blender. GPT-6 Astra did it. Critically, it didn’t download any assets and sourced textures from my photogrammetry scan and made procedural shaders. Man did it do a nice job!
Then I turned down gravity and opened a portal. Because why not. If you can perfectly model reality, suddenly you can infuse it with imagination. And that really has been the theme in my life - connecting the world of bits and atoms, and blending reality and imagination.
The big picture is that it turns out multimodal LLMs have crossed the chasm on inverse graphics. Tremendous implications for everything from VFX and AR/VR to robotics.
Process: ~400 2D photos I took in 2019 went through RealityScan and became a photogrammetry model. That’s the classical 3d reconstruction step - outputting a textured 3d mesh.
Then I dropped that mesh into Blender and used Astra to turn it into a fully editable, cleaned up 3D scene. Asked it to use textures from my photo scan and create procedural 3D shaders for the rest.
It’s good but it’s definitely not perfect -- funny thing is some of the mistakes are inherited from the reference mesh RealityScan produced i.e. the bird statue’s legs, contours of the furniture etc.
This is where humans still have much better “spatial autocomplete” and can take an output like this all the way.
My next step will be to see if I can let it use CUA (computer use agents) with the photoscan software to interactively reference the source pixel imagery and refine its work in Blender and get even closer to reality.
What this means for world building?
If you take a step back, the implications to world building are tremendous.
I didn’t touch a button, keyframe or a node graph. It’s like turning my mind inside out, like turning my imagination into reality.
Simultaneously, I can aggregate signals from a more transmissive earth to form an ever more complete picture of our story. I can sit on my couch and collaboratively build the window through which we see the world with a remarkable community. What a time to be alive.
Thank you to everyone who’s been building, sharing, and breaking things. Cannot wait to see what you cook up. See y’all in the next one :)
-Bilawal
GTA level detail at street level for the whole world 🤯
What a time to be alive! I remember being bummed out 5 years ago in my early 20s, thinking I missed out on the opportunity to take advantage of the early .com window people had in the early 2000s or to be one of the first people growing their business with YouTube or Google when that first came out. But the opportunity AI presents is multiples greater and keeps getting more wild by the day. Just have to adapt early, stay optimistic, and work your ass off.