IronSight: Turning 2D Videos Into 4D Reconstructions of Reality
I turned range footage from Ray-Bans and GoPros into a replay I can fly through -- and built the exam that keeps it honest.
I turned a shooting-range session into IronSight -- a desktop web app that automatically lines up footage from Meta Ray-Bans, phones, and action cameras; finds each shot; maps targets into a shared scene; and lets me fly a camera through the 4D replay. The result is deliberately a little cyberpunk. More importantly, it is thoroughly inspectable.
What’s in the video:
An overnight researcher
Saturday morning I woke up to an experiment board full of reconstruction runs. The night before, I’d recorded a 20-minute voice note with every test rattling around in my head: try the new colmap global mapper this way, try pi3 and that other fast feed-forward mapper, plot every camera path, and tell me exactly where tracking falls apart.
By morning, the agents had run the jobs and left me trajectory plots, pass-or-fail notes, and one wonderfully boring result. Two methods traced nearly the same curve. The fast one wandered all over the range. I still had to scrub the footage and decide which solve actually stayed pinned, but the pile of waiting hypotheses was now a board of evidence. Pretty freaking wild.
The shot counter that got out of hand
This started in June 2025 with the simplest thing you could possibly build: a shot counter. I wanted Gemini’s audio and video reasoning to tell me when I fired and whether I hit anything.
The first version was janky. It could hear a shot and sometimes call hit or miss, but it had basically no idea which target was involved. I ran the result through an After Effects script, put a nice HUD on top, and made it look more certain than it was.
The HUD was cool. But it was manual. And all anchored to the screen of a flat video. If the camera moved or another camera watched the same run, those views didn’t agree on a world.
So I wrote the grand future roadmap -- reconstruct the range in 3D, recover the camera trajectories, synchronize multiple views, and put everything into one spatial frame. Then I filed it under someday.
And then Fable 5 dropped.
I blitzed through most of that roadmap in a couple of days and ended up with a browser replay where I could grab the camera, orbit around a run, inspect the targets, and step through every shot.
One clock, one range
My rule for IronSight was pixels only. No LiDAR, no depth stream, no inertial measurement unit. Those sensors would help, obviously, but people show up to a shooting range with a phone, glasses, or a GoPro. They don’t carry a survey rig.
So the first job is time.
A gunshot makes a sharp spike in the audio waveform, so I stack the tracks and shift them until the spikes line up. Sound gives me the shared clock. Then a masked COLMAP solve looks for matching features across the clips, works out where the cameras were, and puts both paths into the same coordinate frame.
Now every clip can land on the same range. I can pause at one shot, leave the source cameras behind, and fly to a viewpoint nobody recorded.
The boring proof
Once the cameras are solved, I train a 3D Gaussian splat. Think of it as millions of tiny translucent blobs positioned in space. Render enough of them together and the flat images become a range I can view from a camera that never existed.
The proof I trust is a match photo. I line up the reconstruction with a real source frame and drag a divider across the screen. Fence post, berm, target, camera pose. If the geometry is wrong, something slides across the wipe. When it holds, the target stays put.
Gemini finds a target inside a 2D crop, and IronSight projects that detection back into the reconstructed range. Once the target has a 3D position, its marker can stay put while the camera moves behind fencing or a wall. Nothing was sensed through concrete; the target had already been seen, mapped, and remembered.
I built the exam into the tool
The classifier has two jobs. Classical audio detection finds the shot, then IronSight crops the likely target, grabs frames from before and after, and asks Gemini to call hit or miss. Ambiguous events go into my review queue.
I had my clanker build an annotation tool for that review. Arrow keys move shot to shot. I can fix a label, draw a target box, or flag the event, and because the cameras are synchronized, a label from one view transfers to the partner view. No annotating the same shot twice.
You build the exam before you grade the model. The working evidence set had 15 human-labeled clips, 213 shot events, and 135 frame-anchored target boxes, with fixed holdout clips kept out of the labeling pass.
Footage I didn’t build around
Shout out to Richard Ryan for sending me clips I hadn’t tuned the whole thing around. His 11-shot run went through the same process and came back as a God’s-eye replay with targets, tracers, a shot log, and inset source views.
I could pick drone follow, orbit the run, frame every target, or set the keyframes myself. I used to develop this kind of treatment through After Effects. Now I grab the exact camera view I want in the browser and hand the render recipe back to the agents. I write code to do my GFX :-)
Video becomes a place, and the camera becomes just one way into it.
Then the spinning star tore away
Static targets are friendly to this reconstruction. The berm doesn’t move. The fence doesn’t move. A steel plate may ring, but its mount stays where the cameras expect it.
Then I pointed IronSight at a spinning star.
The geometry rotates after a hit, and a static Gaussian splat can’t represent that motion cleanly. The star tears away from the reconstructed world, and the classifier starts claiming I hit the one spinning in the middle.
Honestly, good. I want the failure sitting right next to the clean demos.
IronSight is an offline pipeline today. The replay comes after synchronization, camera solving, splat training, human review, and rendering. I think hit detection will move onto edge hardware and feed-forward reconstruction will make the 3D side much faster. Anduril’s EagleEye is a direction of travel here, not an equivalent to what I built.
If you’ve got a pile of multi-cam footage and no idea what it’s actually worth, hit reply and tell me what you’d point this at.
The same workflow that makes a range session inspectable could rebuild any event once enough cameras overlap. Every camera could see a different clip, and somebody could still recover one world from viewpoints nobody captured.
God’s Eye View Update
You asked, so it’s happening. God’s Eye View v1 is coming — free and open source, targeting end of July. Star & watch on GitHub so you’re notified when it’s public: https://github.com/bilawalsidhu/gods-eye-view
As always, if this gave you something to think about, forward it to other Frontier Navigators. And I’ll see y’all in the next one!
-- Bilawal