I tracked a person through a concrete wall -- every step, every turn. Centimeter-level accuracy, using just an iPhone and a $300 pair of Meta Ray-Bans tied into off-the-shelf VPS software.

Try it yourself → https://github.com/bilawalsidhu/see-through-walls

What a Defense Contract Buys

A few months ago, Anduril dropped EagleEye -- an XR headset built for soldiers. It overlays a live mini-map in your field of view, auto-tags friendlies and threats using AI, and yes, lets you see people through walls by fusing sensor data across the squad. It’s impressive. It’s also tied to a classified sensor suite and a DoD procurement process.

I wanted to know how close I could get with off-the-shelf hardware. Turns out -- uncomfortably close.

How $300 Glasses Pull This Off

Here’s the problem: Meta’s Ray-Bans are essentially dumb. One camera on your face. No depth sensor, no LiDAR, no SLAM. Every other “smart” headset has a dozen sensors figuring out where you are in space. The Ray-Bans have nothing.

The answer is something I spent years working on at Google: VPS -- Visual Positioning System. I made a video on this previously: VPS Explained

You pre-scan an environment. Build a 3D model. Then, when someone walks through that space, the camera feed gets matched against the model in real time. Every frame: based on what you’re seeing right now, where must you be standing? Down to the centimeter. Six degrees of freedom on glasses that can’t even find themselves.

Google built a version using Street View imagery across 100+ countries -- the ARCore Geospatial API I helped ship. But that only works where Street View has been. Your office, your home, a factory floor -- that’s where MultiSet comes in.

See it in action in the full video:

They’re the only platform publicly supporting the Meta Ray-Ban SDK. Already earned “Most Robust” in AREA’s 2025 Enterprise VPS Report, ahead of Niantic. The company is barely a year old. Scan the space, upload it, and the glasses just work. Centimeter accuracy. Day or night. Live.

The GPS Problem

I know what you’re thinking. Why not just use GPS?

GPS gives you a couple meters of accuracy and maybe 20 degrees of rotation on a good day -- outdoors. Take it inside, and it falls apart. Your blue dot jumping across a mall knows this. Your Uber driver circling a parking garage knows this.

VPS gives you centimeter accuracy, indoors, without beacons, without infrastructure. Scan once, and then any camera in that space knows exactly where it is. Which floor, which room, which direction it’s facing.

Where This Gets Serious

The see-through-walls trick is cool, but it’s just the first application of a more fundamental capability: shared spatial coordinates.

Once every device in a space is anchored to the same 3D map, things start clicking. A technician on an oil rig walks up to equipment and the last repair notes, maintenance checklist, and a remote expert’s annotations pop up in world space -- anchored to that exact spot. Someone in Germany can see the same view and draw on it. Months later, when the next crew rotates in, all that data is still there, floating where it was left.

Drones need this to fly in GPS-denied environments -- war zones where GPS is actively jammed. Self-driving cars run a version of this under the hood. And once you’ve scanned a space, everything operating in it -- glasses, phones, cameras, robots -- shares the same spatial understanding.

Now scale that. Every sensor in a city, anchored to a persistent 3D map. Privately in your home. Publicly on the highway. Access-controlled on a factory floor. The spatial map becomes the operating system for the physical world.

The Dual-Use Reality

This is where every dual-use technology ends up.

The same capability that gives a maintenance crew situational awareness is the same capability a building owner uses to track every footstep. Grocery stores are probably already doing it -- spatial heat maps, dwell-time analytics, attention tracking. Everything websites do with mouse tracking, but in physical 3D space.

MultiSet offers on-premise deployment, so you own all the localization data. Nobody else sees the tracks. That’s the right default. But the technology is capability-neutral. What gets built on top of it is the part that matters.

The Real Takeaway

I went into this wanting to see if consumer hardware could pull off military-grade spatial tracking. It can. A camera is all you need -- just a pre-scanned map and a phone.

I’ve been in this space since 2013, and we’ve still got a ways to go on the glasses themselves. But the next five years of AR and VR won’t be won on hardware. They’ll be won on the spatial map that pulls all these systems together.

If this gave you something to think about, share it with someone who should see it.

- Bilawal

Share

But one satellite in orbit, 400 miles up, fires a radar pulse straight through the cloud cover, through the darkness, and picks up every vehicle on that road. It measures their speed. Their heading. And when one of them pulls off and parks near a structure, it flags the change -- down to the centimeter.

No human looked at this data. An AI read it in seconds.

This technology exists right now. It’s not classified. And some of it is even free.

I’ve spent the last few months using open-source intelligence -- including satellite data -- to reconstruct global events like the Iran strikes and the Hormuz crisis. And on the defense and intelligence side of all that, synthetic aperture radar -- SAR -- is foundational to how a lot of it actually works.

Here’s what it is, why it’s about to be everywhere, and why that should make you both excited and a little uneasy.

How It Actually Works

Synthetic aperture radar. Satellites fire microwave radar pulses toward the ground and measure what comes back. Pretty standard stuff if you think about it.

But here’s the clever bit. A single radar antenna in space gives you a blurry image. These satellites aren’t static though -- they’re moving, and as they do, they’re collecting thousands of radar returns from slightly different positions. Stitch them all together and you’re basically faking a really long antenna out of pure motion. That’s where the name comes from. Synthetic aperture.

The result is a 25-centimeter resolution reproduction from space. And what makes SAR different from every electro-optical satellite up there is that it does not care about clouds. Doesn’t care about darkness, smoke, or rain. Optical satellites go completely blind in bad conditions. SAR just punches right through.

Then there’s a technique called InSAR -- Interferometric SAR. You take a radar image of the same spot today, and again several days later, and you start stacking them. What this lets you do is compare tiny phase differences and produce a deformation map -- a record of how much something moved at sub-millimeter precision. Stuff no human eye, not even most physical sensors sitting on the structure, could catch.

The really high-res stuff is either pay-to-play or available only on the defense side. But there’s a satellite constellation in orbit doing this for free, and it’s called Sentinel-1. It covers the planet on a 12-day cycle and anyone can pull it. Pretty fricking powerful for a free tool.

Watch the full breakdown here:

The Pyramid

If you’ve seen SAR mentioned anywhere before, it’s probably because of this.

Researchers used a technique called SAR Doppler Tomography -- or Tomo SAR -- from this Italian satellite called COSMO-SkyMed, and they used it to map the interior of the Great Pyramid of Giza from space.

I want to be precise about how this thing actually works, because it’s wilder than just “radar sees through stone.” The radar isn’t passing through limestone like an X-ray. COSMO-SkyMed uses X-band -- a 3-centimeter wavelength that bounces off surfaces. What the researchers actually did was analyze the micro-vibrations of the pyramid itself. Just the Earth humming -- seismic background noise that propagates through the structure differently depending on what’s inside. Voids, chambers, density changes. The satellite reads those tiny Doppler shifts in the reflected signal and reconstructs a coarse 3D map of what’s in there.

This is peer-reviewed and published in remote sensing literature.

To put it simply: they mapped the inside of a 4,500-year-old pyramid from space by listening to it vibrate. That is fricking cool.

Sinking Cities And Bridges That Aren’t Supposed To Move

It doesn’t stop at ancient monuments. The same InSAR technique, applied to the world’s biggest river delta cities like Jakarta, Bangkok, and Shanghai, shows us something pretty alarming. These cities are sinking faster than the sea levels are rising.

This tech can detect sinkhole formation weeks before the ground opens. And after earthquakes or floods, it tells first responders which structures have shifted before anyone sets foot near them. The crazy part is this data’s been sitting there for years. It’s just a matter of reading it.

Now let’s talk about things that really aren’t supposed to move too damn much. Bridges.

A team of researchers analyzed 744 long-span bridges globally using stacked Sentinel-1 data and published it in Nature Communications back in October 2025. The findings kind of blew my mind. North American bridges are in worse shape than African bridges. Worse than anywhere else on the planet. This is literally aging infrastructure showing up on a satellite map.

There are 46,000 structurally deficient bridges out there in America. Most of them get a physical inspection every two years -- someone drives out, looks at it, writes a report. These satellites can watch all of them every 12 days. Automatically. And when you add satellite monitoring to the mix, it actually cuts a bunch of the high-risk classifications by a third. Bridges we thought were about to fall turn out to be fine. The ones genuinely at risk, we catch them way earlier.

And again -- since this is Sentinel-1, the data is free.

The Village That Was Moved To The Wrong Place

This one I can’t stop thinking about.

A team at the University of Melbourne built a physics-guided machine learning system that takes Sentinel-1 data and runs it through an AI trained on how ground actually fails -- soil particles separating, microfractures propagating. It generates a risk map that highlights unstable slopes before anything visible happens on the surface.

And they pointed it at a village called Kimtang, in central Nepal.

Here’s where this gets really heavy. The people living in this village had been relocated there in 2019 after a landslide destroyed their previous village. The government moved them there thinking it was a safe location.

And this AI model flags a massive high-risk zone -- completely in red -- directly beneath that village. They were quite literally moved to the most dangerous spot in the region, and nobody had any idea until an AI read the radar data from space and caught it.

The British Geological Survey ran the same approach across all of Great Britain. 300,000 slopes analyzed. 3,000 actively moving. Affecting 14,000 km of road and 360 km of railway. Then after the 2025 Sumatra landslides, the same group mapped over 4,000 landslide sites in a single pass. What used to take months runs in hours now.

That’s the AI unlock. We’re not just collecting this data. We’re reading it, at a scale no human team ever could.

The Commercial Arms Race

This is where it stops being a science story and becomes a geopolitical one.

You’ve got ICEYE -- a Finnish company that operates the world’s largest commercial SAR constellation. They launched five new satellites in November 2025. Then six more on March 30th, just a few days ago, on a SpaceX Falcon 9 out of Vandenberg. 25-centimeter immaculate resolution. Their customers include the Polish Armed Forces, the Portuguese Air Force, BAE Systems, and even the insurance giant AXA.

ICEYE is not alone. You’ve got Capella Space, Umbra, and Synspective too. A whole commercial SAR industry that basically didn’t exist a decade ago. And this year, NASA selected ICEYE as one of seven commercial SAR providers for Earth science research.

The reason nations and defense contractors alike are buying into commercial SAR at this pace? Because of what comes next.

What The Military Already Knows

Everything we’ve talked about so far -- pyramids, sinking cities, bridges, landslides -- that’s the civilian side. Same technology, very different purpose on the military side.

One of these capabilities is called GMTI -- Ground Moving Target Indication. SAR filters out everything that isn’t moving. Ground, buildings, trees -- all gone. Anything left is in motion. Vehicles. Ships. Convoys. In the dark, through the clouds. And if you’ve got longer wavelengths, through jungle canopy. The military calls that FOPEN -- foliage penetration radar. You can detect vehicles and even people hidden under very dense tree cover from an aircraft. That’s real tech, and it exists right now.

The US used to do a version of this from an aging aircraft called the E-8 JSTARS. Circle overhead, track ground movements. That plane is retired now. But what replaces it isn’t another plane -- it’s a constellation of satellites that do the same thing from orbit. Continuously.

The Space Force and the NRO -- the National Reconnaissance Office -- are building exactly that capability. GMTI satellites launching in 2028. At full operational capacity, we’re talking about tracking every moving ground target globally in near real-time. They’re targeting the early 2030s.

And then AI closes the loop. Booz Allen built a system for the Air Force that takes all this SAR data and automatically classifies ground vehicles -- five classes, greater than 90% accuracy. Trained in five months using synthetic SAR images because there wasn’t enough real data. NATO is working on the same thing at alliance scale.

Chain it all together -- persistent SAR constellations, GMTI filtering, AI classification -- and you get a system that watches every road in every country, flags anything that moves, identifies what it is, and tells a human within minutes.

If you remember that convoy at the start of this post -- that’s not a hypothetical. That’s an acquisition schedule.

What’s Next On The Horizon

Last year, NASA and the Indian Space Research Organisation launched NISAR. For global monitoring -- not spying. For that purpose, it is the most advanced SAR satellite ever built, at an estimated cost of $1.5 billion.

On the commercial side, the push is toward what Capella calls “Earth intelligence infrastructure.” Not raw data -- processed insights delivered to operational workflows in near real-time. Monitoring as a service. You don’t parse the data yourself. You subscribe to be notified for the kinds of change you care about.

The honest final unlock with all of this stuff is just time. Sentinel-1 revisits the same spot every 12 days. As more constellations come online, those revisit rates drop to hours. Then minutes. And at some point you’re no longer looking at snapshots of the Earth. You are watching it live. All of it. All the time.

So What Do We Actually Do With This

SAR is some insanely cool and powerful tech. One technology -- radar from space -- can tell you which bridge is about to fall, which hillside is about to collapse, and what’s hidden inside a 4,500-year-old pyramid. AI can now read it faster than any human team ever could. And every month, more satellites go up.

On one hand, we can respond to landslides, monitor critical infrastructure, react to disasters faster than ever before. There’s a village in Nepal that should have been moved before the ground gave way -- this technology was the thing that finally noticed.

On the other hand, that exact same technology can track every vehicle on a grid. In pitch black. Through heavy cloud cover. With an AI automatically cataloging every single target. We are weeks, not years, from a world where if it’s outside, it’s seen.

SAR is just another layer of sensors adding to how we monitor the entire planet at once. Flights, shipping, GPS interference, satellite imagery, and now radar that sees through clouds, darkness, and stone -- all stacking on top of each other. The more layers we add, the harder it gets to hide anything on the surface of the Earth. The capability is already here. More of it is coming online. So you should know about it. But what we do with it -- that’s the part that’s still being written.

If you’re curious about how to use radio frequency as a sensing modality -- WiFi, 5G, the kind of stuff that lets you see through walls without satellites at all -- I made a video about that one too. Check it out.

If this gave you something to think about, share it with someone who should see it. Reality is getting more legible than most people realize.

-Bilawal

Share

Then on April 7th, Trump gave Iran a deadline to reopen the Strait of Hormuz by 8 PM ET -- or else “a whole civilization will die tonight” -- and by that evening, Pakistan had brokered a two-week truce. Oil futures crashed 15% in minutes. The headlines said the crisis was over.

One week later, a US naval blockade is in effect, the ceasefire exists in name only, Israel has launched a ground assault into southern Lebanon, and Iran is calling the whole thing piracy.

So I loaded up God’s Eye -- swapped in the latest AIS maritime data, overlaid the strike coordinates, and started scrubbing the timeline to see what the data actually says.

Watch the full video here:

Is It Open?

Before the conflict, roughly 130 ships transited the Strait of Hormuz per day. On the first day of the ceasefire -- April 8th -- 15 got through. The day after: about 6. The oil futures that crashed on the ceasefire announcement? They bounced back above $120 within 24 hours. Traders figured it out faster than the headlines did.

If you scrub through the AIS data in God’s Eye day by day, it’s stark. Where there used to be a constant stream of vessels, there’s just... nothing. A handful of dots where there used to be hundreds.

Trump called the strait “completely, immediately, and safely open.” Then a few days later said Iran was doing a “very poor job” of handling traffic and that “that is not the agreement we have!”

When I started clicking through vessels post-ceasefire, the picture was pretty clear. The ships that are getting through aren’t taking the normal shipping lanes at all. Iran published a map confirming mines in those lanes and is routing ships north, past Iranian-controlled islands near Larak -- putting them within range of light artillery from Iranian shores.

The toll booth from the last video? Still operational. $1 to $2 per barrel. A fully loaded VLCC pays $2 to $3 million per transit. IRGC escort, one ship at a time. Insurance premiums have hit 2.5 to 5% of hull value -- we’re talking $600,000 to $1.2 million per trip for a $120 million tanker. Several major insurers pulled Gulf war risk cover entirely. The US doubled its reinsurance backstop to $40 billion to keep the market from freezing.

Two bottlenecks. Insurance and Iran. The strait was technically open the way a highway is technically open when every lane is closed except one, and there’s a toll collector with a rifle.

Eternal Darkness

Within 24 hours of the ceasefire, Israel launched what it called Operation Eternal Darkness -- 50 IAF jets hitting over 100 Hezbollah targets across Lebanon in 10 minutes. Lebanon called it Black Wednesday. Over 350 killed. They struck during rush hour in central Beirut without warning. Netanyahu basically said Lebanon has nothing to do with this deal. The White House confirmed it.

There’s a rumor circulating in the OSINT community about how Israel managed to pull that off -- how you hit 100 targets simultaneously in 10 minutes. You’d need to know the location of every single one of them at the same time. The speculation is that Israeli intelligence got onto a Hezbollah Zoom call and used it to geolocate everyone who dialed in, timing the strikes to the confusion around the ceasefire. Unconfirmed. But Hezbollah is treating it as the biggest security compromise since the pager incident.

When you overlay the strike data on the globe, the density is insane. The targeting was so tightly packed it looks like a single cluster, not a hundred individual strikes.

This war relies heavily on AI-assisted targeting to execute. And that changes the calculus for every country thinking about how to communicate, how to organize, and how to operate in a world where a single compromised call can paint a hundred targets simultaneously.

The Retaliation

Hours later, Iran fired back. We’re talking ~94 drones and ~30 missiles across the UAE, Kuwait, Bahrain, Qatar, and Saudi Arabia. The UAE alone intercepted 17 ballistic missiles and 35 UAVs. Kuwait reported severe damage to its oil infrastructure. The Habshan gas complex in the UAE got hit for the second time.

Pull up the Gulf in God’s Eye, and you can watch the retaliatory strikes light up in real time -- UAE, Kuwait, Bahrain, Qatar, Saudi Arabia, all within hours of each other. The pattern tells the story: Iran systematically hitting every US-host country in the region.

And Iran went after the bypass infrastructure -- the Petroline and Fujairah pipelines that Saudi Arabia and the UAE built for exactly this scenario. Iran is saying: you think you can route around us? Think again.

And both sides couldn’t even agree on what the ceasefire said. Iran proposed a 10-point plan. The US countered with a 15-point version. Iran released multiple versions of theirs with differences between the Persian and English texts. The White House said the plan Iran published wasn’t the one the US approved. They were literally working off different documents.

The FARP

If you rewind to the week before the ceasefire, the events leading up to it are genuinely wild.

On April 3rd, Iran shot down an F-15E Strike Eagle deep in Iranian territory -- callsign Dude 44. The pilot was recovered quickly. The weapons systems officer, a Colonel, ejected onto a 7,000-foot ridge in the Zagros Mountains and hid in a crevice for 36 hours while IRGC forces searched for him.

On April 5th, DEVGRU and Delta Force airdropped in with 155 aircraft in support and roughly 100 special operations forces on the ground. They set up a forward base on an abandoned agricultural airstrip about 14 miles north of Shahreza, Isfahan province -- approximately 50 kilometers from Isfahan Nuclear Center -- one of Iran’s three major nuclear sites, with Natanz further to the north.

Think about that for a second. 155 aircraft and 100 SOF for a single airman. That’s a disproportionate response by any measure. Outlets, including the BBC, have speculated about whether the rescue was a cover for a parallel mission. 440 kilograms of 60% enriched uranium have been missing from Iran’s program since Operation Midnight Hammer last year. That uranium was last known to be at Isfahan. The FARP is right next door.

A C-295W from the 427th Special Operations Squadron -- a rare USAF transport designed for short unprepared strips -- was spotted on OSINT flying low over western Iranian desert around the same time. It can carry half a ton. The two MC-130Js at the FARP were destroyed. Official line: stuck in soft sand. OSINT speculation: they took fire coming in and couldn’t fly out.

I want to be clear -- this is all OSINT community speculation. No US official has confirmed anything beyond the rescue. But the data points are just... there. And the proximity to Isfahan is not something anyone in defense circles is willing to call a coincidence.

Meanwhile, the broader strike campaign was intensifying on both sides.

The US and Israel were systematically dismantling Iran’s industrial and military infrastructure. The B1 Bridge in Karaj -- the tallest bridge in the Middle East, reportedly used by the IRGC for missile transport -- was hit on April 2nd. Israel assassinated the head of IRGC Intelligence in Tehran. The coalition hit petrochemical complexes that produce missile propellants, airports where military aircraft were staged, and shipyards building IRGC fast-attack boats.

Iran was hitting back across the Gulf. Desalination plants in Kuwait. Oil complexes and petrochemical facilities in the UAE. Air bases hosting US forces. Iran’s message was clear: if you hit our infrastructure, we hit your allies’ infrastructure. And they were specifically targeting the pipeline bypass routes that Saudi Arabia and the UAE built for exactly this scenario.

Then April 7th: Kharg Island hit again -- 50 additional strikes on top of the 90 from March. Oil terminal preserved both times, by design. Hours later, the ceasefire.

The Blockade

JD Vance flew to Islamabad for direct talks with Iran -- the highest-level meeting between the US and Iran since the 1979 Islamic Revolution. A 300-member US delegation led by Vance, Witkoff, and Kushner sat across from a 70-member Iranian team led by parliament speaker Ghalibaf and foreign minister Araghchi. Pakistan’s PM Sharif mediated. They talked for 21 hours across three rounds. The first round was indirect. The second and third were face-to-face.

They failed.

Iran refused to commit to not pursuing nuclear weapons, refused to stop funding allied militant groups, and refused to open the strait toll-free. Iran’s parliament speaker Ghalibaf later claimed they were “inches away from an MoU” and accused the US of moving the goalposts at the last moment. Vance boarded Air Force Two without a deal.

Hours later, Trump announced a naval blockade. “Effective immediately, the United States Navy will begin the process of BLOCKADING any and all Ships trying to enter, or leave, the Strait of Hormuz.” The Navy would “seek and interdict every vessel that has paid a toll to Iran.”

On April 13th, the blockade went into effect. The USS Abraham Lincoln carrier strike group is leading the operation with 11 destroyers and the USS Tripoli amphibious group -- roughly 15 warships in total. Mine-clearing operations are underway. Trump warned that Iranian fast attack ships approaching the blockade would be “immediately ELIMINATED.”

Iran’s response? They called it piracy. The IRGC threatened “new forms of warfare” and warned that if Iranian ports are threatened, “no port in the Persian Gulf will be safe.” Thousands rallied in Tehran, Kermanshah, Shiraz, and Yazd -- though the protests are mixing anti-blockade and anti-regime sentiment, with university students chanting against both. Iran is now threatening to implement a permanent mechanism of control over the strait -- not a temporary toll, but a permanent fixture.

On the Lebanon front, it keeps escalating. Israel launched a ground assault on Bint Jbeil -- a town that saw some of the fiercest fighting in 2006 -- and aims to occupy southern Lebanon up to the Litani River. Over 2,000 people have been killed in Lebanon since the war started. Hezbollah’s chief has called on Lebanon to cancel talks with Israel entirely.

China and Russia vetoed a UN Security Council resolution calling for the strait to reopen. Both had been granted toll-free passage by Iran back in March. They’re not in a rush to change the arrangement.

As of right now, no shots have been fired at the blockade. Iran hasn’t sent warships to challenge it. But Iranian-bound vessel traffic is at a standstill. Oil spiked past $103 on the blockade news, then fell back to around $95 on speculation that talks might resume. It’s been a rollercoaster -- but oil is still up over 30% since this war started.

Where This Goes

So we went from a precarious ceasefire to failed peace talks to a US naval blockade in under a week. The toll booth Iran set up is now being challenged by aircraft carriers. The mines Iran laid are being cleared by the Navy. And the strait that was supposed to reopen is the site of what could become the most significant naval standoff since the Tanker War in the 1980s.

And you can feel it in the real world. Gas prices in the US are up over a dollar a gallon since this started, tracking toward $5 if the blockade holds. Jet fuel has spiked 95%. Sri Lanka reintroduced wartime fuel rations. The ECB is warning that Germany and Italy could tip into recession. When oil gets expensive, everything gets expensive -- it’s not just fuel, it’s plastics, fertilizer, the supply chain that moves every physical product on Earth. One 21-mile chokepoint cascading into food inflation, which cascades into a global recession. We covered this in the last article. It’s happening faster than anyone projected.

Trump says Iran called and wants to negotiate. Pakistan is pushing for another round of talks. But Iran’s demand list hasn’t changed -- unfreezing assets, lifting all sanctions, uranium enrichment rights, and an end to Israeli attacks on Lebanon. The US hasn’t budged on any of those. And that passageway now has 15 warships sitting on it.

If you watch the strait in God’s Eye right now, the picture is stark. Where there used to be 130 crossings a day, there’s almost nothing. The data doesn’t editorialize. It doesn’t have an opinion. It just shows you what’s actually happening -- and right now, what’s happening is that the most important waterway on Earth is functionally closed, with two of the world’s most powerful militaries staring at each other across it.

I’ll keep God’s Eye running to monitor the situation as it continues.

Previous God’s Eye videos:

• The Chokepoint: Strait of Hormuz

• Building God’s Eye to Monitor Iran in 4D

• Ex-Google Maps PM Vibe Coded Palantir In a Weekend

If this gave you something to think about, share it with someone who should see it. This affects all of us.

-Bilawal

Share

Since then, the crisis moved to the water.

The Strait of Hormuz is 21 miles wide. Some of the vessels that transit it are longer than the Empire State Building is tall. On a normal day, 130 ships squeeze through carrying a fifth of the world’s oil. Right now, almost nothing is getting through. Iran has effectively shut down the most important waterway on Earth.

So I retooled God’s Eye to follow it -- swapped the flight transponders and satellite passes for maritime AIS beacons, every vessel broadcasting its position, heading, speed, and flag state through that narrow passage.

That’s the thing about open-source intelligence. The data layer for almost everything is just... out there. Flights, satellites, GPS interference, shipping -- different feeds, same principle. Fuse them together on a timeline and a globe, and you start seeing things you probably shouldn’t be able to see.

I’ve been recording since February 25th. The video breaks it all down visually. But there’s more in the data than I could fit in 15 minutes, and some of it is pretty alarming.

Watch the full video here:

What The Data Actually Shows

Before the strait closed, traffic was robust -- 130 crossing events per day on average. Ships stream through this narrow passageway around the clock.

Then Operation Epic Fury goes down. And suddenly we’re at single digits. Some days, zero.

A 92% drop.

If you pull back and watch the time-lapse, it’s eerie. Dozens of massive tankers just... parked in the open ocean. Literal parking structures in the middle of the water, waiting for their moment to get out of dodge.

The inbound and outbound traffic lines tell the whole story. Pre-crisis average: about 130 crossings per day. Post-crisis: about 10. Choke out a fifth of the world’s oil supply and you start hearing about it everywhere -- South Korea declared a state of emergency, Pakistan is rationing fuel.

The Toll Booth

So here’s where it gets interesting. The ships that are making it through aren’t taking the normal route. The main transit lane through the deepest part of the strait? Completely empty. Instead, the handful of vessels getting through are hugging the northern coastline -- right up against Iran.

What’s been reported is that Iran has effectively set up a toll booth. Make a deal with us, we’ll let you through, and you get an IRGC escort for the privilege.

The price? Reports say $1 to $2 per barrel. These tankers carry two to three million barrels. So we’re talking $2 to $6 million per ship, per transit. Iran is making a killing -- literally formalizing the process.

Look at the ships that are getting through and you start seeing the pattern. India-flagged LPG tankers. Pakistan-flagged crude oil carriers. Bolivia. Liberia. Comoros-flagged vessels, where god knows who the actual owner is. Countries that are so dependent on petrochemicals that they don’t have a choice.

And this matters beyond just energy. Crude oil is obvious -- fuel. But the byproducts, the petrochemicals, are responsible for fertilizer production. Which means this isn’t just about your commute getting more expensive. It’s about whether countries can feed themselves.

Dark Vessels

Here’s where it gets really wild. A bunch of ships are turning off their AIS transponders right as they transit the strait.

AIS -- Automatic Identification System -- is basically the ship equivalent of a flight transponder. Every commercial vessel broadcasts its position, heading, and speed. When you have all this data, you can spot the gaps.

Take the Jag Vasant, an India-flagged LPG carrier. On the 23rd, you can watch it mosey on over toward the strait, and then right as it’s about to cross -- the signal disappears. Dashed line. Dark transit. Probably getting an IRGC escort. Then when it’s safely through, the signal pops back on.

This pattern repeats over and over. Iranian cargo vessels doing it. Comoros-flagged carriers doing it. Ships going dark for the crossing, then reappearing on the other side like nothing happened.

And then there’s the metadata. Some ships are transmitting AIS messages that read “ALL MUSLIMS ON BOARD” or “ALL CHINESE” or “NO IL LINK” -- trying to signal their way out of getting blown to smithereens.

Think about how low-end that feels. You’re a massive oil tanker worth hundreds of millions of dollars, and your survival strategy is transmitting a text message over a decades-old transponder protocol.

The Bypass

So if only a handful of ships are making it through the strait, how does oil get out of the Gulf?

Pipelines.

Saudi Arabia has the East-West crude oil pipeline that runs across the entire desert to the port of Yanbu on the Red Sea. Saudi Aramco built this thing for exactly this scenario. The UAE has the Habshan-Fujairah pipeline -- built specifically so that if Iran ever holds the strait hostage, they can route around it.

These pipelines are currently handling about 42% of normal flow. They’re the reason oil prices haven’t gone completely to the moon.

But here’s the problem: Iran knows about every single one of them. And some of those strikes? They’ve been landing right on Yanbu. Iran is sending a message.

The UAE pipeline? Same thing. Iran wants them to know that the bypass has a target on it.

And if things escalate further -- if Iran mobilizes the Houthis to start targeting ships on the Red Sea side, the side where Yanbu exports its oil -- then the bypass routes get choked too. At that point, we are absolutely screwed.

Here’s the thing about chokepoints though -- they’re connected. The Strait of Hormuz connects to the Gulf of Oman, which connects to the Arabian Sea. The Red Sea connects to the Suez Canal. Close one and the pressure moves to the next. The global economy is just fluid flowing through pipes, and someone’s squeezing.

The Escalation Ladder

The strike map is insane when you see it all at once on the globe. But what matters more than the individual hits is what each one was saying.

The initial US-Israel strikes went after Iran’s nuclear and command-and-control capacity -- Natanz, aerospace complexes, and underground missile bases. Basically saying: we can reach your most protected stuff.

Iran didn’t hit back symmetrically. They went after a power and water station in Kuwait and facilities in Qatar. They didn’t touch the US or Israel directly. They made their friends’ lives miserable instead.

By March 7th, Israel and the US hit refineries -- Shahran oil depot, Tehran refinery. Apocalyptic walls of fire. Iran retaliates at the Sitra refinery complex in Bahrain. Operations halted.

March 13th -- Kharg Island. 90 targets. When you look at it on the map, it genuinely looks like a battlefield, and was quite literally a map in the video game battlefield 3.

And throughout all of this, ships in the Gulf are getting hit. The Skylight -- struck by a projectile, twenty crew evacuated and four injured. The MKD VYOM -- hit the same day by a remotely operated boat, leaving one Indian crew member dead. The Stena Imperative -- first US-flagged vessel attacked, hit by two missiles while docked in Bahrain. The Mayuree Naree, entire superstructure engulfed in flames in the Strait itself, three crew missing. Ship after ship after ship.

Both sides are still showing remarkable restraint, though. Nobody has outright bombed a pipeline, because that would be a point of no return. But the strikes keep landing on infrastructure that gets closer and closer to the line.

And for anyone thinking a ground campaign is the answer -- look at Iran’s terrain. Mountains everywhere. This isn’t Iraq’s flat desert. Anyone who thinks ground operations in Iran look like anything we’ve done before hasn’t looked at a topographic map.

The Ripple Effect

Zoom out and the global picture gets scary fast.

Brent crude went from $70 a barrel to over $110. The US strategic reserves initially absorbed the shock -- prices lagged behind the rest of the world for the first couple of weeks. But that buffer is eroding fast. The national average just crossed $4 a gallon for the first time since 2022, up over a dollar in a single month. California diesel is pushing past $7. And it’s still climbing.

Now look at who’s really exposed -- the countries that import the majority of their oil through the Strait of Hormuz:

• Japan: 80%+

• South Korea: 80%+

• Taiwan: 80%+

• Singapore: ~70%

• India: 60%+

And strategic reserves -- how long each country can survive without that Hormuz import:

• Pakistan: 10 days

• India: 22 days

• Singapore: 78 days

• South Korea: 91 days

• China: 173 days

• Japan: 174 days

• US: 400+ days

• Europe: 400+ days

Pakistan has ten days. India has three weeks. The US is chilling with over a year. The asymmetry is staggering.

The downstream effects are already hitting. The Philippines ordered a four-day work week to cut fuel consumption. Egypt is closing shops by 9 PM to reduce power usage. Bangladesh closed its universities early -- so students don’t have to commute. Fuel rationing across Africa. Fertilizer prices are up 40%.

The Department of Energy is projecting things won’t normalize until 2027.

And here’s the thing people miss -- when oil gets expensive, everything gets expensive. Oil isn’t just fuel. It’s plastics. It’s fertilizer. It’s the supply chain that moves every physical product on Earth. One 21-mile chokepoint can cascade into food inflation, which cascades into a global recession.

We can talk about green energy all day, but our civilization as it stands is exceedingly dependent on petrochemicals.

The EIA projects the shock will persist through the summer, keeping Brent crude above $95 for the next two months before dropping below $80 in the third quarter. They expect prices to fully cool to around $64 a barrel in 2027.

Why I Keep Building God’s Eye

We consume so much information about this crisis -- tweets, news reports, hot takes. But you can’t internalize what’s actually happening until you combine all the sources and visualize them at once on a globe.

The shipping collapse. The dark transits. The pipeline bypasses getting targeted. The tit-for-tat strikes escalating. The oil futures climbing. The reserves depleting. Each of these is a headline on its own. Together, on the same map, on the same timeline, they become understanding.

You start to actually grasp what it means when someone says “the Strait of Hormuz is closed.” It means a billion people have three weeks of oil left. It means Pakistan’s strategic reserve wouldn’t last two weeks. It means the bypass pipelines are being targeted too, and that there’s no clean exit.

First it was flights, satellites, GPS jamming, and no-fly zones. Now it’s shipping lanes, dark transits, and oil pipelines. The data layer keeps expanding. The principle stays the same: fuse every public source together and the whole becomes dramatically greater than the sum of its parts. That’s the value of spatial intelligence. Data stops being abstract and starts being geographic, temporal, physical.

I’m heading to TED in Vancouver on April 13th -- curating sessions with robots, drones, OpenClaw, Reddit, and a bunch of incredible people. Once that’s done, my entire focus goes back to making God’s Eye View the most complete model of reality you can actually get your hands on.

Previous WorldView videos:

• Ex-Google PM Builds God’s Eye to Monitor Iran in 4D

• Ex-Google Maps PM Vibe Coded Palantir In a Weekend

If this gave you something to think about, share it with someone who should see it. This affects all of us.

-Bilawal

Share

I actually sat down with the guy running this — Brian McClendon, SVP of Engineering at Niantic Spatial, the man who co-created Google Earth — on The TED AI Show last year. What he told me paints a very different picture than what the headlines are saying. I made a full video breaking this down, plus the dozen other companies quietly mapping the world at every layer, from your doorstep to orbit.

Pokéstop data isn’t what you think it is

The MIT Technology Review story broke that Niantic had licensed its massive spatial dataset to Coco, a delivery robot company, and John Hanke — CEO of Niantic Spatial — said something that set the internet on fire:

“It turns out that getting Pikachu to realistically run around and getting Coco’s robot to safely and accurately move through the world is actually the same problem.”

Reddit and X threads blew up. Some calling it a betrayal of trust. Others saying you agreed to the terms of service. But here’s what most people missed.

Pokéstop data isn’t what you think it is. It’s not a surveillance panopticon of your neighborhood. These are static captures of parks, statues, murals, landmarks — the places people congregate. Brian described it as “building the map from the bottom up, from the locations where people spend time.”

Think of those twenty million waypoints as basically the inverse of what Google mapped with Street View. Google mapped the drivable streets. Niantic mapped where people actually hang out. Cool data, genuinely useful for visual positioning — but very different from what the headlines imply.

So unless you’re concerned about a panopticon of all the fountains in the world, I wouldn’t be too worried.

From Pokémon to centimeter-accurate positioning

The map doesn’t just tell you where things are — it stays current. And that might be the most important part. Brian told me: “My rule of thumb has always been that roads change at 1 to 2% a year, and that local businesses change at 10 to 20% a year. Keeping your map up to date is a really big challenge. You need people on the ground.”

I spent six years at Google working on what’s called a Visual Positioning System — the tech that uses cameras, LiDAR, and 3D maps to let a device know exactly where it is down to the centimeter. GPS can find your position within five meters of accuracy — and five meters is the difference between a sidewalk and the middle of the road. A delivery robot doesn’t care about the highway. It needs to know: is there a curb here? A bench? A set of stairs?

That’s exactly what Niantic is now building with this data — centimeter-level accuracy, from the ground up, using the exact scans that players collected while catching Pokémon. Traditional maps can’t keep up. A map built by satellites gets updated maybe once a year. A map that gets updated every time someone plays a game? That can keep pace with the real world.

And now they’re taking it further with a neural version of this tech. You can think of it like Rainbolt — the GeoGuessr player who can identify a location from a single blurry photo? Brian put it this way: “I like to think of him as building a neural net in his brain from studying hundreds of thousands or millions of these pictures. And what I think we are talking about with a large geospatial model is to reproduce that neural net by feeding it not a million photos, but hundreds of billions of photos.”

A Large Geospatial Model that can localize you anywhere on Earth from a single photo. That’s where this is headed. And the training data? Already collected. By people catching Pikachu.

Why people are actually upset

And I get it. Because the anger isn’t really about what the data contains. It’s about how the data was collected.

People scanned those Pokéstops to make the game better. They wanted a higher-quality AR experience in their park. They wanted their neighborhood to look good in the game. They were proud of their local landmarks and wanted them represented well. Brian confirmed this:

“People are really proud of their neighborhood, of their city, of their landmarks. And so being able to have a high quality representation of their neighborhood is, I think, a strong motivator.”

That’s a genuine, wholesome motivation. You’re making your game better. You’re contributing to your community’s digital representation. Cool.

And then one day, you find out that the same data is now training a delivery robot to navigate a sidewalk.

That feels different. Even if the terms of service technically allowed it. Even if the data is anonymized. Even if the use case is completely benign. It feels different because the intent shifted underneath you. You were playing a game. You weren’t signing up to be a data collector for a robotics company.

Brian himself acknowledged the side effect nature of it:

“The games themselves were not designed to build this map. The map became a follow-on side effect of making the games better.”

And that’s honest. But “side effect” is a complicated word when twenty million people are the ones producing it. The side effect of what? Of having fun? Of catching Pokémon? That’s not how side effects usually work. Usually, there’s a pill involved, and someone tells you about the side effects before you take it.

The thing is — Niantic isn’t actually the villain here. They split the company. The gaming division was sold to Scopely for $3.5 billion. The spatial platform spun out as Niantic Spatial. Players can opt in to scanning. The scans get anonymized. The incentives are actually more aligned now than they were before.

But the feeling remains. And that feeling — “I was doing something for one reason, and it got used for another” — is worth sitting with. Because this pattern doesn’t stop at Pokémon Go.

Everyone else is mapping the world too

And lest we forget — Niantic is just one of many companies building their own map of the world right now. They’re all capturing different facets of reality:

🚶 Person-level: Axon body cams on hundreds of thousands of officers. Meta Ray-Ban glasses capturing first-person POV at scale — overseas operators reviewing images every time someone says “Hey Meta.” They were in hot water for that recently.

🚗 Vehicle-level: Tesla dashcams on every car in the fleet, massive onboard compute extracting and distilling data to the cloud. Waymo with centimeter-accurate 3D maps of every city they operate in. Fleet telematics cameras on delivery vehicles globally.

🏠 Street & home-level: Flock Safety deploying CCTV across neighborhoods and cities. Amazon with Ring cameras on every doorstep and mailroom — they recently got dragged over that Super Bowl commercial about fusing all their cameras together to find a lost dog. The backlash was so bad they walked the whole thing back. Plus dashcams on every Prime delivery van. Roomba mapping your floor plan every time it vacuums — Amazon wanted that data badly enough to try acquiring iRobot for $1.7 billion before regulators shut it down.

🥽 Headset-level: Apple Vision Pro and Meta Quest build a 3D model of whatever room you’re in every time you put them on. That’s how they know not to let you walk into a wall. Between Ring, Roomba, and your headset — your entire home is being spatially understood by at least three different companies.

📍 Platform-level: Google with Street View cars, aerial planes, satellite imagery, and live location from every Android phone in your pocket. Apple is doing the same with mapping cars — and every LiDAR iPhone is a 3D scanner. And yeah, despite the “Apple is too privacy-conscious” narrative, they’re collecting location data too.

🏃 Trajectory-level: Strava mapped every running and cycling trail on Earth. And accidentally exposed secret military bases in Afghanistan and Syria because soldiers logged their jogs. When you aggregate enough individual trajectories, patterns emerge that were never supposed to be visible. Even Uber — when they plotted millions of rides in Toronto, they essentially reproduced the blueprint of the city. All from location data.

🛰️ Space-level: Planet Labs imaging the entire Earth’s landmass every single day from orbit. Vantor capturing it in higher detail. Iceye doing it in 3D using synthetic aperture radar. If something changes anywhere on the planet — a building goes up, a forest burns down, a military convoy moves — before-and-after imagery within twenty-four hours.

Fused together — body cam, dashcam, doorbell, phone, headset, satellite — every layer of physical reality is being mapped by somebody right now. Different sensors, different angles, different purposes capturing different facets of reality. And most of it is way more invasive than Pokémon Go scanning a statue in a park.

Pokémon Go cracked the incentive problem

The interesting part is how they incentivize the collection. Because mapping the world is expensive.

Google spends massive opex and subsidizes it with local ads and the maps platform. Mapillary tried altruism — asking dashcam owners to contribute footage to departments of transportation. That worked until Meta bought them. Hivemapper pays drivers with crypto.

Pokémon Go cracked something none of them could — a game mechanic that subsidizes the scanning behavior. You’re not building a map. You’re catching Pokémon, or unlocking where one can be caught. The map is just a side effect.

And that game mechanic is honestly kind of brilliant. But it’s also why people feel the way they feel. Because the more fun the incentive, the less it feels like you’re producing valuable data. And the less it feels like you’re producing valuable data, the more jarring it is when someone tells you it’s worth $3.5 billion.

3D scanning is still a niche hobby for reality capture nerds like me. The moment somebody gamifies dense 3D capture at scale — that’s when this blows wide open. And honestly? Looking at the trajectory? That moment might already be here.

What this is really about

The Pokémon Go backlash is real, and the feelings are valid. But it’s also kind of a misdirection. Niantic mapped statues in parks. Meanwhile, Tesla is mapping every road from every angle. Amazon tried to acquire your floor plan. Strava exposed military bases. Planet Labs photographs the entire planet every single day.

If you’re going to be upset about spatial data collection — and there are good reasons to be — Pokémon Go probably shouldn’t be at the top of your list. It should be somewhere in the middle, right between your Roomba and your headset.

Gaming is becoming a way for humans to contribute real-world trajectories that help physical AI learn about the real world. Google does it with live traffic. Tesla does it with autopilot. The mechanic is different but the pattern is identical. The map is always a side effect. The side effect is always more valuable than the original product. And the people who produced the data almost never know what it’s going to be used for until after the fact.

That’s the conversation worth having. Not whether Niantic violated your trust by licensing park scans to a delivery robot company. But whether the “side effect” model of spatial data collection — where the map is a byproduct of something else you wanted to do — is sustainable when the maps become this valuable.

Because the next time someone builds a fun app that asks you to point your camera at stuff, you should probably ask: what’s the side effect?

If you want the full visual breakdown with all the layers mapped out, here’s the video:

If you want the deep dive on how Visual Positioning Systems work and why every major tech company is racing to build one, we made a full video on that too: VPS Video

If this gave you something to think about, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Cheers,
Bilawal Sidhu

Share

Not with a camera. Not with a sensor you bought. Just the WiFi signal bouncing off your body.

I made a video about this last week and it kind of blew up - primarily because people didn’t even think this was possible or long suspected it was; but didn’t think it could get so good.

But YouTube only gives you 11 minutes, and honestly, the rabbit hole goes way deeper than what I could fit in the video. So let’s get into it.

The Basics: Your Router Already Knows You’re There

Here’s how this works. Your WiFi router is constantly sending out radio waves at about 5 GHz. These waves bounce off everything -- your walls, your furniture, your body. When you move through your home, those reflections change in ways that are really specific to what moved and how it moved.

Your router already tracks something called channel state information, or CSI. It’s basically how the router optimizes the signal between itself and all your devices. But CSI also encodes the shape and movement of everything between the router and your devices. And that includes you.

Xfinity figured this out. They shipped a feature called WiFi Motion to every customer with one of the newer gateways. No extra cost. No extra hardware. Your existing network essentially becomes a motion sensor. I’ve got this working in my house -- walk into a room, lights turn on.

Companies brand it as “WiFi Motion” or “WiFi Sensing” or whatever sounds less creepy. But the underlying capability is the same. Your router is doing presence detection. It knows you’re there.

If you want the visual version, here’s the video:

Step Two: WiFi Can See Your Skeleton

Motion detection is just step one. And honestly, it’s the boring step.

What follows requires more than your off-the-shelf router -- needing CSI-capable hardware that exposes per-subcarrier signal data. Which by the way is exactly what 802.11bf is about to change.

In 2023, Carnegie Mellon University published a paper called “DensePose from WiFi.” Using just WiFi signals from a standard router, an AI model reconstructed full body poses. Not just “someone is in the room.” The actual position of your arms, your legs, your posture. Multiple people. Through walls. No camera.

And it’s gotten better since then. A 2024 paper called Person-in-WiFi 3D did multi-person 3D pose estimation from WiFi. The resolution achieved is kind of nuts -- joint localization errors of about 92mm for a single person -- and that’s approaching what you get from camera-based systems. With three people in the room, it degrades to about 125mm.

Meaning it’s still good enough to know that one person is sitting on the couch while another is standing at the kitchen counter. A January 2025 paper called DT-Pose addressed the cross-domain problem, meaning it works across different rooms and environments.

There's an open-source project on GitHub inspired by the CMU research -- ruvnet/ru-view with over 37,000 stars. It implements the signal processing pipeline and presence detection, though full pose estimation requires CSI-capable hardware like an ESP32 -- commodity hardware you can readily buy on Amazon.

Just to be super clear about what happened here. We went from “someone is in the room” to “here is a real-time 3D skeleton of every person in the room, reconstructed from radio waves, through walls, in the dark.” That’s a massive jump and most people don’t even know it happened.

Step Three: WiFi Knows Who You Are

This is where it gets really interesting. And really uncomfortable.

WiFi signals interact with your body in a way that’s unique to you. Your height, your body mass, your bone density, the way you walk, the way you shift your weight, even how you sit -- all of it creates a biometric signature in the WiFi signal.

In 2025 researchers at La Sapienza University of Rome built a system called WhoFi that uses a Transformer neural network to match these signatures to specific individuals. Person re-identification from WiFi. No visual information at all.

But here’s the one that really spooked me. The Karlsruhe Institute of Technology published research in 2025 showing they could identify people with near 100% accuracy across 197 participants. The kicker? They didn’t even need CSI-capable hardware. Their approach uses standard beamforming feedback information -- stuff that’s available on any WiFi device. No special router firmware. No modified hardware. Your off-the-shelf WiFi router can already do this. The researchers explicitly warned about the implications for authoritarian states. The paper was published at ACM CCS, one of the top security conferences.

So let’s stack these up:

• Step one: WiFi detects movement. Already shipping in millions of homes.

• Step two: WiFi reconstructs your precise body pose through walls. Published research, open-source code on GitHub.

• Step three: WiFi identifies the specific person just based on their unique signature. Also published.

We went from a coarse motion sensor to a surveillance system that works in the dark, through walls, on hardware you already own. And the gap between “detect motion” and “identify the person” is a software update. The hardware is already installed.

The Standard Nobody’s Talking About

Here’s something that didn’t make the video because it’s dense and technical, but it’s arguably the most important development in this space.

In September 2025, the IEEE ratified 802.11bf -- the WiFi Sensing standard. This officially makes sensing a native capability of WiFi. Not a hack. Not a research project. A standard.

It defines MAC and PHY modifications for sensing operations across WiFi 6, WiFi 7, and 60 GHz bands. What this means in practice: every major chipset vendor -- Qualcomm, Broadcom, MediaTek, Intel -- is now implementing standardized sensing APIs into their next-generation WiFi chips.

By 2027 or 2028, WiFi sensing will likely be a standard feature of most new routers. Not an add-on. Not a firmware update. Built in from the factory.

That’s the inflection point. We’re going from “researchers demonstrated this is possible” to “every WiFi router on the planet can do this by default.” The infrastructure for ambient sensing at global scale is being standardized right now.

A Billion-Dollar Company Built on This Exact Thesis

A company called ZaiNar emerged in February after nine years in stealth. $1 billion valuation. More than $100 million in funding. The investor list includes Steve Jurvetson from the SpaceX board, the co-founders of Yahoo and Siri, and Skype’s founding engineer.

Nine years in stealth is a long time. What they built is a system that synchronizes existing WiFi and 5G signals to sub-nanosecond precision. Radio waves travel at 30 cm per nanosecond. If you can measure the timing of those signals precisely enough, you get sub-meter positioning that works indoors, outdoors, through walls. No GPS required. No new sensors required.

They’re calling it “the foundational layer for physical AI.” And honestly, that tracks. GPS doesn’t work indoors. Cameras can drift. Nobody has cracked a universal positioning layer for the physical world. ZaiNar is saying that we already have one -- it’s called WiFi and 5G. We just weren’t listening to it correctly.

They’ve filed 100 patents. 90 issued. Zero rejections. Over $450 million in contracts and MOUs. They haven’t named their carrier partners publicly yet, and MOUs aren’t signed deals -- but that investor list and patent portfolio tells you this isn’t vaporware.

Now Scale It to the Battlefield

The same physics applies everywhere. Your WiFi router uses radio waves to sense your body. Defense contractors use radio waves to sense everything else.

Palmer Luckey -- co-founder of Oculus, sold it to Facebook -- started Anduril, which is raising $4 billion at a $60 billion valuation. With a16z and Thrive Capital leading. One of their systems, Pulsar, is an AI-powered electromagnetic warfare platform that passively senses and classifies every RF emission in an area. It identifies what’s producing the signal, geolocates the source, and can deliver an electronic attack. All autonomous. All running on edge AI.

They also deploy autonomous surveillance towers on the US-Mexico border through CBP. Radar, thermal, cameras, all fused together through their Lattice AI platform into a single real-time operational picture. Same principle as your WiFi router sensing your body -- except more modalities and a $60 billion defense contractor behind it.

Now Scale It to Space

HawkEye 360 has 30+ satellites flying in clusters of three, about 500 km up. Their job: detect and geolocate every RF emitter on Earth. Ship radios, radar systems, GPS jammers, and emergency beacons.

They use it to track dark ships -- vessels that turn off their transponders to hide because they’re doing something illegal. But even if you go dark on AIS, you can’t go dark on physics. The satellites still see the radio signal.

Another company, Spire Global, has over 100 CubeSats collecting RF data from orbit for weather, maritime, and aviation tracking. In essence, there are satellites overhead right now whose entire purpose is to light up every single RF emitter on the planet. Every radio, every radar, every phone is a dot on a map.

That little minimap in your home -- your WiFi router sensing your body -- is effectively global. The same physics, just different altitudes.

The Bigger Picture

These are not separate stories. Your living room, ZaiNar, Anduril, HawkEye 360 -- they all share the same fundamental insight. Radio waves are a sensing medium. They always have been. We just didn’t have the compute and algorithms to make sense of it.

We treated radio waves as dumb pipes carrying data from point A to point B. But every WiFi signal, every 5G pulse, every radio broadcast paints a spatial picture of the physical world.

RF is the next spatial modality. It sits right alongside LiDAR, RGB cameras, and infrared -- except it’s already deployed in every building, on every battlefield, and overhead in orbit. It works through walls, in the dark, in terrible weather, on hardware nobody thinks twice about.

For the physical AI stack, this matters a lot. I spent six years at Google working on the Visual Positioning System -- the tech that uses cameras, LiDAR, and 3D maps to let a device know exactly where it is down to the centimeter. VPS already works indoors and outdoors. But RF sensing is another modality entirely. It could complement VPS as an additional sensor layer, or it could stand on its own. Either way, you’re adding all-weather, through-wall spatial awareness using hardware that’s already everywhere.

That’s what makes this so powerful for physical AI in the real world. Not just AR -- robotics, home automation, logistics, search and rescue, autonomous vehicles. Like, all of it.

The Question We Should Be Asking

The question isn’t whether radio waves become a primary sensing layer. That’s already happening at scale. Most people just don’t know their WiFi can see.

The real question is what we do with it.

It took us years to realize that GPS location data -- aggregated at scale -- was both incredibly useful and pretty scary. We’re about to go through that same uncomfortable period with RF sensing. But we have lessons from the past, and we should use them.

Personally, I think cameras outside your home make a lot of sense. Cameras inside your home are kind of creepy. So if you can use passive RF sensing to figure out who’s in a room and what they’re doing -- without a single camera pixel -- that’s amazing for a lot of applications. Home automation. Elder care. Energy management. Accessibility.

And on the defense side, if you’ve got people in the wild doing bad things and evading authorities, being able to sense them through walls without risking a person’s life is genuinely life-saving tech.

But I keep going back to that scene from The Dark Knight. Batman uses sonar from cell phones to catch the Joker. He builds the system. He uses it once. And then he destroys it. Because some tools are too powerful to leave running.

We’re building that tool right now. In every router, in every 5G tower, in every satellite overhead. The question is whether we’re going to be as responsible as a fictional billionaire in a bat suit.

I don’t have the answer. But I think we should be asking the question -- loudly -- while the 802.11bf standard is still fresh and the infrastructure is still being built. Because once it’s everywhere, we don’t get to un-deploy it.

If you’re a builder, researcher, or just someone who wants to map the frontier -- come hang out. I try to make sense of this stuff every week.

-- Bilawal

Share

Then Saturday morning, the Iran strikes happened.

I’m sitting on my couch watching the news and I realize — holy crap, I have the exact infrastructure to monitor this. So I did what any reasonable person would do: I sent a WhatsApp message to my AI agent and told it to start recording everything.

By evening, I had banked every open-source signal I could find before the caches cleared. What came out the other side is a full 4D reconstruction of Operation Epic Fury — from the military and commercial satellites overhead to the cascading airspace shutdowns across the Gulf. Replayable, scrubbable, minute by minute on a 3D globe.

And I did it over the weekend. By myself.

The Layers

Here’s what you need to understand about open-source intelligence: each data source by itself is kind of meaningless. ADS-B flight transponder data? Cool, you know where some planes are. Satellite orbital elements published by NORAD? Great, you can track objects in space. Maritime AIS beacons? Ships are moving around.

None of this is hidden. None of this is classified. It’s all just... out there.

But when you start layering these pieces together on the same timeline, on the same 3D globe, something weird happens. The picture that emerges starts feeling classified-grade. The whole becomes dramatically greater than the sum of its parts.

For the Iran reconstruction, I fused six primary layers:

Commercial flights. 3,400+ aircraft tracked via ADS-B. You can watch the airspace clear out in real time as the strikes approach — flights literally routing around Iran, going “nope, I’m getting the hell out of dodge.” A few brave Turkish Airlines flights went full Leroy Jenkins straight through. Imagine being a passenger on that flight.

Satellite constellations. This one blew my mind. Using publicly available orbital data, I mapped every commercial and military satellite that was in orbit over Iran before, during, and after the strikes. American KH-11 Keyhole spy satellites. Russian BARS-M reconnaissance birds. Chinese Gaofen dual-use microwave sensing satellites. Capella synthetic aperture radar — the same tech that saw through the structures beneath the pyramids (and can more usefully see through clouds and at night). Because of massive constellation density, their natural orbital tracks constantly intersected over the exact same patch of Earth.

The moment that got me was scrubbing through the timeline and realizing: wait, is that a Russian reconnaissance satellite passing directly over the strike zone right before zero hour? And then right after, the next Chinese and American satellites in the constellation lined up to image the aftermath. Everyone wanted to see what just happened. While we can’t get access to the tasking priorities to really know the exact AOIs of all these orbital passes — damn does this feel close.

GPS jamming. This is the sneaky one. Every commercial aircraft broadcasts its GPS confidence level. When you aggregate enough of those signals, you can map where active GPS interference is happening — without any classified sensors at all. You’re basically mining the global fleet of commercial aircraft as a distributed sensor network for electronic warfare. And man, did the jamming light up before the strikes.

Maritime traffic. The Strait of Hormuz is one of the most critical shipping chokepoints on the planet. When the Islamic Revolutionary Guard effectively shut it down, you could watch the panic in real time — tankers scattering, one vessel getting hit, commercial shipping fleeing the bottleneck. Just think about mapping the commercial implications of these movements — South Korea has literally entered a state of emergency, because this closure threatens 70% of their crude oil imports and 20% of it’s LNG.

No-fly zones. The cascading airspace shutdowns told a story on their own. Iran goes dark. Then Iraq. Then Kuwait. Then the retaliatory strikes hit US bases in Bahrain and Qatar — al-Udeid, the Fifth Fleet — and suddenly half the Gulf is a no-fly zone. One of the busiest commercial air corridors in the world, completely emptied.

Strike coordinates. Geolocated from open reporting and correlated against everything else. This is the area where there’s much more work to do. I was able to take public reporting and corroborate it with official sources — but so many of those conflict depending on which journalistic outlet you look at. I decided that would only graduate the highest confidence events to the map.

Layer all of this together and you can watch Operation Epic Fury unfold minute by minute. Not from a CNN desk with a red dot on a country-shaped outline. On a 3D globe with actual flight paths, actual satellite passes, actual GPS interference zones, actual ship movements. All from public data.

The Absence Is the Signal

One of the things that’s hardest to explain about OSINT is that what disappears tells you as much as what appears.

Military aircraft can and do turn off their transponders. So you don’t see them directly. But when you suddenly have a hole in the map — an area where there should be commercial traffic and there’s nothing — that itself is intelligence. When 3,400 flights simultaneously clear an airspace, you don’t need anyone with a security clearance to tell you what’s coming.

The GPS jamming data works the same way. You’re not directly measuring the jamming — you’re inferring it from degradation. Every plane that reports a drop in navigation confidence is an unwitting sensor. Aggregate enough of them and you’ve mapped the electronic warfare environment from publicly available data.

This is why the democratization of OSINT is so significant. It’s not that we suddenly have access to spy satellites. We don’t. The US government has sensor systems that can identify a vehicle from orbit. I have NORAD’s public catalog and some API calls.

But here’s the thing — the AI available to the general public is on a pretty level playing field with what the intelligence community has for analysis. The government has way better sensors. But the analytical tools? Those have been commoditized. That’s the shift.

What Used to Take Quarters

I spent six years at Google, four of them on Maps. I worked on the ARCore Geospatial API, Immersive View, 3D mapping at global scale — fusing satellite, aerial and ground level imagery together for public and private sector applications. I’ve worked with incredibly talented engineers on projects like this. We would spend quarters — literal three-month cycles with full teams — to build what I just built over the weekend.

The AI agent swarm changed everything. During the strikes, I was on my couch. I sent a message to my agent. It started snapshotting every data feed before the caches cleared. When the dust settled, I had all the raw material. Then came the fun part: fusing it all together, aligning the timelines, and building the visualization.

That responsiveness would not have been possible before AI. Certainly not for one person. A couple years ago, you’d need a team just to do the data engineering — normalizing formats, aligning coordinate systems, cleaning timestamps. Now you jump straight to the human analysis part, which is where the interesting work actually is.

There’s a quote from Major Claire Randolph of the Air Force Central Command that I keep coming back to: “If American analysts were doing some of these things, we would classify that as secret or perhaps even top secret. But this stuff is just out there on the open internet.”

She’s right. And it’s only going to get more true. Fuse enough public data together and you get an understanding that “feels” like it should be classified.

The Trillion-Dollar Question

Here’s the part that keeps me up at night.

The US government officially recognized the value of OSINT in 2024. Small teams of analysts punching way above their weight compared to classified systems. That same year, Anthropic’s Claude started operating in classified settings for the Department of Defense. The analytical capability gap between a motivated individual with public tools and a nation-state intelligence apparatus is narrowing fast.

Which raises an uncomfortable question: how much of this is transparency, and how much of it is a genuine security risk?

Journalists track fleet movements by watching TikToks from people who live near naval bases. Satellite imagery that used to be the exclusive domain of three-letter agencies is now purchasable by hedge funds trying to count cars in parking lots. The same commercial satellite capacity that the military buys for routine coverage is available to anyone with a credit card.

I don’t have a clean answer. I don’t think anyone does. The data is just... out there. What’s the alternative — close down the internet?

But what I do know is this: the intelligence monopoly is over. The question isn’t whether individuals and small groups can do things that used to require a security clearance. They already can. The question is what we do with that capability.

I’m building tools that make the world more legible. That fuse public data into spatial understanding. That give anyone the ability to see what’s actually happening — not through the lens of a cable news desk with a red dot on a map, but through actual data, on a globe, scrubbed minute by minute.

WorldView opens to the public in April.

The full video breakdown of the Iran reconstruction is on YouTube:

If you want to follow the build, I’m @bilawalsidhu on X: https://x.com/bilawalsidhu

Bilawal, signing off. ✌️

Share

By the time I woke up, the Palantir co-founder had replied to defend his company.

That wasn’t what I was expecting. But it’s also exactly the kind of thing that happens when you accidentally poke a nerve — and it told me more about what I’d built than any of the positive replies did.

Here’s what WorldView does: it lets you look at any place on Earth through the eyes of an intelligence analyst. Night vision. FLIR thermal. CRT scan lines. Live air traffic. Real satellite orbits. And — the part that really starts feeling surreal — actual CCTV camera feeds draped directly onto the 3D model of the city.

All of it running in a browser. No classified clearances.

Just open a tab

If you want to see WorldView in action, you can check out my full walkthrough on YouTube:

What’s actually happening under the hood

The foundation is Google’s Photorealistic 3D Tiles — the same technology that powers Google Earth’s volumetric city models. I spent six years at Google as the PM who helped build this. We were turning the physical world into a navigable 3D map at global scale. Millions of images. Petabytes of photogrammetry. Entire cities reconstructed from aerial photography.

That technology is now publicly accessible via an API.

On top of that, I layered real-time data feeds:

• OpenSky Network — 7,000+ live aircraft positions, updated constantly

• ADS-B Exchange — crowdsourced military flight tracking.

• CelesTrak TLE data — 180+ satellites in actual orbital paths, tracked in real time. Click any one and follow its orbit.

• OpenStreetMap — vehicle flow on city streets, rendered as a particle system

• Public CCTV cameras — real traffic camera feeds from Austin, geographically located and projected onto the 3D model

And then the shaders. CRT scan lines, night vision (NVG), FLIR thermal, anime cel-shading. Built from studying actual military display specifications — not for the aesthetics, but because those display systems were engineered to extract maximum information from sensor data. Turns out they also look absolutely wild layered over London.

The anime mode is the one nobody expected. Same volumetric city model, same data feeds, same satellite tracking — but rendered with cel-shading that looks like a Studio Ghibli film. The contrast between military surveillance infrastructure and anime aesthetics is jarring in a way that makes you reconsider what “the aesthetic” actually communicates. Switch from anime to FLIR on the same view and the mood shifts from wonder to menace in a frame.

You can stand over Austin at 3 AM in night vision mode and watch a plane descend into the airport. Switch to FLIR and look at Tower Bridge in London through the thermal spectrum, military targeting reticle and all. Pull back to globe view and click on any satellite to start tracking its orbital path in real time.

At some point, it stops feeling like a demo.

Man, it starts feeling magical.

That’s what I said during the recording session. I meant it.

How I actually built this

Between Gemini 3.1, Claude 4.6, and Codex 5.3 — it’s honestly wild what you can build right now.

I didn’t write this code by hand. I described features in voice notes and screenshots, threw them at multiple AI agents running simultaneously, and steered the results. Running sometimes 8 agents at once, each working on a different subsystem. One’s building the satellite tracker, another’s wiring up the CCTV feeds, a third is implementing the shader pipeline.

I don’t touch even Cursor anymore. I’m going straight into the terminal with each agent open in its own window.

When I worked on Google Maps, it took developers months to get an understanding of how to use the geospatial APIs. This was a weekend. One person, a browser, and a few AI models doing the heavy lifting.

Why I built this

I’ve been saying “spatial intelligence” for a while now. In strategy docs, in tweets, in conversations about what I think matters next after language models. But it’s been mostly words.

WorldView is the first attempt to make the thesis visible.

The thesis: we’re building AI that understands the physical world the way it understands text. Not images — space. Not object recognition — spatial relationships, change over time, movement through a scene, the difference between “a car” and “that car, at that intersection, at that speed, at that hour.”

That’s spatial intelligence. And it changes everything about security, logistics, city planning, disaster response, autonomous systems — basically anything that cares about the real world as it actually exists.

WorldView doesn’t have the intelligence layer yet. But it has the view. It has the data fusion. It has the interface that makes you feel, viscerally, what becomes possible when the physical world becomes queryable and programmable.

I spent six years helping build Google’s version of this infrastructure. I’ve been thinking about the next version for a long time.

This is the prototype. The tip of the iceberg.

Why the Palantir founder responded

There’s a moment in the WorldView demo where I switch on what I was calling “God mode.”

Detection overlays. Every vehicle on the street highlighted. Military flights circling overhead with callsigns and altitude data. Satellites in orbit. CCTV feeds projected onto buildings. Panoptic view — the term comes from Jeremy Bentham’s panopticon, the theoretical prison designed so the guard can always see every prisoner, and the prisoners know it, so they behave as if they’re always being watched.

”You can see everything.”

I said it with genuine wonder. Then I sat with it.

Joe Lonsdale replied to the post with:

I get it. From his vantage point, tools like Palantir exist because this capability has always been real — it just wasn’t accessible. The intelligence community has had something like WorldView for decades. The question Palantir answers is: how do you analyze it? How do you act on it? The data was never the moat.

What’s new with WorldView isn’t the capability. It’s the accessibility.

What I built is fundamentally a surveillance aesthetic built on top of public data. Military-grade shaders over open-source feeds. The visual language of classified intelligence systems running in a browser tab, available to anyone with internet access. And the CCTV integration — real camera footage draped onto real buildings, in real time — I actually built that.

There’s a concept I keep coming back to that applies here: the two viellances. Surveillance (the state watching you) versus sousveillance (you watching back). We’ve spent decades arguing about the power asymmetry in one direction — who has the cameras, who owns the data, who gets to watch.

WorldView is sousveillance aesthetics. Same data streams. Same satellite feeds. Same CCTV cameras. But the interface is in your browser, and you control it.

That’s a different power dynamic. I don’t know yet what all the implications are. But Lonsdale responding told me it’s real enough to push back on — and that matters.

Where this goes

WorldView is a demo. SpatialOS is the actual project.

The spatial intelligence stack — the one that builds a continuously updating model of the physical world, ingests sensor data from satellites and cameras and IoT devices, and makes that model queryable by AI agents in real time — that’s what I’m building.

WorldView shows what the view looks like. The thesis is about the intelligence that runs on top of it.

Think of it this way: this demo is to SpatialOS what Google Maps is to Google’s location intelligence infrastructure. The map is the product people interact with. Underneath it — the data pipelines, the model updates, the location APIs that power a thousand other apps — that’s the real infrastructure.

I’m building the infrastructure. The demo just happens to look like something out of a spy thriller.

And apparently that’s enough to get the attention of people who’ve been building the actual thing for twenty years.

I’ll keep writing as it develops.

If this gave you something to think about, share it with fellow reality mappers. The future’s too interesting to navigate alone.

— Bilawal

Share

Meanwhile, Google’s Project Genie rollout tanked gaming stocks across the board. Unity went down 35%. Take Two, Roblox, and Nintendo are all bleeding. The market is betting on a future where world models like Genie 3 make game engines obsolete.

So let’s talk about all of it, cut through the hype, and find what will have real implications on the future.

Seedance 2.0: The Composition King

For Seedance 2.0, the big unlock is multimodal inputs. Not just text and images, but audio files and video references. You can throw in a collection of assets, and the model composes them together into a coherent generation. I predicted this back in December for audio references. It’s February and it’s already here.

The text rendition alone is the nano banana moment for video. Perfect text, synchronized audio, and consistent characters. If you’re a UI/UX designer who’s been suffering through the Figma-to-After Effects-to-prototyping-tool pipeline just to make a product demo, this could change your entire workflow. Throw your designs into Seedance, and you’re off to the races. We were all so focused on tools like Remotion for hard-coded motion graphics, and video models just made all of that potentially irrelevant. The bitter lesson is bitter for a reason.

Visually, where Seedance really separates itself is in action sequences. Models like Veo have struggled with explosions, muzzle flashes, and high-energy cinematics. I tried doing this stuff with Veo a few months ago and it was painful. Seedance nails it with accurate physics, aerial sequences, and slow-motion damage. The camera motion feels natural and grounded, not that floaty AI drift we’ve all grown to recognize. We’re past the uncanny valley on this one.

Kling 3.0: The 4K Powerhouse

Where Seedance is the composition king, Kling 3.0 is the resolution and consistency king. 4K outputs. 15-second multi-shot sequences with self-consistent characters and environments across camera angle changes. Best-in-class native audio generation. Phenomenal lip sync.

When you compare both Chinese models side-by-side against Sora and Veo on the same prompts, the edge is clear, not just in image quality, but in how natural the voice sounds and how grounded the cinematography feels.

Why China Keeps Winning

The answer is threefold: data, labeling quality, and workforce.

Generation quality is directly proportional to how scared companies are of getting sued. Adobe trains on licensed data only and has no state-of-the-art models. OpenAI made a strategic call with Sora 2 to train on major IP and deal with takedowns later. Google has YouTube. But China trains on absolutely everything under a completely different IP regime. And when you’re open-sourcing the models, there’s nothing anyone can do about it.

On top of that, China has film students doing dense video labeling — people who can describe scenes with actual cinematic vocabulary. That produces fundamentally better training data than more engineering-first annotation used by other labs. As Martin Casado posted, “Models are largely smart people caches. The harder the people work and the cheaper they are, the better the model.”

The only dimension where the US leads is compute. The best chips are still controlled by the US and its allies. But on data and labeling? China’s at parity or ahead.

World Models vs. Game Engines: The Real Question

Now let’s shift gears. Google’s Genie 3 sent the gaming industry into a panic, and the question everyone’s asking is whether game engines are actually cooked.

They’re not. Not yet. But the convergence is real, and I think the right framing is explicit vs. implicit approaches racing toward the same goal.

The explicit approach takes existing tools like Unity, Maya, Blender, and supercharges them with AI. Text-to-3D generation, visual language models composing scenes from concept art, AI animation tools like Cascadeur, and LLMs writing your Unreal Engine nodes. The pitch: automate the drudgery while keeping everything addressable and editable.

The implicit approach says screw it, scale the neural network. More data, more compute, and out comes something that eats the game engine for lunch. And Genie 3 is the strongest proof point yet: Google went from training on gameplay footage to a chunk of YouTube, and it just worked. Emergent physics, lighting, and spatial consistency that Genie 2 couldn’t touch.

And here’s what people keep getting wrong: just because Genie outputs pixels doesn’t mean it’s “just 2D.” The 3D structure lives in the model weights. We’ve seen this with NeRFs and Gaussian splats. The medium is new, and the possibilities — stepping inside artwork, walking around a photo, interactive feeds of remixable experiences — are things conventional game engines are actually terrible at.

I keep thinking about self-driving cars. We went from a federation of inspectable classical systems to a black box where pixels come in, and driving policy comes out. That took a decade. Why wouldn’t the same thing happen here on a compressed timeline?

Where I Net Out 🔭

On video models: China has the edge. Seedance 2.0 and Kling 3.0 are both production-ready in ways that US models aren’t. Google has the best shot at closing the gap with the next Veo, but the gap is real, and it’s widening.

On game engines: Rockstar isn’t cooked — yet. Genie is still a slot machine, stochastic and unpredictable. But that’s where things are, not where they’re going. And the companies building world models aren’t actually trying to build the next GTA. They’re building AI that understands the physical world because that’s a fundamental prerequisite for AGI. Reinventing the game engine is just a byproduct of teaching AI how reality works.

That might be the most important takeaway from this entire week.

If this gave you something to think about, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Share

Cheers,
Bilawal Sidhu

bilawal.ai

We’re recording every movement, every heartbeat, and can reproduce a game from any angle in 3D. But barely any of that reaches your living room. In my latest video, I break down the companies and technologies that are starting to close that gap.

Watch the full video here:

Video Chapters:
0:00 – The Immersion Gap
0:48 – Computer Vision & Hawk-Eye
1:42 – Player Tracking & Wearables
2:08 – AI Analytics & Real-Time Stats
3:07 – VR Sports Viewing & Why It Fails
4:16 – Radiance Fields & Gaussian Splatting
5:10 – Viewpoint Pro: Virtual Replays
5:59 – Arcturus: 4D Gaussian Splatting
7:31 – Muybridge: Virtual Camera Systems
9:53 – F1 Vision Pro Experience
11:26 – XR Sports Alliance
11:47 – COSM: The $600 Ticket
13:04 – The Future of Sports Broadcasting

Subscribe now

The Data Foundation Already Exists

This starts with computer vision. Sony’s Hawk-Eye system mounts high frame rate, perfectly synchronized cameras across a stadium and reproduces the 3D location of every player with millimeter accuracy. The NFL is embedding RFID tags in shoulder pads, pylons, and even the ball itself — essentially 3D AirTag coverage of every moving entity on the field.

All of that raw data feeds into AI platforms. The MLB partnered with Google Cloud to analyze 15 million data points per game in real time. The NFL runs Next Gen Stats through AWS. Bat speed, hit strength, and throw length are all computed instantly. This data already exists. The question is: what do we do with it on the viewing side?

Why VR Keeps Failing

We’ve had immersive headset experiences for years. Apple acquired NextVR, which built camera systems for stereo sports capture, but has yet to do anything with the acquisition. There are a number of other apps to watch sports on Quest headsets, and Meta has sold around 20 million headsets to date. But how many are actually in use? Likely well under 10 million. More importantly, sports are inherently social. You’re with your friends, drinking a beer and eating stadium food. Strapping on a headset kills all of that. The slight immersion gain just doesn’t justify the tax, and that truth has burdened headsets for years.

The New Wave of Volumetric Capture

But new tech is making the immersion far more compelling. Arcturus is using 4D Gaussian Splatting — multiple cameras reproducing the action in full 3D so you can pause at any moment and view it from any direction. The kind of angles The Matrix spent years of R&D to achieve is now possible for any moment in an MLB game.

Muybridge is approaching it differently — deploying strips of cameras non-invasively around a court and using multi-plane images to create a virtual camera slider. No neural radiance fields, no bulky rigs. Just an elegant array that acts as a virtual cinematographer, following the action without ever getting in a player’s way. MPI is tech I actually had a chance to work on during my time at Google, so it’s exciting to see it applied to live sports.

Then there’s the F1 Vision Pro experience — what started as a viral concept is now arguably one of the best reasons to own a Vision Pro. A spatial mini-map of the track, multiple camera feeds, car tracking with smooth easing — all contextualized in a 3D canvas. Apple has the F1 rights and the hardware. This is becoming a very strategic play for them.

The XR Sports Alliance is also working to connect all of these pieces — bringing together streaming providers, chip makers like Qualcomm, and sports organizations to make it easier to onboard games into immersive platforms.

The Winner (So Far) Doesn’t Use a Headset

Perhaps the most telling signal is that the most commercially successful immersive sports experience today isn’t in a headset at all. It’s COSM — a massive 8K wraparound LED dome, essentially a mini version of the MSG Sphere dedicated to sports viewing. People are paying up to $600 a ticket. You get the best seat in the house, insane visual quality, and most importantly, no dorky headset on your face. You can talk to your friends, sip your drink, and actually enjoy the social experience that makes sports what they are.

Where This Goes

The experience is likely going to bifurcate. On one side, premium in-person venues like COSM. On the other, personalized broadcast experiences — imagine creators on Twitch or YouTube casting their own version of a live event with custom camera angles and commentary, rather than reacting to the same vanilla feed everyone else gets.

All the pieces are there. The data is being collected. It’s a question of how it reaches you as the consumer. I’d argue that COSM’s sold-out domes prove the demand exists. Now we wait for the hardware to catch up.

Check out recent discussion on X:

• Arcturus’s 4DGS tech that will change sports viewing

• The new SOTA AI video model

• The hybrid approach of explicit 3D data combined with implicit Gen-AI creativity to build world models.

• Dead Internet Theory is becoming a reality. (Moltbook)

  ---

If this gave you something to think about, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Share

Cheers,
Bilawal Sidhu

We have officially moved past simply watching videos; we are now stepping inside them. With today’s release, the boundary between static media and interactive simulation is vanishing.

Watch the full video here:

The Move from Watching to Walking

For the past year, we’ve been tracking the rapid evolution of video generators. But Genie 3 changes the substrate entirely. Instead of a linear clip, you provide a text prompt or an image, and the model—powered by Nano Banana Pro—spawns an interactive, 3D simulation that you can navigate in real-time.

During my early hands-on testing, I explored everything from infinite backrooms to underwater shipwrecks with impressive caustic effects and dynamic lighting. One mind-blowing moment, however, was stepping inside a vintage photo. The model inferred an infinite 3D environment from it, allowing me to walk the streets of a retro San Francisco and explore corners that weren’t visible in the original frame. I can’t help but imagine the incredible opportunities this will open up for educators to step inside a moment in time, or for an artist to bring a piece of theirs to life.

Emergent Intelligence: The GPS Moment

One of the most fascinating aspects of Genie 3 is its emergent behavior. In one generation, I prompted a first-person view holding a GPS device in a desert. As I moved around in the environment, the mini-map on the device synchronized perfectly with my movement—an affordance the model learned simply from its training data, not through explicit programming.

Insights from Google DeepMind

I had the chance to sit down with Jack Parker-Holder (Research Scientist) and Diego Rivas (Product Manager) from the DeepMind team to discuss what this means for the future of creativity.

A few key takeaways from our conversation:

• Virtual Production: Creatives can use Genie as a tool for rapid ideation—essentially teleporting onto a virtual set to test camera angles and lighting before a single frame is shot.

• Prototyping Worlds: While it isn’t a full game engine, it allows developers to iterate on world concepts at a speed that was previously impossible.

• Shattering the 60-Second Barrier: While generations are currently capped at one minute, the team is already looking at several options to extend beyond that, and it’s not a fundamental barrier to overcome.

Early Access is Live

If you are a Google Ultra user in the US, early access is live today, January 29th. You can jump into Project Genie right now and start building.

For those looking to dive deeper into the technical mechanics and creative implications of this tech, I highly recommend checking out these previous deep dives on the channel:

• Is Google’s Genie 3 About to Replace Game Engines?: My original overview of Genie 3.

• World Models Will Break the Internet: A foundational look at why world models are the next frontier beyond LLMs.

Many thanks to the Google DeepMind team for the early access. We’re at the very beginning of this journey into playable reality. I’m excited to see what worlds you all decide to build.

If this gave you something to think about, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Share

Cheers,
Bilawal Sidhu

The narrative has been shifting. Every major AI lab—Google, Meta, Luma, Runway—is no longer racing to build a better video generator. They’re racing to build a world engine.

In my new video, I break down the technical shifts that got us here: from multimodal training that unlocked native audio-video generation, to autoregressive architectures that finally solved consistent editing, to real-time models that predict the next frame on demand. And of course, my predictions on where we’re going next.

Watch the full episode on YouTube:

YouTube Chapter Links:

• 00:00 - Intro

• 00:33 - Video Models in 2025

• 01:15 - Video With Audio Generation

• 01:54 - Emergence of World Models

• 03:50 - Multimodality

• 05:29 - Solving Inconsistency

• 07:01 - Image Models in 2025

• 07:47 - Diffusion vs Autoregressive Models

• 09:20 - Benefits of Multimodality with Image Generation

• 11:34 - Real-World Simulation Using Video Generation

• 16:02 - Real-Time Interaction in World Models

• 19:19 - 3D Native World Models

• 22:49 - What’s Coming in 2026

• 25:34 - The Effect on Content Creation Workflows

• 29:03 - What We Won’t See In 2026

• 30:12 - Conclusion

Subscribe now

The Slot Machine Is Dead

Remember re-rolling generations endlessly hoping for something usable? Autoregressive image models like Nano Banana changed that. By predicting tokens sequentially—with a thinking step to plan composition before generating—these models reason about their own output before you ever see it. The model does the slot-machining for you now.

From Synthetic Camera to World Engine

Video diffusion models treat a clip as one 3D block—height, width, time—and denoise it all at once. Great for consistency within a generation, terrible for real-time interaction. Autoregressive models like Genie 3 flip the paradigm: predict the next frame, accept user input, repeat to infinity. Suddenly you’re not just generating video—you’re directing a world that responds.

The Hybrid Future

The 2026 workflow is taking shape: use World Labs or SpAItial to build your environments in 3D, then prompt next-gen video models with image references, audio clips of your characters, and let the autoregressive architecture retain likeness across generations. Virtual production meets generative media. Startups like Artcraft and Intangible AI have the right idea.

On The Horizon 🔭

The missing piece: embodied agents within these world models. Imagine directing AI characters on a sound stage who reason about their environment and deliver performances in real-time. We’re not there yet—but that’s where this is heading. And it’s an area where robotics and entertainment share more in common than you might imagine.

Check out recent discussion on X:

• Every video is now a spatio-temporal portal you can revisit from any angle

• Apple’s 2D-to-3D just dropped—and the internet had ideas

• Casey Neistat is posting about Gaussian splatting. Reality capture has officially gone mainstream.

• NORAD Santa Tracker is getting suspiciously realistic

• Classical 3D for control, generative AI to finish—the hybrid workflow is unbeatable

If this gave you something to think about, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Share

Cheers,

Bilawal Sidhu

https://bilawal.ai

As someone who spent six years building this tech at Google, I’ve seen firsthand how we’re moving from satellites in the sky to pixels on the ground.

Check out my latest video for a deep dive into the necessity of VPS and a look at the tools shaping this future:

VPS: The Precision Layer for the 3D World

The solution to GPS’s coarse accuracy is VPS (Visual Positioning System). Instead of relying on timestamps from satellites 12,000 miles away, VPS uses the camera on your device to look at the world and match it against a pre-existing 3D model.

• Sub-meter Positioning: While GPS is accurate to a few meters, VPS enables centimeter-level accuracy.

• Rotational Accuracy: VPS eliminates the “which way am I facing?” problem, providing precise orientation that’s critical for AR and robotics.

• The Google & Apple Advantage: By leveraging over a decade of Street View data, Google can provide high-fidelity localization across 100+ countries, and Apple can use imagery from its “look around” feature in 80+ cities.

Why This Matters: Robots, Drones, and AR

We are moving into an era of autonomous agents and spatial computing where “close enough” isn’t gonna cut it.

• Robotics: A delivery robot needs to know the difference between the street and the sidewalk. VPS provides the map-to-world alignment to make that possible.

• Contested Environments: In areas where GPS signals are jammed or denied, drones can use VPS and a camera to navigate perfectly.

• Persistent AR: Imagine leaving a digital sticky note on a specific piece of equipment for a colleague. VPS allows 3D annotations to stay locked to the physical world, even when you walk away.

The Emerging Ecosystem

It’s not just the tech giants. A new wave of startups like Multiset.AI is making it possible for anyone to scan a street with an iPhone and create a localized 3D model in real-time. We’re even seeing high-end capture devices like XGrids’ PortalCam using Gaussian Splatting to turn LiDAR and images into photorealistic, navigable scenes.

The Big Picture: Spatial Intelligence 🔭

VPS is the “where,” but the next frontier is the “what.” The real magic happens when we combine visual positioning with AI that understands the semantic meaning of the environment—a field known as Spatial Intelligence. We are moving toward world models that don’t just map space but understand it, opening the door for robots to step inside.

If you found this useful, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Cheers, Bilawal Sidhu

bilawal.ai

Subscribe now

YouTube is expanding its ecosystem by introducing Playables Builder, a prototype tool powered by Gemini. This move pushes the interactive layer on the platform into the gaming sector, allowing creators to build engaging experiences that live inside the site for creators’ audiences to play.

• Multimodal Prompting: Using Gemini’s multi-modal capabilities, creators can build these experiences using natural language, images, or video clips as a starting point.

• Audio and Game Logic Generation: The tool goes beyond visuals, allowing for the generation of sound effects and the complex underlying logic of the games, which is where it really shines.

• Expanding Audience Engagement: This opens up a massive toolkit for audience participation. Creators can now prompt interactive games that relate directly to their video content. On top of that, audiences can duplicate anyone else’s game and remix it to their liking.

• Zero-Friction Access: These Playables are designed to be played on both desktop and mobile without requiring any downloads or leaving the YouTube app.

The tool was extremely engaging to work with and offered an awesome glance into the future of creating interactive experiences.

You can check out my games, Xeno Rancher and Cosmic Cookout, which are live on Playables now!

Also, check out my video for more insight into the ideation & creation process behind these games:

YouTube vs Roblox – The UGC Battle

This move represents a strategic shift in the user-generated content landscape. For years, Roblox has been the workshop for interactive creator-built games, while YouTube was the amplifier where games were recorded and watched alongside most other video content. However, as the barriers to creation blur, YouTube is blending the interactive layer into its own feed.

While Roblox remains a heavyweight with a deep economic ecosystem, YouTube’s advantage is zero friction. You don’t need to open a separate app or learn complex studio software; you can prompt an experience into existence where your audience already lives. Moving us toward a world where watching a video is just one part of creator-led content.

Adobe Inside ChatGPT: The Creative Partner Evolution

Adobe is expanding its workflow options by embedding tools directly into ChatGPT. This moves beyond simple AI media generation and into more professional, non-destructive control.

Photoshop as your Orchestra

You can now upload an image to ChatGPT and use natural language to command Photoshop’s professional engine.

• Fine-Tuned Control: It offers sliders to dial in the perfect look, ensuring you have the final say on the output.

• Non-Destructive Edits: Whether it’s making a subject pop or adding a long-exposure effect, the tool works intelligently with layers and effects without destroying original pixels.

You’re the conductor. Photoshop is the orchestra.

Solving Desktop Chores in a Single Thread

The integration allows you to move between different creative tasks within a single chat thread.

• Acrobat: You can merge multiple PDFs (like scripts and proposals), reorder pages, and compress files for email through simple prompts.

• Express: Use those documents as context to have Express find social media templates and generate a first pass of copy.

• Cross-Tool Harmony: You can use Photoshop to remove backgrounds from headshots and drop them directly into an Express design for a social post—all in the web.

The Big Picture: Agentic Creativity 🔭

We are shifting from AI as a generator to AI as a creative partner that can execute complex tasks across professional applications. By removing the need to dig through endless menu options or learn an entire game engine, the barrier between a creative idea and its execution is effectively disappearing.

A Special Thanks

I want to extend a massive thank you to both YouTube and Adobe for providing early access to these tools and for sponsoring this exploration. Seeing how Playables and Adobe are evolving to empower creators has been an incredible experience.

If you found this useful, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Cheers, Bilawal Sidhu

https://bilawal.ai

Subscribe now

Meta just dropped a massive open-source release that bridges that gap at scale in two model families: SAM 3 and SAM 3D.

Together, they form a complete perception system that also solves the training data bottleneck that has held back 3D AI for years. Meta did this by applying the Large Language Model playbook to the physical world.

We are looking at the infrastructure for Physical AI.

For a complete breakdown of the models and the data engine behind them, watch my video below:

SAM 3: Understanding “Concepts,” Not Just Objects

Previous segmentation models were limited to fixed categories—person, car, dog. If it wasn’t in the training set, the model was blind to it.

SAM 3 changes the game by detecting visual concepts via text prompts. It doesn’t just find an object; it finds every instance of a description. You can prompt for things like “people sitting down,” “person holding a package,” or “person on the ground” (essentially baked-in fall detection).

Meta achieved this by mining Wikipedia to build a massive ontology of relationships, specifically focusing on the long tail distribution—those rare edge cases that rarely show up in training data but matter critically when they do.

SAM 3D: The “LLM Playbook” for Geometry

The real revolution here, however, is in 3D.

Historically, getting 3D training data was a nightmare. You either had expensive synthetic assets or messy, 3D scans of the real world. Neither scaled well.

Meta’s solution? RLHF for 3D reconstruction.

They built a data engine that mimics how we train LLMs:

1. Generation: Models generate multiple mesh options for a scene.

2. Verification: Human evaluators rank them.

3. Loop: The best data feeds back into the model, which improves the data engine, which improves the model.

Suddenly, you can annotate close to a million images with 3D ground truth. This industrializes the process of crossing from synthetic simulation to real-world perception.

To handle the complexity of the physical world, Meta released two versions: SAM 3D Objects, which reconstructs general objects and scenes, and SAM 3D Body, which focuses on accurate human pose estimation.

The Convergence: A Unified Perception Stack

When you combine SAM 3 (Concept Detection) with SAM 3D (Spatial Reconstruction), you get a unified stack for understanding the physical world.

• Consumer: In Facebook Marketplace, you can now visualize if a lamp fits in your living room in 3D, just from a 2D seller photo.

• Robotics: Robots get a queryable world, allowing them to segment and track objects in 3D space for manipulation and navigation.

• VFX: Artists get automated rotoscoping and 3D pose estimation, making it trivial to reskin reality.

The Quiet Danger

However, there is a double-edged sword here that we need to talk about. This same technology is also the perfect infrastructure for comprehensive surveillance.

We now have a deployment-ready system that can detect any concept, track it across video, and reconstruct its precise 3D pose and spatial relationships. Because this is open-source and commercially licensed, the gap between what is technically possible and what is legally regulated just got a lot wider.

On The Horizon 🔭

We are moving toward a world where physical spaces are as programmable and queryable as a database.

SAM 3 and SAM 3D are the early foundation layers for this shift. Whether it’s for wildlife tracking, populating the metaverse, or giving spatial intelligence to the next generation of robots, the tools are now out in the wild.

Go build something amazing (and responsible).

If you found this useful, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Cheers, Bilawal Sidhu

https://bilawal.ai

Subscribe now

A drone sees a hostile behind a building.
That person immediately appears in your headset as a red silhouette—even through the wall.
Spot someone in a window, and every drone in the air locks onto the same target.

It’s a shared battlefield map updated in real time, with both humans and autonomous systems operating on the same information.

But this tech has larger implications beyond the defense sector.

To see the full breakdown of the system, check out my latest video:

Human & Robot Views Combined

What makes EagleEye novel is that it brings people and machines into one operational environment. Robots already perceive the world with structured geometry: line-of-sight, depth, object permanence, and path planning.

This system aligns human perception with that same model.

Thermal overlays and night-vision fusion aren’t new. The military has been able to see in the dark, peer through smoke, and blend sensor layers for years. The novel shift is AR becoming the interface for all of it—turning raw sensing into something spatial and intuitive.

Lattice is the engine behind this. It’s Anduril’s real-time operating platform that fuses every sensor in the battlespace—drones, satellites, ground vehicles, soldier-worn devices—into one coherent 3D model. The headset projects that model back onto the environment. Once everything is spatially aligned, entirely new interactions emerge.

It’s the beginning of a common interface for mixed human-robot teams.

The Emerging Pattern

As soon as systems begin to understand environments in 3D, whole categories of tasks become dramatically easier: navigation, targeting, inspection, coordination, and multi-agent teamwork.

We’ve been watching the shift as AI models learn to interpret visual data the way humans do—identifying structure, depth, motion, and intent— and here we’re seeing the inverse trend unfold as well. AR systems like this let humans tap into the structured, machine-readable view of the world.

On The Horizon 🔭

This tech is military-only today, but the underlying architecture won’t stay confined to defense. The same building blocks of sensor fusions that create AR-enabled human-robot interfaces are going to spill outward into civilian domains.

Construction crews will navigate sites with live spatial overlays.
Search-and-rescue teams will see through smoke, rubble, and darkness with fused sensor views.
Inspection drones and ground robots will share a common map with operators instead of streaming disconnected feeds.
Cities will gain operational pictures that unify everything from CCTV to infrastructure sensors.

We’re moving toward a world where interacting with software increasingly means interacting with a live model of physical reality—updated continuously, understood collectively, and shared across humans and machines.

As these systems mature, the real world becomes the interface.

If you found this useful, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Cheers,
Bilawal Sidhu

https://bilawal.ai

Subscribe now

The result? A look into what future creative tooling will feel like as models learns to both understand and create 2D, 3D, and interactive formats with an unreal level of spatial understanding.

And this week, as people started putting both models through their paces, we got a surprisingly clear look at what that future might feel like.

MIT cartography goes multimodal

A few days ago, a researcher at the MIT Media Lab took my Nano Banana cartography prompts and pushed them to map “urban informality” — complex settlements that are notoriously difficult to map manually. With Gemini’s spatial reasoning, messy inputs became coherent conceptual maps — reducing a process that could have taken days of manual labor into seconds.

Apparently, the results rival task-specific models for this case. The bitter lesson remains bittersweet.

To get a look at all of the examples and how you can create with these tools today, check out my full video:

A Few Standout Examples

• Create entire games or websites from a single image

• You can create and play AR games with Gemini in minutes

• Build a functional, real-time JARVIS HUD

• He built a Crossy Road clone in 5 minutes

• 3D black hole simulation

• Vibecode a CapCut editor in 4 minutes

  

The Emerging Pattern 🔭

Once models can understand structure this deeply — not just pixels or text, but layout, physics, geometry, and interaction — the barrier between designing software and using it starts to dissolve.

Gemini handles formats we used to treat as separate disciplines: 2D interfaces, 3D scenes, diagrams, scripts, and code. Nano Banana Pro shows the power when you take that knowledge and distill it down to the visual layer. The creative empowerment that comes with access to use these models in tandem is still underutilized, and we’re in for a wild ride as they continue to evolve at a rapid rate.

For most, we are left with one substantive question: What will you make of the power that has suddenly gathered at your fingertips?

In a world where anyone has access to a professional-level software engineer, game developer, technical director, and more, the creations to come are almost incomprehensible.

If you found this useful, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Cheers,
Bilawal Sidhu

https://bilawal.ai

Subscribe now

How do you recreate the whole world in 3D without needing to fly planes everywhere?

This paper didn’t get a ton of attention, but it’s the closest thing I’ve seen to solving a fundamental constraint that’s limited 3D mapping for decades. Researchers have essentially cracked the problem of creating detailed city-scale 3D from satellite imagery alone, no planes, no drones, no street-level photos, just AI filling in what physics makes invisible from space.

Skyfall-GS creates navigable 3D cities from satellite imagery alone

Combining 3d gaussian splatting with the world knowledge of diffusion models.

In the full video breakdown below, you’ll see exactly what makes this a paradigm shift. Their trick is honestly kind of beautiful. They train gaussian splats on satellite views, but as it descends toward ground level, the renders turn to absolute garbage - artifacts everywhere.

Instead of fighting this, Skyfall-GS just treats those nightmare renders as the input to a diffusion model. Basically - “hey FLUX, fix this mess.” It’s like neural scene completion but actually practical, and it unlocks basically the entire world.

The nuances for getting multi-view consistency are a little more complicated than that, so watch the full video to see the full breakdown:

Implications of Skyfall-GS:

• TL;DR on how this paper works and why it matters

• Demo and paper

If you look at the latest computer vision research on matching features between satellite and ground-level imagery, you can imagine what the next step is — go top down AND bottom up. The applications for public and private sectors are obvious. Conflict zones, restricted airspace, aerial-denied regions, suddenly, everything becomes mappable.

Beyond Google Earth and Microsoft Flight Simulator — U.S. Army has a massive one-world terrain program that would benefit tremendously from this approach. All covered in my YouTube video.

Google’s triumphant return as the king of AI 🔭

It is perhaps no understatement to say that this has been Google’s week — with the AI behemoth dropping a one-two punch between their state-of-the-art multimodal reasoning model Gemini 3 Pro, and their state-of-the-art image generation and editing model Nano Banana Pro.

Check out the TL;DR on the key announcements here:

Nano Banana Pro — the new SOTA for image generation and editing.

Meanwhile, Nano Banana has fixed all the issues with the previous release. We get that crispy 2K and 4K resolution. Text generation is a first-class citizen, and you can squeeze up to 14 elements cohesively into one generation while retaining their individual identity and likeness.

In case you missed it, here’s my talk with two of the researchers behind Nano Banana. Including the deep lore of how this iconic name came to be.

But perhaps more critically…

You can finally generate your favorite celebrities — and meme lords are running wild.

Google is taking a page out of the Sora 2 playbook; IP generation is far less restrictive in Nano Banana Pro. It is perhaps the best model to generate a perfect likeness of celebrities and throw yourself into it. Case in point:

Nano Banana Pro excels at diagram generation and text rendering.

On the utilitarian side, the detailed diagram generation ability is absolutely fantastic, and great news for visual learners, presentation makers, and designers alike. Essentially, dump a bunch of context (whether it’s a long research paper or a Gemini 3 Pro response) and ask Nano Banana Pro to summarize it, as I do below. I was really impressed with how intuitive and accurate this is.

Nano Banano Pro is an expert cartographer.

While looping back to the geospatial maps theme, one experiment I tried made me realize that this model is a really good cartographer. And now with the ability to do 2K and 4K output, it’s actually high resolution enough to capture granular map features.

Check out this video experiment I did that went viral on X:

Google DeepMind’s vision for one intelligence that is multimodal in & out.

Google is flexing their full-stack approach to great effect. And after a wobbly start, it’s hard not to be impressed. In particular, the multimodal in-and-out organizing principle of the entire Gemini effort seems to be bearing fruit. Google has unlocked massive headroom on both pre-training and post-training.

While I’m not one to say I told you so, Google’s triumphant return seems obvious in hindsight, doesn’t it?

Thanks to the Google homies for early access, and stay tuned for the next edition, where I’ll be sharing my experiments and curating some of the best work being done with Gemini 3 Pro and Nano Banana Pro.

If you found this useful, share it with fellow reality mappers. The future’s too interesting to navigate alone.

Cheers,
Bilawal Sidhu

https://bilawal.ai

Subscribe now

Right now, that moment is happening between AI models that can create 3D worlds and AI agents that can act inside them.

For years now, we’ve had models that could generate images and agents that could take text-based actions. But now, for the first time, we’re seeing both sides of intelligence — perception and action — emerge inside the same spatial environment.

And the implications stretch far beyond game engines or VFX workflows.

This is the start of AI that understands space.

For a complete breakdown of the models, the workflows, and how they’re beginning to merge, watch my video below:

The Rise of AI World-Builders

A few years ago, turning images into a 3D environment required photogrammetry pipelines, cleanup passes, sculpting, and hours of rendering. Today, tools like World Labs’ Marble can do it from a single image, a handful of photos, or even an old 360 pano.

Marble can fuse perspectives, rebuild spaces, merge worlds, edit layouts, and export splats straight into tools like Octane. What once took days now takes minutes. And once it’s generated, you can edit it with ease: remove a wall, add a hallway, re-skin an entire room. It feels less like 3D modeling and more like world authoring.

At the same time, implicit systems like Genie-3 are using video diffusion to create interactive spaces in real time — no splats, no meshes, just autoregressive video generation. The power of this implicit world understanding has yet to be explored to even a fraction of its potential.

But Worlds Alone Aren’t Enough

A world without an actor is just a set. A world with an actor becomes a simulation. That’s where Google’s new AI agent, SIMA 2, enters: an embodied system that can navigate environments, follow multi-step instructions, reason about objects, play games, and even self-improve. It makes decisions based solely on what a player would see on a computer screen and closes the gap between understanding and action.

The Convergence: A New Digital Medium

The breakthrough isn’t world generation or AI agents — it’s both happening at once. World-builders give AI a body. Agents give AI a mind. Together, they create a sandbox where intelligence can grow.

Imagine Robots learning skills in synthetic simulations.
Games that build themselves while NPCs evolve through experience.
Self-driving cars learning to navigate in a synthetic world with every possible scenario.
Agents simulating rare diseases or edge-case complications to improve clinical preparedness.

This future is already taking shape.

Check Out Recent Discussions on X:

• Google’s SIMA 1 vs SIMA 2

• Why is spatial intelligence so important?

• Motion Stream - Live Video Diffusion Control

• Reskin a 3D environment with World Labs

On The Horizon 🔭

The next decade of physical AI development will be defined by agents living inside persistent worlds, learning, adapting, and interacting. We’re moving from AI as a tool to AI as a participant.

Intelligence is becoming spatial. And the systems emerging today — world-builders, agents, simulators — are early prototypes of a future where digital and physical space blend. Not because the world becomes virtual, but because intelligence finally has room to move.

If you’re building worlds, making films, or thinking about where storytelling goes next, this conversation is for you.

Cheers,
Bilawal Sidhu

https://bilawal.ai

Subscribe now

These movies went beyond innovative VFX, it was advanced R&D disguised as entertainment. A literal 25-year roadmap for the AI and 3D tools we all use today.

The Matrix Built Today’s AI Revolution

In the full interview below, John walks through how bullet time evolved into reality capture, why the Matrix sequels pioneered digital humans in 2003, and how The Matrix Awakens became a playable real-time world in Unreal Engine 5.

The truly wild part? Hollywood killed their next project, an interactive Matrix prequel using AI and real-time rendering, because it was “too risky.” So John built Escape, a platform for creators to bypass gatekeepers entirely.

Watch the full conversation to hear it directly from the Oscar winning VFX architect behind The Matrix Series:

YouTube Chapter Links:

00:00 - Intro: The Matrix’s 25-Year Secret Plan
01:04 - Solving the “God’s Eye” Problem
03:15 - Recruiting PhD Students to Invent Bullet Time
06:11 - How Sci-Fi Becomes a Roadmap for Reality
07:54 - Matrix 2/3: Inventing “Universal Capture”
09:47 - Matrix 4: Stepping Inside the “Real” Matrix
14:40 - Building a City-Scale Simulation in Unreal Engine 5
16:47 - The AI Animatrix Prequel Hollywood Killed
18:55 - How Hollywood Kills Innovation
22:03 - Gen AI: A New “Golden Age for Expression”
26:11 - AI Isn’t Efficiency, It’s “Permission”
28:04 - A New Platform Bypassing Gatekeepers
35:28 - The Future: AI Agents as Your Sixth Sense
41:01 - John’s Warning: The Simulation Is Here

Check out some recent discussion over on social:

• Matrix Awakens project simulating living cities

• The $250K computers that powered The Matrix

• AI-powered bullet time for video

The Warning On The Horizon 🔭

Near the end of our conversation, John said something that stuck with me:

“The Matrix was a cautionary tale. We don’t want to end up being enslaved by these wondrous, remarkable, and occasionally potentially dangerous systems we’re inventing.”

They made a movie about AI simulating reality so convincingly that humans couldn’t tell the difference. Then the industry spent 25 years building the tools that make that simulation possible.

The question isn’t whether we’re building the Matrix. It’s whether we’re building it consciously, and who gets to shape it.

If you’re building worlds, making films, or thinking about where storytelling goes next, this conversation is for you.

Cheers,
Bilawal Sidhu

https://bilawal.ai