From 123D Catch
to Old Manhattan

3D reconstruction has interested me ever since Autodesk’s 123D Catch. The idea that you could walk around an object, take a bunch of photographs, and turn those images into something three-dimensional felt pretty remarkable.

That was photogrammetry: finding corresponding features across photographs, estimating where the cameras were, and using those relationships to recover the scene’s geometry.

Microsoft was exploring similar territory with Photosynth, which grew out of the Photo Tourism research project. It could organize collections of photographs in 3D, letting you move between the viewpoints where the pictures were taken. Ordinary photos started to feel like pieces of a larger place.

The point cloud looked better

What kept frustrating me in my own experiments was the transition from a point cloud to a mesh.

A point cloud—a collection of colored points in 3D space—could capture a surprising amount of the character of an object. But when I tried turning it into a continuous surface, something often got lost. Fine details disappeared, edges softened, and the result felt like an ugly approximation of something that had looked much more convincing a moment earlier.

The mesh was usually what I needed, though. It was the representation I could bring into other tools to edit, prepare for 3D printing, or use as a starting point for reverse engineering. Getting something that looked good and getting something useful were often two different challenges.

A different way to represent a place

Then NeRF, or neural radiance fields, brought another approach into focus: learning how a scene looks from different positions so it can render views the camera never captured.

3D Gaussian splatting offered a different representation for that same goal. Instead of requiring a triangle mesh, it represents a scene with many overlapping, soft-edged 3D blobs. Each has a position, shape, orientation, opacity, and color information. Together, they can reproduce remarkably detailed images and render quickly enough to explore interactively.

What appealed to me was that it seemed to preserve more of what I liked about point clouds. The representation could remain a collection of separate elements while looking much more continuous when rendered.

It doesn’t automatically give you a solid object you can print. There’s research into extracting meshes from splats—SuGaR is one example—but producing a visually convincing scene and producing a clean, usable surface are still separate problems.

Trying it on old Manhattan

Recently, I came across SplatAnything, and wanted to try it on something I’d attempted before: colorized archival footage of Manhattan’s waterfront.

It’s a difficult source. The camera is moving, boats are moving, the water is changing, and the footage wasn’t shot with reconstruction in mind. The enhancement and colorization add another layer between the original scene and the images being processed.

Still, the results were better than I’d managed previously.

Adding more footage didn’t necessarily help.

A reconstruction spanning roughly four minutes covered more waterfront, but buildings started overlapping and becoming translucent. Going back to shorter sequences produced much cleaner results. After comparing several windows, I settled on about 80 seconds of footage, using 20 selected frames.

Selected footage

80 seconds

Input frames

20

Inference resolution

1008 × 672

Where the illusion breaks

There’s still plenty to improve. Trees stretch, boats leave ghosts, and parts of buildings float or disappear when you move too far from the original camera path.

Those failures become especially obvious when you try novel view synthesis—rendering the scene from a viewpoint that wasn’t in the input. That’s the central challenge these methods are trying to solve, and this footage gives them plenty of opportunities to struggle.

But even with those limitations, I find the result compelling. A film that used to offer one fixed journey along the waterfront now lets me move a little to either side and look again.

That’s the part of 3D reconstruction that has kept my attention since 123D Catch.

Explore the waterfront ↑