Duke engineers create a new algorithm that fixes shaky video

Duke engineers create a new algorithm that fixes shaky video
August 21, 2017

It’s time for a little TV magic. Emotional close-ups and sweeping shots of landscapes really lose their punch when the footage is even a little blurry. That is why reporters and videographers intensively labor over getting every shot into the best focus possible.

Bad lighting and poor focus can cause blurry footage but the most noticeable cause is when the camera shakes. Movies like the "Blair Witch Project" and "Cloverfield" make use of that technique to add to the suspense and mystery, but a blurry, shaking camera can make an otherwise perfect shot unusable.

Now, a new algorithm developed by Duke University engineers can take the shake out of blurry footage. The math is already being used in video editing programs to salvage footage that would otherwise end up on the editing room floor.

Electrical and computer engineering professor Guillermo Sapiro and former Duke postdoctoral researcher Mauricio Delbracio decided to take a crack at a problem as old as video itself.

Photo and video editing programs like Photoshop, Final Cut and Adobe Premiere have filters and functions that can sharpen an image, but those algorithms can take a long time—sometimes minutes according to Delbracio—to bring even a single frame into focus.

The problem, Delbracio says, was that these programs go through each frame, one-by-one, then identify which way the camera is moving to cause the blur and fix it. At UNC-TV, we shoot video at 30 frames per second, so if it takes a minute to sharpen a single frame, two seconds of sharpening would take a full hour. There are functions that do this more quickly, but the quality is not as good.

The insight the Duke researchers had was that since each shot has dozens of similar images, they can cobble together the most in-focus pieces of every frame to build the sharpest image possible.

Delbracio and Sapiro wrote an algorithm that can quickly calculate which frames are the sharpest and piece together the new images. Their algorithm works in seconds as opposed to minutes to sharpen shaky video. You can see a video of the difference, here.

They published their algorithm in 2016 in the journal IEEE Transactions on Computational Imaging, and it was incorporated into the spring 2017 update of Adobe After Effects, a video effects software that many filmmakers and TV stations (including UNC-TV) use to touch up and add digital effects to footage.

This research shows how science permeates every industry and can enter our lives even through what's displaying on our screens. So the next time you watch a video from UNC-TV with sharply-focused mobile shots or Go-Pro footage, or if you are a videographer yourself and need to clean up some wobbly shots, thank math and computer science: both part of the magic of television!

—Daniel Lane

Daniel Lane covers science, medicine, engineering and the environment in North Carolina.