Remote Reality

Letting the Viewers Control 3D Television

By Doug Peterson

August 2007

Phillips has announced that it will release a 3D television in 2008, which will be able to provide a 3D experience from 2D high-definition disks without the need for special glasses. However, CSL researchers are already hard at work on taking the 3D viewing experience to the next level.

It’s called “remote reality,” says Minh Do, a CSL professor who is collaborating on the project with fellow CSL researcher Doug Jones. By combining images and sounds from multiple cameras and microphones, the system would allow users to control the 3D viewing experience.

For example, Do says, people watching a televised football game would move a mouse, joystick, or other device to get different 3D views of the action on the field -- as if they were really on the sidelines. It’s like virtual reality, except the images are real, not virtual. Hence the name “remote reality.”

Presently, cameramen and television producers choose what views you get. But with remote reality, you make the choice by moving the joystick, just as if you were turning your head or even moving around in the remote reality environment. The sounds you hear will also change as you alter your perspective.

“It’s not a passive viewing experience any more, sitting there and watching what they present to you,” Do says.

The television experience is one of the most obvious applications for remote reality, but it is not the only one, he adds. The system could also be used for video conferences, in which 3D images of co-workers on the other side of the country are transmitted to your conference room; then, once again by manipulating a joystick or other device, you can change your view, as if you were actually present in the same meeting room.

Do is handling the video signal work, while Jones tackles the audio signals. For the video part of the project, Do has developed a testbed in CSL, in which he has aimed three stationary cameras at a small train set. The three cameras capture this dynamic scene from the different viewing angles and then send the data to a computer, which processes the information. Finally, the computer transmits the data across the Internet to another computer, where someone with a mouse is able to change the 3D viewing angles.

CSL researchers have developed the fundamental mathematical theory for handling this high-dimensional data, Do says, but they are trying to reduce the amount of information that needs to be transmitted. They are also determining what the optimal number of cameras needs to be.

“These are specialized, high-end cameras,” Do says, “but we want to eventually show that we can do this with cheaper web cameras.”

The devices to create the system are already available and so is the computing power, Do says. “What’s missing are the algorithms for processing and fusing the data to create novel applications -- new visual and audio experiences.”

CSL researchers are developing these missing algorithms to eventually make remote reality a functional reality.

CSL News

home