About a year ago I wrote about the Photosynth technology from Microsoft that creates 3D spaces from collections of photographs. At the time it only worked with a few sample data sets created by Microsoft, but there was a very nice demo to download and it was a pretty exciting piece of tech. Just a few days ago Microsoft released a full version that allows you to create your own “photosynths” by uploading collections of photos.

I just had to have a go. The image above shows a screen-grab of the Photosynth viewer with some data that I took.

Read on for more on the new release, how I got on with it, videos and links to the 3D scenes that I created.

It may be worth reading my original posting for a description of what Photosynth does, but in a nutshell it attempts to make a 3D spatial model by matching up lots of photos of the same subject taken from different viewpoints. What Microsoft has just released is a new version that can take your own photos, a website to host these, and plugins for Internet Explorer and Firefox to view the scenes in 3D. You get all you need from photosynth.net.

When you get to this site you get three things:

1. A plugin for your web browser just for viewing photosynths.
2. An application to install on your computer that will process and upload your own photos to create a photosynth.
3. A brief “how-to” guide in PDF form that tells you how to go about taking photos to make good photosynths. There is an online video too, but to date I have not watched it.

The first two are wrapped up in the same installation packet. You will need a WindowsLive ID or MSN Messenger account to use the latter, but you don’t need to create an account if you just want to view other people’s work. Also note that the application and, I think, the viewing plugins are Windows only so Mac users are out of luck.

If you have had the earlier demo version of the viewer installed on your machine, then either uninstall or disable the original plugin. In Firefox I had the problem that it failed to recognise I had the viewer installed, and I think it was getting confused with the two plugins there. Once I disabled the old one everything worked fine.

Browse the existing photosynths to get a good feeling for it – I would recommend only choosing those that have been marked at or near “100% synthy”. If not, then things look very boring indeed. The user interface is quite intuitive, and you certainly don’t need to read a manual to use the basic navigation features.

I went out to our local church that this weekend was having its annual Flower Festival. This gave me the opportunity to wander around both inside and outside snapping away lots of shots to use with Photosynth. The following video shows the results when uploaded and processed by Photosynth, and put into “play” mode (which moves the viewpoint around the scene between camera viewpoints).

The scene was reconstructed fom just 35 photos. It was incredibly easy to do – just point the Photosynth application at the images, press “upload”, and then wait. Once complete you are given the option of viewing the result immediately in your browser. Part way through the video I used the “p” key option to toggle the display of the point cloud that Photosythc has calculated that represents the 3D information extracted from the photos. Normally this is obscured as the photos are shown by default, but as you can see it has done a fine job of picking out the shape of the church building. The following two stills show the default scene, and then with point cloud display turned on.

Of course, viewing this as a video does not do Photosynth justice. Once you have the viewer installed, then jump straight to this dataset using this link (note, will open in new window/tab).

And the final bit of niceness is that you can also geo-locate your photosynth from within the viewer, assuming you are logged in as the creator of the scene. This uses Virtual Earth as its back-end, and is very easy to do.

Taking photos to make nice 3D spaces is pretty straghtforward – the short guide on the Photosynth website is worth a read. The main rule is to ensure lots of overlap between images, as this gives the software something to use to help work out areas of similarity. For the church shoots I just wandered around with my digital SLR on medium resolution mode (2.4k x 1.6k pixels) and storing as JPEG. There is really no point in using RAW for this, and picking a lower resolution image saves on upload time. I think even the res I used is higher than is needed.

I also took about 30 interior shots, and I was very pleased with the results. The video below shows it running live. You can also see the actual Photosynth at this link here.

The videos were captured using the excellent free CamStudio application.

Summary

I think Photosynth is quite simply awesome. I highly recommend at least viewing some of the content created using the system. And creating them yourself is so easy you may as well have a go.

Start Slide Show with PicLens Lite

I was fortunate enough to go to Siggraph this year in San Diego, which is the premier conference for new research relating to computer graphics. While the technology I’m going to write about in this article was not presented there this year, it another example of one of an increasing number of research projects to use the vast array of photos available through online communities such as Flickr.

Microsoft have recently released a techology preview of a stunning new piece of technology called “Photosynth”. The aim is to create three dimensional “spaces” of real places and buildings by analysing and processing photos taken from differing viewpoints. The idea of creating spaces is subtly different from creating models. The former places the emphasis on something that is navigable and in which you feel immersed. Model reconstruction, on the other hand, is the creation of a 3D representation that can be places within a virtual environment, perhaps as part of a larger scene.

It is incredibly cool stuff. This article introduces the ideas and shows where to go for more information.

(Update August ’08: See this link for another post on the recently released version that anyone can use to make 3D spaces from their photos.)

The technology to create a 3D model from a set of photos is not all that new. Indeed, I worked on a project with Real3D Inc a few years ago that was all about creating 3D models of people’s faces from two or more photos. But many of these approaches depend on a photographer setting up his shots in a certain way, and explicitely defining the relationships between images. What Photosynth does, however, is use the millions of images available in online photo communities to assemble a 3D space of a real place using a montage of pictures taken by many different people at different times. This montage could consist of thousands of individual source images.

A great way to see this in action is to watch the demo video on the excellent TED site. It is only about seven or eight minutes long. If you are interested in technology and want to feel inspired by some great thinkers, then a browse around TED will be time worth spending.

If you fancy having a go yourself, then you can access an online prototype at the Photosynth research homepage. It is free, and just requires you to download a small plugin that works with both Internet Explorer and Firefox. Here’s a screenshot from my PC:

The scene is of the Piazza San Marco in Venice. The main image is a combination of one or more photos from an online collection placed within a 3D scene represented by a pointcloud that Photosynth generated from processing the photos. The strip along the bottom shows the primary contributing image to the scene, and also related images that are spatially close. Remember that these pictures could have been taken by hundreds of different people, and many are just holiday snaps. You can navigate around the Piazza in 3D using the controls on the image or the mouse.

At the moment the creation of spaces can only be done by the authors using their own specially processed database of images derived from a public photo community. But imagine the possibilities if you combined this with the Panoramio geotagged image database? Of course there is a small issue of corporate politics that would prevent this, unless Google and Microsoft merge. Yeah, right….

Expect to see more and more applications that exploit vast online image libaries for various uses. In upcoming articles I will describe a couple of recent research projects presented at Siggraph.

Start Slide Show with PicLens Lite

The use of 2D views to aid interaction and visualisation within 3D Collaborative Virtual Environments

When I started it I was a partner in a new tech startup called RealiMation Ltd, and doing a research degree into some aspect of collaborative virtual reality was a natural extension to my work. But shortly after I started, RealiMation was acquired by Criterion Software, and a few years later by Electronic Arts, which in combination provided a radical – yet enjoyable – change in life. But I persevered, and finally manage to submit my thesis seven years after I started. Apparently this is not the record, however, which was somewhat disappointing!

At the time of writing I have no idea if I have passed, but that’s almost beside the point. I could so easily have given up, but stuck at it and produced a decent piece of work in the end.

Just for interest, here’s the abstract:

The goal of this thesis is to present research into technology to support the production of events created within three dimensional computer generated collaborative virtual environments (CVE’s). The demands of the roles of different participants, and the tasks they perform, are examined in a variety of different application areas including command and control, multiplayer networked games, and the broadcast of televised events performed within a virtual space.

Published research into subjectivity tends to concentrate on subjective filtering mechanisms for 3D perspective views, normally from the first person view point. The focus of the research presented here, however, is the feasibility of utilizing parallel map-centric projections within a CVE application as a type of subjective view to be used alongside the perspective view.

The experimental phase of the research resulted in the conclusion that some type of overview visualisation is needed to help users understand what is going on within a CVE. An additional finding was the need to be able to add extra annotation information only within the overview, which in turn implies subjective layering features so that such annotation only appears in views used by the management roles, and not by the regular participants themselves.

The conclusions are combined with a set of observations from other related application areas to present a set of requirements for CVE software systems. The requirements relate to the awareness of activity within the performance, and of interactive navigation tools to allow this information to be exploited. Details are given of an implementation of these requirements for the MASSIVE-3 collaborative virtual environment. Key features of the implementation such as the object model are discussed as having potential to be extracted and used in other CVE frameworks. Areas for future research based upon the experience of this software are outlined.

The body of the thesis is now in review and pending a vivre – unless of course it is rejected out of hand as a pile of old junk.

One interesting thing to point out is that the practical part of the research work that I did in support of a live event called Avatar Farm was referred to and discussed in at least two publications. One was an article in the Communications of the ACM (July 2001), and another was a paper as part of an Esprit project.

Update: November 2007 – I got news that I have been awarded my Masters (an MPhil) based on this thesis. So not a pile of junk after all then!