the uncanny valley
Yesterday the New York Times as well as the Wall Street Journal published an article about a new motion capture device called “Contour” which was apparently shown at siggraph yesterday and which was developped by former Apple Computer engineer Steve Perlman.
If an image of an object has a precise contour/silhouette against a different coloured background then the computer can easily trace/recognize this contour. So having a 3D object one can take images of it from various positions and combine the contours of all the images into a 3D mesh by a mathematical process. Every silhouette 3D scanner works in this way. The problem with scanning in e.g. faces is usually that the contours are not very well visible against the various backgrounds.
What I understand from the articles Steve Perlman seems to use fluorescent powder in order to circumvent this problem. In the article it is not described in which wavelength the fluorescent powder is used, so I have to guess a bit of how the system works.
As I understood the visible light is shut off in very very short moments. In particular this means that the face is left in the dark in these moments. However – due to the fluorescent powder- the face itself is glowing out of the dark and hence its contour can be traced. The fluorescent effect is reactivated by using apparently a stroboscopic light which operates probably in the UV region (…and leaves the actors more tanned than before )
Looking at the movie in the new york times this seem to work pretty well.
However concluding – my open questions are: Does the system work in real time?
and
What are the involved wave lengths? The light which is emitted by the fluorescent powder is lower energetic than the light the powder was exposed to (the rest energy is lost to “vibrational or heat” energy) –thats why I assume that the actor is exposed to UV light, but is this right?…or is the fluorescence in the visible/infrared region -which would probably make more expensive cameras necessary? (but they write that they have cheap cameras)
… or in other words: what will Brad Pitt say to this??!?? (as I understood he is supposed to work with the system in an upcoming film)
and
Why is tracing the face by using infrared light -without the ugly powder – not similarily sucessful? (put the cameras behind the face in cooling packs and assume that there are not too many other heat sources in the room :)) ? Tracking with infrared is not something new – see e.g. this randform post.
Remark: the uncanny valley was also the region, which was of most interest to psychologist Prof. Siegfried Frey in his talk on the NMI conference.
I read about it first on grandtextauto blog