Xbox Project Natal - Estimated Specs

This video went up yesterday, was pulled down and then restored later for some reason (not before other youtube users posted their own copies and gathered many more views than the official channel)



The video states 30Hz, and there is an interesting shot starting at two minutes.




From the above screenshots it's possible to count the number of pixels horizontally and vertically, and also most of the depth bins (there may be more range beyond the wall behind the person but a minimum count is better than nothing).

So there at least 18 depth bins, counted at 2:01-2:04 in the video. Each is maybe 2 or3 inches spaced.

The resolution looks like 64x64.

Structured Light For 3d Scanning

I learned a little about structured light from the 2008 House of Cards Radiohead video and more since Kyle McDonald created an open source google code implementation- but only very recently really gotten into how it works.

Simulated Structured Light

I had trouble getting good scans initially and I thought it would be good to create a perfectly controlled setup that didn't require a projector or camera- instead generate scenes from software, projecting textures onto 3d objects.

One of the generated input phase images:


Output from the google code ThreePhase processing app:



The obvious improvement is to add in an obj loader to this program, instead of just generating a semi-random blobby shape.

Real World Scans

I was hoping for a lot more but managed to get a few scans of people during a New Year's Eve party:

From 2009.12.31_nextyear

From 2009.12.31_nextyear

From 2009.12.31_nextyear

Those were all made using modified versions of Kyle McDonald's slDecode and slCapture programs. I'll post those somewhere eventually.


Matlab Script

This script is currently very very slow, but it's good to be able to debug in matlab and have easy access to fft and other functions.

Wrapped phase:


Unwrapped phase:


Note the vertical lines where phase propagates vertically in glitchy ways- some filtering (and even slower processing time) ought to be able to clean that during the flood fill.

Next

I hope to go mainly in the direction of high resolution high fidelity scans, as opposed to high frame-rate low sample-time scans, though I think I can access several high frame rate cameras for use in the latter. There is also lots of room for improvements (especially at the phase unwrapping stage) that would benefit either one of those.

Natal competitor: Optricam?

For a while I was worried that after Microsoft acquired 3DV Systems and their depth sensing ZCam, which might have been available at the beginning of this year, they were going to release the Natal exclusively for the Xbox and the possibilities for non-MS sanctioned applications would be severely hampered. And then to have to suffer delays for software I don't care about when the hardware may be ready now. But there may be similar products from other vendors due to arrive soon.

I learned about the PMDTec Camcube a few months ago. It is a lot less expensive than a Swissranger, but there are no U.S. distributors and nothing on the web from customers that are using it.

But this last week there was this press-release about a Belgian company Optrima (Israel, Germany, Switzerland, and now Belgium- all the U.S. flash lidar vendors seemed to have missed the boat on low cost continuous-wave IR led range finding systems to instead focus on extremely expensive aerospace and high end mobile robotics applications) teaming with TI and a body motion capture software vendor (Softkinetic) to produce Natal/Zcam-like results in the same application area- games and general computer UI. There is mention of Beagleboard support in the press-release, so having say Ubuntu be able to communicate is very likely.

Hopefully TI will really get behind it and all the economies of scale that MS can bring to bear can be matched, and it will only cost around $100.

Also, I'm seeing more and more red clothing and jumpsuits- possibly with specially IR reflective component, which makes me suspicious about the limitations of these sensors (also using them with windows letting in direct sunlight is probably out of the question).

Using a canon camera to view jpegs

I tried using my Canon camera as a picture viewer, putting some downloaded jpegs on it- the pictures didn't show up at all when I tried to review them, only the pictures I had taken with the camera. Renaming the pictures to have camera names like DSC_0025.jpg made the camera show a question mark icon for the picture at least.

A little searching later I discovered a tool called paint.net, which saves jpegs in the proper format so the Canon will like them. The other trick is to re-size the canvas the picture is on so all dimensions are multiples of 8.

paint.net itself is somewhat interesting, quicker to load than Gimp but not so much easier or more intuitive or like Deluxe Paint that I'll use it for anything else.

Instructions for rendering with Processing on Amazon EC2

There are detailed instructions elsewhere on how to get started with EC2 in general, here are the high level things to do for my headless rendering project:

Get a unix command line environment that has python and ssh, I use cygwin under Windows, other times I dual boot into Ubuntu.

Get an Amazon EC2 account, create a ~/username.pem file, and make environmental variables for the keys (follow boto instructions).
Make sure pem permission are set to 700.

Edit ssh_config so that StrictHostChecking is set to no, otherwise ssh sessions started by the scripts will ask if it's okay to connect to every created instance- I could probably automate that response though.

Make sure there are no carriage returns (\r) in the pem file in Linux.

Get Elasticfox, put your credentials in.

Get boto

Get trajectorset

Create a security group called http that at least allows your ip to access a webserver of an ec2 instance that uses it.

At this point it should be possible to run ec2start.py, visit the ip address of the head node and watch the results come in. The ec2start script launches a few instances, one head node that will create noise seeds to send to the worker nodes via sqs, and then wait for the workers to process the seeds and send sqs messages back. The head node then copies the results files and renders the graphics, copying the latest results to folder that can be seen by index.html for web display.

My code is mainly for demonstration, so the key things I did that will help with alternate applications follow:

Custom AMI

You can use the AMI I created with the id 'ami-2bfd1d42', I used one of the Alestic Ubuntu amis and added Java, Xvfb, Boto, and a webserver like lighttpd (I forget if Xvfb was already installed or not).

Headless rendering

The EC2 instance lack graphics contexts at first, and trying to run a graphical application like an exported Processing project will not work (TBD did I ever try that?). Xvfb creates a virtual frame buffer that Processing can render to after running these commands:

Xvfb :2
export DISPLAY=:2


Launching processes and detaching from them

I use python subprocess.Popen frequently to execute commands on the instances like this:

cmd = "Xvfb :2"
whole_cmd = "ssh -i ~/lucasw.pem root@" + dns_name + " \"" + cmd + "\""
proc = subprocess.Popen(whole_cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout,stderr) = proc.communicate()

The problem is when one wants to run something and close the connection, and leave it running - like Xvfb above, it needs to run and stay running. One method is to leave the ssh connection open, but there is a limit of about 20 ssh sessions.

The trick is to use nohup:
cmd = "nohup Xvfb :2"


Don't put extra quotes around the command to execute, which brings me to the next topic.

Quote escaping

There are a few bash commands that require parts to be in quotes- but in python the bash command is already is in quotes, and python will not understand the inner set of quotes unless they are escaped with the backslash:
cmd = "echo \"blah\" > temp.txt";

Then at other times an additional level of quote escaping is required:
cmd = "echo \\\"blah\\\" > temp.txt";
(I do this when I pass all of the cmd variable to be executed by ssh, and ssh wants it in quotes)

One backslash escapes on level of quoting, three escapes two levels? It's because the escaping backslash itself needs to be escaped. This gets confusing fast, and some experimentation with python in interactive mode is required to get it right.

Config file driven

It's not currently, not as much as at it needs to be, which makes it very brittle- to change plots requires making about three different edits, when a source config file should specify it for all.