Comments on binarymillenium: Photosynth Export Process Tutorial

Hi All, I'm very interested in using PhotoSynth o...

2010-05-29T22:10:53.986-07:00

Hi All,
I'm very interested in using PhotoSynth output to plug into dense point-cloud reconstruction (such as PMVS2), as PhotoSynth seems to get better results than the offline reconstruction code.

SynthExport has just added exporting of the camera parameters to CSV, and I'm trying to write a quick javascript form to generate the next stage of input for PMVS, but I am having problems with the format of the parameters.

The parameters output by Bundler (which is based on the same code as PhotoSynth) are listed in the documentation, and I have applied calculation to convert the X, Y, Z rotation into a matrix, only to find that the rotation values are almost definitely in radians and not degrees.

I need some sanity checking on my conversion, as I'm not fantastic with matrices.

Kind Regards,
Josh

http://photosynth.net/view.aspx?cid=0dcf06d3-8dc7-...

2010-03-26T09:16:40.879-07:00

http://photosynth.net/view.aspx?cid=0dcf06d3-8dc7-4c35-b5e7-ff810bd5489a

Thanks! Have a good weekend.

Tracy, just the URL for now. I won't be able to l...

2010-03-26T09:12:25.807-07:00

Tracy, just the URL for now. I won't be able to look at it until early next week tho. Heading out for the weekend. -Mark

Mark, Do you want the URL address of the synth or ...

2010-03-26T08:47:17.013-07:00

Mark,
Do you want the URL address of the synth or the point cloud we've extracted, which is HUGE.
Thanks,
--Tracy

Tracy, sounds to be like you model's Z axis isn't ...

2010-03-26T07:52:25.983-07:00

Tracy, sounds to be like you model's Z axis isn't aligned with the real world Z axis. Which is basically what Vinuuen is getting at and sounds like what you've been experimenting with.

Can you show us the Synth you're starting with?

-Mark

Thanks...more stuff to ponder and much appreciated...

2010-03-26T07:41:05.327-07:00

Thanks...more stuff to ponder and much appreciated.
--Tracy

Bleh.. err, the other one reads in y,x,z. dern tyo...

2010-03-25T20:21:41.783-07:00

Bleh.. err, the other one reads in y,x,z. dern tyops.

I know from some other software I have used in the...

2010-03-25T20:20:27.475-07:00

I know from some other software I have used in the past for CnC Operations that one program may export a DXF as x,y,z. Then the importing program reads, x,y,z, and another reads y,x,y. So that may be your issue there.

Another thing to look at is to make sure you are only using the first set of points. points_0*.bin and not the points_1*, 2* and so such. Those have different orientations and are unmatched points, so that can also screw things up.

So...another GIS-related question, wondering if an...

2010-03-25T20:07:19.997-07:00

So...another GIS-related question, wondering if anyone else has noticed weirdness of this sort: we got some lovely photos of a valley, bottom to top and terrain on either side and ran them through Photosynth.
Then we exported the point cloud and put it into the demo version of VRMesh, cleaned up the few stray points and exported as a .dxf. We used Quick Import through Data Interoperability to get it into ArcMap. Then we made it into a 2d shapefile, then a 3d shapefile from which we can render TINS.
The weird thing is, when we render TINS, we get different colored bands that represent changes in elevation, which is expected, BUT they're running vertically -- up and down the valley -- rather than horizontally as one would expect (and hope for!) across the valley (I am over-explaining, but if you're looking into the valley, the bands of elevation run north/south if you pretend north is "up" when they should be running east/west..side to side).
Anybody have any thoughts on why this is occuring and what can be done about it? I have some suspicions about the .dxf export process from VRMesh...we get different resulting data in ArcMap, depending on the position the point cloud is in when exported from VRMesh, as in how it's tilted, spun, oriented when we save it as a .dxf. But it seems no matter how we tilt, spin and export, we get elevation bands going the wrong way in our Arc TINS.
Any help or musings is/are appreciated.
--Tracy

So helpful, thanks again, your suggestions and com...

2010-03-18T05:15:15.774-07:00

So helpful, thanks again, your suggestions and comments rock! Lots of different things to try today...

Doing most of the extra programming for Mark, I ca...

2010-03-17T19:18:56.159-07:00

Doing most of the extra programming for Mark, I can say that some of the programs have problems with negative numbers. So the +100 always moves any negatives to positives. Though a +10 should work as well from my experience. The *1000 is so you always get a 3 place precision. Most points in the point cloud wont match a x/y/z of 999/999/999 so you get a larger varied cloud of all positive numbers. Depending on the scale you use of +SS you then get a SS,xxx/SS,yyy/SS,zzz point for every point in your cloud. You always scale first so that you end up with positives. If you multiply first you get negatives unless you use an absurdly large scaler. The scaler is just moving the point cloud's origin to the scaler before multiplying. Once you multiply it you can just get rid of the the numbers after the decimal. And if you need more precision just Change your multiplier to a higher number. The actual picking of +100 and *1000 was just testing to see what worked best.

And no I dont know much about VRMesh or any of the others.

Hope this helps.
- vinuuen, The other Mark

Eh, I forgot this...we're using the demo of VRMesh...

2010-03-17T19:12:38.110-07:00

Eh, I forgot this...we're using the demo of VRMesh to clean the point cloud. My coworker found that the way the points imported into Arc is dependent upon how the point cloud was oriented in VRMesh when exported to a .dxf.
He spun the pt. cloud around and saved several times and we got different results -- once it came into Arc -- every time. He spent a lot of time with the Widget Transformation tool in VRMesh tilting the point cloud with top towards us, away from us, tilted flat away from us, upside down, etc., but we never got an understanding of orientation in VRMesh. It seems friendlier than Meshlab but the "help files" haven't been so helpful. Does anybody here understand how orientation works in VRMesh, as far as saving out the point cloud? Does the Widget actually do anything or is it something just for reference of some kind?
Thank you all again and good night...;)

Mark, THAT's what we've been wracking our poor mus...

2010-03-17T19:01:13.899-07:00

Mark,
THAT's what we've been wracking our poor mushy brains on!!! We knew we needed to scale the xyz's to something more usable but had no idea what kind of multiplier to use. We have z's at 0.81xxxx and 1.25xxxx and that relates to nothing in the real world that we could figure.
We printed out Nathan's step by step guide and were going to try to use Java Graticule 3d (thanks to you too, Art79, we will try that approach as well; better for us to try several avenues and compare the results).
Why'd you pick "+ 100 (* 1000)"?

Thanks again, you guys are the best. Probably saved our Arc software from getting whacked with a baseball bat tomorrow...

Tracy, if you're using VRMesh Studio, you should s...

2010-03-17T08:33:38.666-07:00

Tracy, if you're using VRMesh Studio, you should still save the file as ascii PLY or as a comma delimited file. Open that file in notepad, delete the header info, select all the data (ctrl-a), copy (ctrl-c) and paste (ctrl-v) that into an empty Excel or Access Database. You'll end up with six columns (X,Y,Z,R,G,B). Because ArcGIS has issues with floating points, you'll need to scale the data. What I do is create three new columns (X1,Y1,Y2) where X1=(X+100)*1000, Y1=(Y+100)*1000, and Z1=(Z+100)*1000. By adding 100 to XYZ I'm removing all the negative numbers (This is just my personal preference). By multiplying by 1000 I get rid of the floating point problem. The next step is to save the database. Open ArcMap. In ArcMap go to TOOLS>ADD XY DATA>Pick the database file you just made> Choose your X and Y Field. Leave Coordinate System as Unknown. Click "ok". This will give you a bunch of pointsin the map view. Right click the new file under layers, go to Data and then EXPORT. This will allow you to save the 2D data as shapefile. ArcMAP will ask you if you want to add the newly exported file. Click Yes. Ok, now we have a 2D point shapefile. To create a TIN or similar data you need a 3D shapefile. To do this open 3D Analyst toolbar, click 3D Analyst>Convert>Features to 3D. That will open a dialog box, make sure the 2D shapefile is selected under Input Features, then select the radio button next to "Input feature attribute". In the drop down box next to that pick your Z2 column. This will associate the 2D points with a Z value when you export it. Export a new 3D shapefile.

This 3D shapefile is what you need to generate TIN, Grids, contours, etc with. You do that in 3D or Spatial Analysts.

Easy, huh? =)

Art79 is correct about the georeferencing. Most people use "Georeferencing " under Tools>Customize>Tool Bars. Note that this assumes your synth's ground plane is aligned with the real world ground plane.

I guess I should get around to actually writing a tutorial...

Tracy, There are tools in arcGIS to do georeferenc...

2010-03-16T16:11:51.203-07:00

Tracy,
There are tools in arcGIS to do georeferencing (these are mentioned in Marks second link). These tools only give you a 2d georeference which may be good enough for some applications but photosynth point clouds really need a 3d georeference that will correct for issues such as a tilt in the surface.

For this you are probably better off using Java Graticule 3D that is mentioned in Marks first link (I haven't tried it but it sounds good... or find something similar).

ArcGIS does have the capacity to do a 3d georeference but it seems to only be available through arcobjects code.

Mark, Your response helps a lot, thank you so much...

2010-03-16T10:02:12.459-07:00

Mark,
Your response helps a lot, thank you so much. We have access to the full suite of ArcMap (3d Analyst, Spatial Analyst, etc.) Do you or does anybody else here know how to get the point cloud registered using ArcMap? Or what steps in your process do you think lend themselves to ArcMap?
We have gotten through the steps where we export the ply file from VRMesh as a .dxf, then import it into ArcMap via Data Interoperability/Quick Import. We used a .dxf because we were unable to get an ascii file to work. Either way, when we bring the point file in and create a TIN, our elevations categorized by color do not follow the terrain as we know it to be. It does not "elevate" to north...the color bands runs east-west, if that makes any sense. When we manipulate the orientation of the point cloud in VRMesh, this does give us some favorable results, but not the accuracy we need. Any help is greatly apppreciated. My coworker also adds that he can't believe how much we've learned here in these forums from you guys...way more than any tutorials from software sites.
--Tracy

@Simon Love the video. Great work.

2010-03-10T17:09:17.741-08:00

@Simon Love the video. Great work.

Tracy, both Nathan Craig and I have done this. Na...

2010-03-10T17:02:53.436-08:00

Tracy, both Nathan Craig and I have done this. Nathan has written an excellent page detailing his methods. You can find it here: http://www.personal.psu.edu/nmc15/blogs/anthspace/2010/02/structure-from-motion-point-clouds-to-real-world-coordinates.html

My work flow is a little different and is avilable here:
http://photosynth.net/discussion.aspx?cat=6b63cb81-8b57-4d5d-a978-41d5509bf59a&dis=7b771e05-8a31-4109-8258-e97e4a9f41ae

Hope that helps,

Mark

You guys have done some amazing stuff here. I've l...

2010-03-08T06:57:46.892-08:00

You guys have done some amazing stuff here. I've learned more about point cloud processing and photosynth here than any other place on the net. I have a question...has anybody tried aligning a point cloud to any gps or real world ground control? If so, would you mind sharing how you did it? If not, any thoughts on where to find such info? Thanks.

Just wanted to share a recently completed project ...

2010-03-03T19:44:45.639-08:00

Just wanted to share a recently completed project from a visual studies course I am taking at Columbia University. My partner and I used some of the methodolgies posted here with Rhino and Grasshopper scripting to scan my partners face and create and manipulated a surface modeled from PS point cloud. Here's our final project presentation:

http://www.vimeo.com/9901360

I'll post a full description of the project with a link to imagery in a week or so. Enjoy the video!

binarymillenium, thank you for your quick respons...

2010-02-28T07:26:29.685-08:00

binarymillenium,

thank you for your quick respons!

I checked out the website of VRMesh, Pointools and Geomagic - it says it only runs on xp/vista, but I'm not running windows on my machine as of now - big cleaning job needed for that to work...

I have only downloaded some synths from the photosynth website to experiment with - can't upload my own, being on a mac you know...

Maybe I'm just in over my head here... but it would have been so cool if Meshlab worked ok. Anyone else gotten Meshlab to work without crashes on these photosynth point sets?

Thanx again! :)

Tor, VRmesh looks like the least expensive softwa...

2010-02-27T10:33:25.288-08:00

Tor,

VRmesh looks like the least expensive software that can turn the point clouds into meshes- it took several hours for a few hundred thousand points when I tried it (it was a lidar dataset not photosynth)- also it has a trial edition that will last a few weeks.

There are much more expensive ones like Geomagic and maybe Pointools, I haven't tried them out.

If your points are are in roughly spherical or cylindrical shape there are some more naive but quicker methods I've been using with my own (unpublished) scripts in matlab. If you have a link to your synths I could evaluate it for that approach.

Hi, first I must say that the tutorial and the com...

2010-02-27T07:12:16.545-08:00

Hi,
first I must say that the tutorial and the comments are great! This is amazing :)

But I've got some problems.

The wireshark, 'downthemall' and phyton routine works fine. Then I put the header and footer of the ply file in manually and manage to open the point set in Meshlab, but when I try to make a mesh out of it Meshlab crashes every time.

Any suggestions on how to do it in Meshlab or other software that might be workable? I'm an architect and would like to be able to combine these synth models with some C4D and ArchiCAD models and stuff.

I'm on a Macbook Pro with 2.8 GHz Intel Core 2 Duo and 4 GB of RAM, running OS X 10.6.2.

Thanx!

froland, That triangulation of a synth is really ...

2010-02-09T07:46:08.929-08:00

froland,

That triangulation of a synth is really impressive. The other day I gave VRmesh a try on some laser scan data and it left a lot of holes I don't know how to seal up (I'd know how in Wings 3D but it dies above a few hundred thousands polygons)- the holes were where the laser scan left a lot of shadowed areas, maybe a multi-pov scan would work better.

The synths I've tried to triangulate with meshlab have not turned out well, mainly due to noise- but that face synth is extremely un-noisy so synths of that quality might work.

I'm working on a not-very-robust triangulation method that should work well for producing watertight meshes out of the right kind of point cloud data. If you want to collaborate on something send me an email at the name of this blog at gmail.

What you say about texturing should be possible- though it seems like if you can manually align say a head point cloud with xyz axes, and you have photos of the head along those axes and that are more telephoto (to be more orthogonal), lining up those reference photos should be easy in a 3D program.

A script to automatically align a symmetrical object like a head wouldn't be that hard either, I forget the name of the technique but in Matlab I think it is easy to pass in a bunch of points and find a line or plane that bisects them.

binarymillenium, Thanks for your help. I'll see i...

2010-02-05T19:04:55.197-08:00

binarymillenium,

Thanks for your help. I'll see if I can get bundler running some time soon.

As a test, I had tried the first reconstruction on a good public synth:
http://photosynth.net/view.aspx?cid=19483106-a1af-4e66-9dac-2615ee4427c6

I used VRmesh, which has a comfortable wizard to generate the mesh from a point cloud. You can see the result here:
http://www3.picturepush.com/photo/a/2882841/img/2882841.jpg
http://www4.picturepush.com/photo/a/2882842/img/2882842.jpg

The face worked out quite well, however, ear and ponytail are not as good. Meanwhile, I did some similar synths myself, which actually made nice 3D prints from shapeways.com. Unfortunately, my VRmesh trial is over now and so I'll see what I get with meshlab. Any tutorials on this?

Regarding UV texturing: Both blender and photoshop CS4 extended can paint texture from a photo onto a mesh, like a projection onto the UV map. Here's a tutorial for blender if anybody's interested:
http://www.youtube.com/watch?v=LkzLzRIv3XE
This face was textured that way it seems, but the mesh could be better: http://www.youtube.com/watch?v=bfgvzFzM9hU

I found it quite hard to get the projection angle right when it comes to organic shapes, eg. heads. Thus, it would be good to get the relative coordinates of the original image and 3D mesh. If I get it right they are in the .json file. Ideally, a blender script could texture the mesh using the json and a few selected images. At least that's my rough idea on how it could work.

Any thoughts on that? Is it feasable?

Cheers!