Our capstone project is officially DONE. We submitted our paper around 6:15 PM today. It is posted on our website at cs.plu.edu/~scanners.
It has been a great ride...sure there were some stressful moments and some struggles, but in the end, I am very proud of the project we were able to put together and of the results that we obtained.
I hope readers of this blog have gained a better understanding of 3D scanning and has had gained a deeper interest in the world of 3D graphics.
Thanks for reading.
Jeff
Wednesday, May 15, 2013
Thursday, May 9, 2013
Week of 5/6
Well, Academic Festival is behind us...and boy was our presentation a success.
We filled the room with quite a few people and gave an hour-long presentation, including questions. We got quite a few questions, which I am not too surprised with given the nature of our topic. Overall I've heard positive things about the presentation. I am proud of how we did and in our project and results overall.
We are now working on our final document which is due next Wednesday. I will likely have one more blog post following that.
Check out our updated website with the presentation slides and video at www.cs.plu.edu/~scanners.
Until next week!
We filled the room with quite a few people and gave an hour-long presentation, including questions. We got quite a few questions, which I am not too surprised with given the nature of our topic. Overall I've heard positive things about the presentation. I am proud of how we did and in our project and results overall.
We are now working on our final document which is due next Wednesday. I will likely have one more blog post following that.
Check out our updated website with the presentation slides and video at www.cs.plu.edu/~scanners.
Until next week!
Thursday, May 2, 2013
Week of 4/29
Academic Festival is just about here!
We have prepared our slides, practiced, and run some good scans. We present Saturday at 10:45 AM.
Since my last post, we did have some problems...It turns out that OpenCV looks for a certain color pattern on the chessboard to know where to start the calibration. Without being consistent in intrinsic and extrinsic calibrations, there will be problems. Anyways, I finally figured out how we needed to do it and drew arrows on the chessboard so we know the correct way to do the calibrations according to our code.
Also, Grady implemented a gear box out of Lego Mindstorms. That combined with the Servo motor provides a good, slow motion of the laser, and we have gotten pretty good results.
Hopefully our presentation will go well! Come out and see it!
We have prepared our slides, practiced, and run some good scans. We present Saturday at 10:45 AM.
Since my last post, we did have some problems...It turns out that OpenCV looks for a certain color pattern on the chessboard to know where to start the calibration. Without being consistent in intrinsic and extrinsic calibrations, there will be problems. Anyways, I finally figured out how we needed to do it and drew arrows on the chessboard so we know the correct way to do the calibrations according to our code.
Also, Grady implemented a gear box out of Lego Mindstorms. That combined with the Servo motor provides a good, slow motion of the laser, and we have gotten pretty good results.
Hopefully our presentation will go well! Come out and see it!
Thursday, April 25, 2013
Week of 4/22
We are now just 9 days away from presenting!
Since my last post, we have mostly been just putting some finishing touches on the code. I tried to get some camera settings in the code but ended up reverting most of them. I was able to figure out how to turn off the auto-focus (sort of--by setting it to the current value), but the user will need to set things like exposure and gain for their own camera.
I refactored the write-to-file method; we now have messages showing where the process of the scan is at (e.g. Finding Red Points). I also added another check to eliminate more noise; anything that is greater than -0.1 in the z-component (since the z-axis of the plane points away from the camera) is not included. This means points on the plane or behind it are not included.
Tyler made a few more GUI changes, and the hardware code is now integrated with the main code. Thus our coding is complete! (Other than the C code used to be put on the dev board, which Grady is/will be tweaking). We have a new Servo motor on its way that will hopefully allow us to get small, smooth increments with the pulsewidth modulation.
On Wednesday, I brought in various items to scan and I was pleased overall with most of the results!
We have started getting our slides ready and spent some time figuring out how our slideshow will be ordered, as well as time estimates for the various segments of the presentation.
In the next week, we will continue working on our presentation, setup the scanning environment, and do the final scans of objects to show as results, as well as one to use as our demo (likely of a wooden kangaroo). This will be done using our actual stage, higher quality camera, and the motor (along with the laser powered through the board). We plan on filming the motor/laser as well to use in conjunction with the screen capture.
We are closing in! Hope everyone will enjoy the project and results!
Since my last post, we have mostly been just putting some finishing touches on the code. I tried to get some camera settings in the code but ended up reverting most of them. I was able to figure out how to turn off the auto-focus (sort of--by setting it to the current value), but the user will need to set things like exposure and gain for their own camera.
I refactored the write-to-file method; we now have messages showing where the process of the scan is at (e.g. Finding Red Points). I also added another check to eliminate more noise; anything that is greater than -0.1 in the z-component (since the z-axis of the plane points away from the camera) is not included. This means points on the plane or behind it are not included.
Tyler made a few more GUI changes, and the hardware code is now integrated with the main code. Thus our coding is complete! (Other than the C code used to be put on the dev board, which Grady is/will be tweaking). We have a new Servo motor on its way that will hopefully allow us to get small, smooth increments with the pulsewidth modulation.
On Wednesday, I brought in various items to scan and I was pleased overall with most of the results!
We have started getting our slides ready and spent some time figuring out how our slideshow will be ordered, as well as time estimates for the various segments of the presentation.
In the next week, we will continue working on our presentation, setup the scanning environment, and do the final scans of objects to show as results, as well as one to use as our demo (likely of a wooden kangaroo). This will be done using our actual stage, higher quality camera, and the motor (along with the laser powered through the board). We plan on filming the motor/laser as well to use in conjunction with the screen capture.
We are closing in! Hope everyone will enjoy the project and results!
Thursday, April 18, 2013
Week of 4/15
We are now just 16 days from presenting our capstone!
Last Saturday, we met and made significant headway (after helping Dr. Wolff with passport weekend). We added additional region clicks to limit the data to the region the object is located in, added a filename field for output, I updated (most of) the floats to doubles, and we finished up the destructors.
This week, we made more good progress. Grady and I met with Dr. Wolff on Tuesday to discuss what we have been working on and what we planned to do this week.
Tuesday, Grady cleaned up some Git problems we were having due to line endings. He has also been working on integrating a QThread into the serial code. This will allow the motor to spin in its own thread while the video is being captured in a separate thread. He had numerous problems getting it to work, but Thursday night, I'm pleased to say he did get it working! So we now have another very key piece of the hardware integration ready. We will wait to merge it with the main code so that I can continue testing the scan.
Tyler meanwhile has been working on improving the GUI. He has made it look quite a bit better (rather than the classic Windows look). The color scheme and buttons are better and the error message functionality is cleaner and informative. There are still a few things to fix (e.g. some directory fields are allowing focus, clearing error messages), but overall, the look of our application is coming along nicely.
I added bounds checking to eliminate points that lie on either plane (within z= +/- 0.1) as well as any data points that are outside the sphere with a radius equal to the distance from the camera to the back plane origin. With the object regions and these restrictions, we improve the process time a bit. Also, I did some code cleanup (such as eliminating repeated identical calculations and changing AB-AC to A(B-C) where A,B,&C are matrices). Also, we now are clearing data as soon as possible from the scanModel rather than calling a resetScan method. I started working on some camera settings as well; I have implemented getting the current settings with the lights on, doing a scan with the lights off with custom settings, and then resetting the camera settings to the pre-scan settings. I tried using a larger version of the image (that shows more), but it slowed down processing so I will likely leave it at 640x480 (though I may mess with the FPS settings and try combining that with a larger image). Hopefully I will be able to find a flag to turn off autofocus of Grady's camera, but we will see...
Last but not least, I did a scan of Dr. Wolff's orange miniature brain, and was quite pleased with how the reconstructed mesh turned out:
Until next time :)
Last Saturday, we met and made significant headway (after helping Dr. Wolff with passport weekend). We added additional region clicks to limit the data to the region the object is located in, added a filename field for output, I updated (most of) the floats to doubles, and we finished up the destructors.
This week, we made more good progress. Grady and I met with Dr. Wolff on Tuesday to discuss what we have been working on and what we planned to do this week.
Tuesday, Grady cleaned up some Git problems we were having due to line endings. He has also been working on integrating a QThread into the serial code. This will allow the motor to spin in its own thread while the video is being captured in a separate thread. He had numerous problems getting it to work, but Thursday night, I'm pleased to say he did get it working! So we now have another very key piece of the hardware integration ready. We will wait to merge it with the main code so that I can continue testing the scan.
Tyler meanwhile has been working on improving the GUI. He has made it look quite a bit better (rather than the classic Windows look). The color scheme and buttons are better and the error message functionality is cleaner and informative. There are still a few things to fix (e.g. some directory fields are allowing focus, clearing error messages), but overall, the look of our application is coming along nicely.
I added bounds checking to eliminate points that lie on either plane (within z= +/- 0.1) as well as any data points that are outside the sphere with a radius equal to the distance from the camera to the back plane origin. With the object regions and these restrictions, we improve the process time a bit. Also, I did some code cleanup (such as eliminating repeated identical calculations and changing AB-AC to A(B-C) where A,B,&C are matrices). Also, we now are clearing data as soon as possible from the scanModel rather than calling a resetScan method. I started working on some camera settings as well; I have implemented getting the current settings with the lights on, doing a scan with the lights off with custom settings, and then resetting the camera settings to the pre-scan settings. I tried using a larger version of the image (that shows more), but it slowed down processing so I will likely leave it at 640x480 (though I may mess with the FPS settings and try combining that with a larger image). Hopefully I will be able to find a flag to turn off autofocus of Grady's camera, but we will see...
Last but not least, I did a scan of Dr. Wolff's orange miniature brain, and was quite pleased with how the reconstructed mesh turned out:
 |
| A photo of the brain |
 |
| A reconstructed model of the brain |
We will likely use something with more profound and pronounced features (that is a bit larger).
Until next time :)
Thursday, April 11, 2013
Week of 4/8
Since my last post, we haven't done a bunch of coding, but have done some.
This week was consumed mostly with presentations: going to class on Tuesday and meeting with Dr. Wolff, preparing for the presentation on Wednesday, and then presenting and seeing other presentations on Thursday. For our presentation, we were able to have a demo of the scan as well as displaying it in MeshLab.
As far as coding goes, Grady has been working on figuring out threading so we can check for the serial flag to stop the motor while also collecting the images from the camera. We figured out today we can simply use the timeout of the timer within the scanningView rather than a while loop in the controller. When the timer times out, we will check that the motor is not yet finished rotating. If it isn't, we will send the image through the controller to the model. If it is finished, we will stop the timer and begin processing the scan data.
Also, I have improved the progress bar so that it obtains values from the model of how many loops will be used; that is, the number of rows to process and two times the number of images, plus 1 for writing to the file. This may get updated with horizontal bounding (see below).
Tyler has worked on some memory management, and I rewrote the createPointCloud method (taken from OpenCV's) so that we can use double points rather than floats. I also started working on getting the filename to save the points rather than hard-coding; there will be an additional field for the filename to append to the directory. (Note: this isn't an ideal way of saving a filename, but due to time constraints, it is probably the easiest with what we have). I plan on changing as much as possible to use double data rather than float so that we can maintain as much accuracy as possible.
In the coming weeks left for implementation, we plan on doing the following:
This week was consumed mostly with presentations: going to class on Tuesday and meeting with Dr. Wolff, preparing for the presentation on Wednesday, and then presenting and seeing other presentations on Thursday. For our presentation, we were able to have a demo of the scan as well as displaying it in MeshLab.
As far as coding goes, Grady has been working on figuring out threading so we can check for the serial flag to stop the motor while also collecting the images from the camera. We figured out today we can simply use the timeout of the timer within the scanningView rather than a while loop in the controller. When the timer times out, we will check that the motor is not yet finished rotating. If it isn't, we will send the image through the controller to the model. If it is finished, we will stop the timer and begin processing the scan data.
Also, I have improved the progress bar so that it obtains values from the model of how many loops will be used; that is, the number of rows to process and two times the number of images, plus 1 for writing to the file. This may get updated with horizontal bounding (see below).
Tyler has worked on some memory management, and I rewrote the createPointCloud method (taken from OpenCV's) so that we can use double points rather than floats. I also started working on getting the filename to save the points rather than hard-coding; there will be an additional field for the filename to append to the directory. (Note: this isn't an ideal way of saving a filename, but due to time constraints, it is probably the easiest with what we have). I plan on changing as much as possible to use double data rather than float so that we can maintain as much accuracy as possible.
In the coming weeks left for implementation, we plan on doing the following:
- Integrating Hardware: Get the motor worked into the project
- Complete Memory Deallocation and Management
- Finish up destructors
- Clearing data as soon as possible during/after sacn
- GUI Improvements
- Qt Stylesheets (if time)
- Threading for progress bar (maybe)
- Implementing additional region clicks: Horizontal bounding on object to reduce error
- Use double calculations whenever possible
- Attempt to get better and more consistent scan results
- Bounds checking for object points
- Don't include plane points
- Ignore extreme outliers
- Set up final scanning environment
- Other (e.g. output file, cleanup/refactoring)
- Mesh work (merge if time allows)
We only have a couple more weeks before we need to prepare for our presentation. Hopefully we can successfully implement most of the above and have a more solid product for academic festival.
Until next time.
Thursday, April 4, 2013
Week of 4/1
Phew! Debugging the scan took a while, but I FINALLY figured out the logic error.
I was able to narrow down what appeared to be an error in our calculations of the points on the back plane. It turns out I was write, and indeed it was an error in our lambda calculation.
The lambda calculation is in the following format:
lambda = (n^t(point_on_plane - camera_origin))/(n^t(image_point))
We can use the plane origin as the point in the plane and (0,0,1) as the plane normal vector. We know the camera origin and calculated the image point.
What we were doing is converting the world pieces to camera coordinates. This is a PROBLEM, because we are converting a normal vector while treating it as a point, using RP+T to convert it to world coordinates. Wednesday, I had the idea of converting things to world coordinates instead, allowing me to use the normal vector. I converted both the image point and the camera origin to world coordinates. I thought I had a breakthrough, obtaining a z-coordinate of zero. However, the numbers didn't look quite right. I used my graphing calculator to do matrix multiplications for checking things. I finally left around 9:45 Wednesday night.
Thursday I was hoping to get it! I felt I may be on the right track (after looking at my calculator some more at home Wednesday night). But I wasn't getting very far. Then I started looking at the lambda calculation and associated geometrical pieces. I realized something and started working it out on the whiteboard and was gratified by my calculator producing the correct answer. Here is what it was: lambda*u is a ray, NOT a 3D point. The 3D point is origin + lambda*u; this must be converted to world all at once. This comes out as the following: R^t*origin + lambda*R^t*u + R^t*T. I then applied the same ray-plane equation and resolved for lambda. I went through the calculations, and when I got lambda, I converted back to camera, divided the x and y by z, and verified that they matched the image point coordinates. BINGO! I made changes to the code as needed, rescanned, and I got not only perpendicularity of the planes, but also got 3D curvature of a scanned coffee cup (there is some noise, but that's likely expected). It's about time :)
Other happenings: Grady merged his branch from serial and all our code is back in one repo. Tyler implemented the progress bar (it shows up but has some bugs to work out) and destructors (these are not all correctly working yet...some of them are commented out until Tyler can look at them more).
We plan to meet Saturday. Hopefully we can get everything working on all three computers (e.g. updated serial lib and files, includes in VS) and continue making headway. Grady is/will be looking at pulsewidth modulation; hopefully he can do it in time and we can have a smoother motion with our motor. Tyler will work on the progress bar and destructors more. And I will continue working on the scan, likely working to disclude the plane points from the scan (things that have z~0) leaving the object as the primary piece.
In all, Thursday was a good day. We have a presentation next Thursday that will take some time to prepare for, but I'm feeling better about the project overall now that the scanning bug is fixed. Hopefully we'll even reduce the noise in the scan if possible.
Until next time...
UPDATE (4/5/13):
Here is what a scan of the coffee cup on its side now looks like:
I was able to narrow down what appeared to be an error in our calculations of the points on the back plane. It turns out I was write, and indeed it was an error in our lambda calculation.
The lambda calculation is in the following format:
lambda = (n^t(point_on_plane - camera_origin))/(n^t(image_point))
We can use the plane origin as the point in the plane and (0,0,1) as the plane normal vector. We know the camera origin and calculated the image point.
What we were doing is converting the world pieces to camera coordinates. This is a PROBLEM, because we are converting a normal vector while treating it as a point, using RP+T to convert it to world coordinates. Wednesday, I had the idea of converting things to world coordinates instead, allowing me to use the normal vector. I converted both the image point and the camera origin to world coordinates. I thought I had a breakthrough, obtaining a z-coordinate of zero. However, the numbers didn't look quite right. I used my graphing calculator to do matrix multiplications for checking things. I finally left around 9:45 Wednesday night.
Thursday I was hoping to get it! I felt I may be on the right track (after looking at my calculator some more at home Wednesday night). But I wasn't getting very far. Then I started looking at the lambda calculation and associated geometrical pieces. I realized something and started working it out on the whiteboard and was gratified by my calculator producing the correct answer. Here is what it was: lambda*u is a ray, NOT a 3D point. The 3D point is origin + lambda*u; this must be converted to world all at once. This comes out as the following: R^t*origin + lambda*R^t*u + R^t*T. I then applied the same ray-plane equation and resolved for lambda. I went through the calculations, and when I got lambda, I converted back to camera, divided the x and y by z, and verified that they matched the image point coordinates. BINGO! I made changes to the code as needed, rescanned, and I got not only perpendicularity of the planes, but also got 3D curvature of a scanned coffee cup (there is some noise, but that's likely expected). It's about time :)
Other happenings: Grady merged his branch from serial and all our code is back in one repo. Tyler implemented the progress bar (it shows up but has some bugs to work out) and destructors (these are not all correctly working yet...some of them are commented out until Tyler can look at them more).
We plan to meet Saturday. Hopefully we can get everything working on all three computers (e.g. updated serial lib and files, includes in VS) and continue making headway. Grady is/will be looking at pulsewidth modulation; hopefully he can do it in time and we can have a smoother motion with our motor. Tyler will work on the progress bar and destructors more. And I will continue working on the scan, likely working to disclude the plane points from the scan (things that have z~0) leaving the object as the primary piece.
In all, Thursday was a good day. We have a presentation next Thursday that will take some time to prepare for, but I'm feeling better about the project overall now that the scanning bug is fixed. Hopefully we'll even reduce the noise in the scan if possible.
Until next time...
UPDATE (4/5/13):
Here is what a scan of the coffee cup on its side now looks like:
Thursday, March 28, 2013
Week of 3/25
Spring break this week, but that didn't stop us from working :)
Grady was having some problems integrating the serial code into the main application. He debugged for hours and couldn't figure it out. Tyler and I met on Thursday, and he basically spent the whole time trying to figure out a linker error. We finally were able to get it working. What it was was a LPCTSTR error. The serial code, which is being used as a lib, uses a LPCTSTR type in some of its methods. However, when it gets compiled, it is converted to an unsigned (const?) char *. So when using the lib, the main app only references the header file, which contains the LPCTSTR type. When trying to use one of the methods with that type as a parameter, it causes a linker error for an unresolved external for the method signature. To fix this, we changed the LPCTSTR to the unsigned char in the serial code's header and cpp file and instead of using a __t() macro, we pass in a string directly when using the code in the main app. This fixed it (finally).
On another note, I continued trying to debug the scanning. I did fix a bug for when no red was found in an image. But I still haven't figured out what's causing the non-perpendicular traits of the planes. I have been converting to world coords and comparing to the origin of the back plane (which is the translation vector). It seemed that the camera-coord z-coord may have been off by 1 unit in one of the executions. I'll look into that more next week. Hopefully I can find something quickly!
Until next week!
Grady was having some problems integrating the serial code into the main application. He debugged for hours and couldn't figure it out. Tyler and I met on Thursday, and he basically spent the whole time trying to figure out a linker error. We finally were able to get it working. What it was was a LPCTSTR error. The serial code, which is being used as a lib, uses a LPCTSTR type in some of its methods. However, when it gets compiled, it is converted to an unsigned (const?) char *. So when using the lib, the main app only references the header file, which contains the LPCTSTR type. When trying to use one of the methods with that type as a parameter, it causes a linker error for an unresolved external for the method signature. To fix this, we changed the LPCTSTR to the unsigned char in the serial code's header and cpp file and instead of using a __t() macro, we pass in a string directly when using the code in the main app. This fixed it (finally).
On another note, I continued trying to debug the scanning. I did fix a bug for when no red was found in an image. But I still haven't figured out what's causing the non-perpendicular traits of the planes. I have been converting to world coords and comparing to the origin of the back plane (which is the translation vector). It seemed that the camera-coord z-coord may have been off by 1 unit in one of the executions. I'll look into that more next week. Hopefully I can find something quickly!
Until next week!
Friday, March 22, 2013
Week of 3/18
Scanning has begun, but there is still plenty of work to be done.
Since my last post, I was able to refactor a lot of the main scanning code to break up where the for loops are. This provides for cleaner code. I also set up the overlay view to be able to test the scan. When running through the scan (with just video of myself -- no laser), I got some memory exceptions. It turns out you cannot multiply a float matrix (32F) by a double matrix (64F), so I fixed the errors and was able to run through a scan with no errors.
On Tuesday, Tyler and I came up with a makeshift scanning stage idea. I put it together and attempted a scan...the results were definitely not good. Wednesday, I came back and started working through it and determined there was some logic errors in our red midpoint calculation. One error was when checking for a zero crossing, I had a check if it was equal to zero...well if it is black, the midpoint is zero (max and min red components are zero). So I took out this piece (at least for now). Also, our idea of calculating the midpoint at each pixel was causing some problems. If a black pixel had a min red point of 0 and a max of 1 for a midpoint of 0.5, it was finding a zero crossing between a -0.5 and a 0.5 value. This means that if we find the midpoint at each pixel, every pixel must be hit by the laser (as the Brown/Siggraph notes mention for the shadow). Instead, I changed it to find the midpoint across a row of pixels. When the lights are off, this should work fine. For the Siggraph structured light shadow, it probably wouldn't as the object itself causes a shadow. But when red is the only visible piece, it is okay if not every pixel is hit by the laser.
When I made that change, this is what resulted as the scan of a coffee cup, shown in a program called MeshLab:
Since my last post, I was able to refactor a lot of the main scanning code to break up where the for loops are. This provides for cleaner code. I also set up the overlay view to be able to test the scan. When running through the scan (with just video of myself -- no laser), I got some memory exceptions. It turns out you cannot multiply a float matrix (32F) by a double matrix (64F), so I fixed the errors and was able to run through a scan with no errors.
On Tuesday, Tyler and I came up with a makeshift scanning stage idea. I put it together and attempted a scan...the results were definitely not good. Wednesday, I came back and started working through it and determined there was some logic errors in our red midpoint calculation. One error was when checking for a zero crossing, I had a check if it was equal to zero...well if it is black, the midpoint is zero (max and min red components are zero). So I took out this piece (at least for now). Also, our idea of calculating the midpoint at each pixel was causing some problems. If a black pixel had a min red point of 0 and a max of 1 for a midpoint of 0.5, it was finding a zero crossing between a -0.5 and a 0.5 value. This means that if we find the midpoint at each pixel, every pixel must be hit by the laser (as the Brown/Siggraph notes mention for the shadow). Instead, I changed it to find the midpoint across a row of pixels. When the lights are off, this should work fine. For the Siggraph structured light shadow, it probably wouldn't as the object itself causes a shadow. But when red is the only visible piece, it is okay if not every pixel is hit by the laser.
When I made that change, this is what resulted as the scan of a coffee cup, shown in a program called MeshLab:
This is actually an inverted view. Since we are converting to the back world coordinates, this is a view from behind. So if we want to plot the view from the front, we will probably want to rotate around the Y-axis by 180 degrees.
The next problem is the depth. This scan isn't really that great depth-wise. It is more of a 2D image with a coffee cup sticking out (in from this view). The stage pieces are not perpendicular but more of 130 degree angles. I spent 5+ hours thinking/debugging this, including scanning just a our stage. I thought it might be how we are obtaining the extrinsic values...but I'm not convinced of this. I looked at the Siggraph notes, but I'm still not sure what the problem is. Does it really matter where the checkerboard is on the stage when you do the extrinsic calculation? The lambda value is n^t(plane_point-camera_origin)/(n^t(camera_point))...so do our assumptions of (0,0,0) as the plane origin and (0,1,0) [or (0,0,1)] as the plane normal work? I want to talk to Dr. Wolff about this problem as so far I haven't been able to find a problem.
Meanwhile, Tyler has worked on fixing some things in the overlayview and scanning and preparing for the progress bar. He changed the overlayview to use an alpha channel.
Grady got the interrupt to work correctly and has a serial library written. He is working on integrating it into the app. He also worked with a DC motor instead...but we might just stick with the stepper motor if we can get a smoother motion.
During spring break, we may meet once or twice. We need to get our scanning stage built. After spring break, crunch time gets closer. We need to figure out the progress bar and figure out the scan problems and getting the stepper motor integrated. The scan problems are what concerns me most as it is very difficult to debug logic errors with it. But hopefully I can find what is wrong and/or get some advice/help from Dr. Wolff.
Spring 'break' is here.
Thursday, March 14, 2013
Week of 3/11
A week closer to a working 3D Scanner.
This week, as a team we didn't do much since we had class on Tuesday (and then a meeting with Dr. Wolff), and on Wednesday we prepared for our presentation on Thursday. We presented about our progress on the project and covered the GUI, calibration, and the hardware. I showed a remapping of found points on an image, and Tyler was able to implement a rough version of the region-clicking GUI.
However, since my last post, I was able to implement the red-component finding, laser-plane finding, and laser-object intersection finding. In the red-component, I determined that we can loop through each row in the object region as well (just like the back and ground planes) and do the same interpolation to determine the x-subpixel location. Based on where in the imageNum loop, we know which frame that particular subpixel value is found and can associate it with the laser plane at the same image. The interpolation used is a basic linear interpolation: x = x_0 + (0-delta(x_0))/(delta(x_1)-(delta(x_0)).
In the laser plane detection, I basically implemented the algorithm described in our design doc and shown in Taubin's notes (such as the approximate intersection). I also used cv functions such as undistort and the homogeneous conversion function.
The laser-object intersection was also basically just an implementation of the algorithm stated in the design document.
These things have yet to be tested, so hopefully in the next week or so, we can get to the point of performing a basic scan of some sort to: test the red-component and then find the plane. The object-intersection testing will likely come a bit later.
Tyler will work on updating the GUI more and Grady will work on figuring out why the interrupt is going twice and then branch off the repo to begin work on integrating it into the application.
I did some research on polygon rebuilding. We will likely want to try and output our data as a VRML file to use in another program (such as an old version of Blender {maybe} or Taubin's Java program) if we don't use OpenGL to build a polygonal mesh (which I think could be difficult and unattainable with our schedule).
Until next week...
This week, as a team we didn't do much since we had class on Tuesday (and then a meeting with Dr. Wolff), and on Wednesday we prepared for our presentation on Thursday. We presented about our progress on the project and covered the GUI, calibration, and the hardware. I showed a remapping of found points on an image, and Tyler was able to implement a rough version of the region-clicking GUI.
However, since my last post, I was able to implement the red-component finding, laser-plane finding, and laser-object intersection finding. In the red-component, I determined that we can loop through each row in the object region as well (just like the back and ground planes) and do the same interpolation to determine the x-subpixel location. Based on where in the imageNum loop, we know which frame that particular subpixel value is found and can associate it with the laser plane at the same image. The interpolation used is a basic linear interpolation: x = x_0 + (0-delta(x_0))/(delta(x_1)-(delta(x_0)).
In the laser plane detection, I basically implemented the algorithm described in our design doc and shown in Taubin's notes (such as the approximate intersection). I also used cv functions such as undistort and the homogeneous conversion function.
The laser-object intersection was also basically just an implementation of the algorithm stated in the design document.
These things have yet to be tested, so hopefully in the next week or so, we can get to the point of performing a basic scan of some sort to: test the red-component and then find the plane. The object-intersection testing will likely come a bit later.
Tyler will work on updating the GUI more and Grady will work on figuring out why the interrupt is going twice and then branch off the repo to begin work on integrating it into the application.
I did some research on polygon rebuilding. We will likely want to try and output our data as a VRML file to use in another program (such as an old version of Blender {maybe} or Taubin's Java program) if we don't use OpenGL to build a polygonal mesh (which I think could be difficult and unattainable with our schedule).
Until next week...
Friday, March 8, 2013
Week of 3/4
Back again for another week :)
The core of intrinsic and extrinsic calibration is done, and I can save and load the matrices!
This week, we had a major hardware breakthrough: the interrupts are getting triggered! Now there is still plenty of work for Grady to do with it, but this was a major hangup that is great to get past.
On the software side of things, this week I spent quite a few hours studying/attempting to understand/thinking about the red component and how we will get the left edge of the laser according to the Brown notes. They find the midpoint value (for them it is intensity, for us the red component) per pixel over all frames and then compute the difference image for each image. Then to find the subpixel location of the laser line in a specific frame, you find where the difference image crosses 0 in a specific row of pixels and interpolate to that point based on the pixels before and after. You take all subpixels for a region and find the best fit line. Then for finding when the laser hit a particular pixel on the object, you loop over a particular pixel over all frames and find the zero-crossing; I made the assumption to choose the left side of the zero-crossing as we will be dealing with the left side of the laser line.
We discussed it with Dr. Wolff and since he trusted the document, we went with it. I began implementing it on Thursday. Dr. Wolff mentioned a potential issue of non-negative numbers being used for the channels in an image (the image we get from the camera is a 3 channel matrix of unsigned chars between 0 and 255 inclusive) since the difference image needs to be allowed to be negative. Uchars, schars, and chars don't provide the range we may need, so I convert the red channel matrix from CV_8UC1 to CV_32F (floats) to be able to use negative value.
In addition, this week involved some more plumbing, including setting up some more MVC, adding OverlayView classes, and some message displaying/error handling (e.g. if no camera is connected).
We are planning on working on Saturday and hopefully we can knock out more of the red component stuff and get the OverlayView working (and hopefully Grady can make more progress on the interrupts).
We present our first technical presentation of the semester and plan on talking about displaying an cv-produced image in Qt (and maybe some other info about Qt) as well as what fixed the interrupt issue (hopefully to a level the audience can at least partially understand).
Until next time...
The core of intrinsic and extrinsic calibration is done, and I can save and load the matrices!
This week, we had a major hardware breakthrough: the interrupts are getting triggered! Now there is still plenty of work for Grady to do with it, but this was a major hangup that is great to get past.
On the software side of things, this week I spent quite a few hours studying/attempting to understand/thinking about the red component and how we will get the left edge of the laser according to the Brown notes. They find the midpoint value (for them it is intensity, for us the red component) per pixel over all frames and then compute the difference image for each image. Then to find the subpixel location of the laser line in a specific frame, you find where the difference image crosses 0 in a specific row of pixels and interpolate to that point based on the pixels before and after. You take all subpixels for a region and find the best fit line. Then for finding when the laser hit a particular pixel on the object, you loop over a particular pixel over all frames and find the zero-crossing; I made the assumption to choose the left side of the zero-crossing as we will be dealing with the left side of the laser line.
We discussed it with Dr. Wolff and since he trusted the document, we went with it. I began implementing it on Thursday. Dr. Wolff mentioned a potential issue of non-negative numbers being used for the channels in an image (the image we get from the camera is a 3 channel matrix of unsigned chars between 0 and 255 inclusive) since the difference image needs to be allowed to be negative. Uchars, schars, and chars don't provide the range we may need, so I convert the red channel matrix from CV_8UC1 to CV_32F (floats) to be able to use negative value.
In addition, this week involved some more plumbing, including setting up some more MVC, adding OverlayView classes, and some message displaying/error handling (e.g. if no camera is connected).
We are planning on working on Saturday and hopefully we can knock out more of the red component stuff and get the OverlayView working (and hopefully Grady can make more progress on the interrupts).
We present our first technical presentation of the semester and plan on talking about displaying an cv-produced image in Qt (and maybe some other info about Qt) as well as what fixed the interrupt issue (hopefully to a level the audience can at least partially understand).
Until next time...
Thursday, February 28, 2013
Week of 2/25
This week was a little busier school-wise. We met in class on Tuesday and I went to the How To Pay For Grad School meeting not long after. So we didn't do much Tuesday, although I confirmed that the findChessboardCorners cv method was working. Then Wednesday, Grady wasn't able to meet and Tyler was only able to meet for a while. I was working on implementing a little better MVC structure, but after talking with Tyler, he suggested hooking up the InputView and the TakePicView to the same contrller (a CalibrationController)--they are each related to calibration. This was a better way to do it and I went ahead and changed the structure.
Then Wednesday night I worked at home for a while, continuing work on calibration. I ran into a memory exception when calling calibrateCamera and finally went to bed. I continued debugging Thursday and finally found the issue was not filling the objectPoints vector with necessary data. What happened was the opencv examples use a switch-case statement that looked similar to:
switch(pattern):
case CHESSBOARD:
case CIRCULAR:
{code}
break;
etc.
However, a case statement with no break means it goes to the next statement, which is where they fill the objectPoints vector. This is done in one of the examples online I've looked at but not in a switch statement; I essentially ignored a very important step and misinterpreted the empty case statement.
Anyways, I corrected the error and tested calibration. I was able to perform an intrinsic calibration; I saved the data to an XML file and opened it up: there was the data! I will have to do more to make sure it makes sense and looks correct. I plan to continue working on any remaining intrinsic calibration as well as get into more of the extrinsic process as well.
One question I have is how much error is acceptable in the calibrate method (it returns a double value of the calculated reprojection error (sum of squares between observed imagePoints and projected objectPoints). A rough intrinsic calibration of 10 boards produced an error of 1.02, but what is acceptable?
Anyways, hopefully we can have a productive week next week. Grady is going to talk to Tosh tomorrow and plans on giving Wayne a call. Otherwise, we may end up going with a USB board instead.
Until next week :)
UPDATE
Thursday night I was able to complete the core functionality of both intrinsic and extrinsic calibration, including saving to file. One thing we will figure out (it is pretty simple, just our choice) is whether to store the 3x1 Rodrigues rotation matrix and then convert it to the 3x3 when loading or just convert to 3x3 and store it that way. It doesn't really matter, just something to figure out.
Then Wednesday night I worked at home for a while, continuing work on calibration. I ran into a memory exception when calling calibrateCamera and finally went to bed. I continued debugging Thursday and finally found the issue was not filling the objectPoints vector with necessary data. What happened was the opencv examples use a switch-case statement that looked similar to:
switch(pattern):
case CHESSBOARD:
case CIRCULAR:
{code}
break;
etc.
However, a case statement with no break means it goes to the next statement, which is where they fill the objectPoints vector. This is done in one of the examples online I've looked at but not in a switch statement; I essentially ignored a very important step and misinterpreted the empty case statement.
Anyways, I corrected the error and tested calibration. I was able to perform an intrinsic calibration; I saved the data to an XML file and opened it up: there was the data! I will have to do more to make sure it makes sense and looks correct. I plan to continue working on any remaining intrinsic calibration as well as get into more of the extrinsic process as well.
One question I have is how much error is acceptable in the calibrate method (it returns a double value of the calculated reprojection error (sum of squares between observed imagePoints and projected objectPoints). A rough intrinsic calibration of 10 boards produced an error of 1.02, but what is acceptable?
Anyways, hopefully we can have a productive week next week. Grady is going to talk to Tosh tomorrow and plans on giving Wayne a call. Otherwise, we may end up going with a USB board instead.
Until next week :)
UPDATE
Thursday night I was able to complete the core functionality of both intrinsic and extrinsic calibration, including saving to file. One thing we will figure out (it is pretty simple, just our choice) is whether to store the 3x1 Rodrigues rotation matrix and then convert it to the 3x3 when loading or just convert to 3x3 and store it that way. It doesn't really matter, just something to figure out.
Thursday, February 21, 2013
Week of 2/18
Implementation has begun!
The last week has been very productive on the software side of things. Over the weekend, Tyler was able to develop the main GUI design that we will be using. This includes a tabbed window where each tab is its own class as a QWidget. This was a great step in further development. On my side of things, I spent a good chunk of the week working on intrinsic calibration. I found some resources for doing this in the C++ version of opencv, including one from docs.opencv.org, that I used (while also having the opencv book open). I also followed the format we found last week for getting video from opencv to show up in the GUI. This is done by getting an image from the camera as a matrix, converting the matrix to a QImage, and setting the pixmap of a QLabel to the QImage. So we have video!
I feel we are getting close to being able to test intrinsic calibration. I have gotten it to successfully call the findChessboardCorners method; this was difficult as I was closing the video capture thread for testing, which deallocated memory, essentially making the matrix a null pointer. But I finally found the error and corrected it. So at this point, hopefully we can test the intrinsic calibration next week.
We also have spent some time both cleaning up the GUI code and further implementing our MVC architecture, which will continue to be developed as we go.
On the hardware end, Grady has been having trouble debugging the serial port as it doesn't give an error but doesn't do anything. He sent/will send an email to the board developer to try and set up a phonecall; if it is unsuccessful, we may just purchase a USB version of the board.
Next week should allow continued implementation of the intrinsic calibration and possibly allow for some extrinsic calibration coding as well. I may start coding some of the math processes soon. Also, unit testing would be good, so that may be something we discuss next week.
Overall a good week. Glad to see things rolling!
The last week has been very productive on the software side of things. Over the weekend, Tyler was able to develop the main GUI design that we will be using. This includes a tabbed window where each tab is its own class as a QWidget. This was a great step in further development. On my side of things, I spent a good chunk of the week working on intrinsic calibration. I found some resources for doing this in the C++ version of opencv, including one from docs.opencv.org, that I used (while also having the opencv book open). I also followed the format we found last week for getting video from opencv to show up in the GUI. This is done by getting an image from the camera as a matrix, converting the matrix to a QImage, and setting the pixmap of a QLabel to the QImage. So we have video!
I feel we are getting close to being able to test intrinsic calibration. I have gotten it to successfully call the findChessboardCorners method; this was difficult as I was closing the video capture thread for testing, which deallocated memory, essentially making the matrix a null pointer. But I finally found the error and corrected it. So at this point, hopefully we can test the intrinsic calibration next week.
We also have spent some time both cleaning up the GUI code and further implementing our MVC architecture, which will continue to be developed as we go.
On the hardware end, Grady has been having trouble debugging the serial port as it doesn't give an error but doesn't do anything. He sent/will send an email to the board developer to try and set up a phonecall; if it is unsuccessful, we may just purchase a USB version of the board.
Next week should allow continued implementation of the intrinsic calibration and possibly allow for some extrinsic calibration coding as well. I may start coding some of the math processes soon. Also, unit testing would be good, so that may be something we discuss next week.
Overall a good week. Glad to see things rolling!
Thursday, February 14, 2013
Week of 2/11
Not a whole lot of progress this week. Kinda getting our feet wet with implementation. We decided to go ahead and use Qt for our GUI design. I installed Qt and the Qt add-in for VS on one of our project machines. We then rebuilt the solution as a Qt application. However, we wanted to keep an MVC architecture, and with the tabbed window of our GUI (using QtTabWidget), we wanted to separate the tabs into classes. Tyler looked into it and found a way to do it, but it requires hand-coding rather than using a GUI designer. We decided that this would be the approach we would take.
I have been working on some early implementation of the calibration methods, slowed by not being very fluent in C++. I ran into circular dependency, but did find a workaround using forward class declaration. We found code for doing the camera calibration using C++ online rather than in the C implementation of OpenCV but wanted to confirm using it with Dr. Wolff. Either way we would basically be copying the fundamentals, but the whole process is provided by the LearningOpenCV book. The C++ will be much cleaner, especially as we don't have to allocate space for pointers but can rather add to vectors. [Unfortunately the updated LearningOpenCV book won't be available until June].
Grady has been working on the motor process. He had some hangups with getting it spinning again but went to some old code and resolved the issue. He also has run into a COM1 error for the parallel port, but since it doesn't cause any errors, it may be difficult to troubleshoot.
Tyler is planning on doing the main GUI design in Qt this weekend. Hopefully next week we can continue the intrinsic calibration process and hookup the views so we can more easily determine what we can code (it is more difficult when you don't have all the classes yet).
Also, Tyler installed Windows and VS 2010 on the third machine. This will make it easier as a whole so we all can be developing on similar stations (rather than Ubuntu or having to bring in a laptop).
We were planning on meeting with Dr. Wolff today, but by the time we finally got to his office (4:30), he was already gone. It's early in the semester and there wasn't a ton of stuff to really talk about (well, sometimes it comes about anyways and the meeting still lasts a while :)
Anyways, we will definitely plan on meeting with Dr. Wolff next Thursday and on a more regular schedule from then on.
Cheers. Happy President's Day.
I have been working on some early implementation of the calibration methods, slowed by not being very fluent in C++. I ran into circular dependency, but did find a workaround using forward class declaration. We found code for doing the camera calibration using C++ online rather than in the C implementation of OpenCV but wanted to confirm using it with Dr. Wolff. Either way we would basically be copying the fundamentals, but the whole process is provided by the LearningOpenCV book. The C++ will be much cleaner, especially as we don't have to allocate space for pointers but can rather add to vectors. [Unfortunately the updated LearningOpenCV book won't be available until June].
Grady has been working on the motor process. He had some hangups with getting it spinning again but went to some old code and resolved the issue. He also has run into a COM1 error for the parallel port, but since it doesn't cause any errors, it may be difficult to troubleshoot.
Tyler is planning on doing the main GUI design in Qt this weekend. Hopefully next week we can continue the intrinsic calibration process and hookup the views so we can more easily determine what we can code (it is more difficult when you don't have all the classes yet).
Also, Tyler installed Windows and VS 2010 on the third machine. This will make it easier as a whole so we all can be developing on similar stations (rather than Ubuntu or having to bring in a laptop).
We were planning on meeting with Dr. Wolff today, but by the time we finally got to his office (4:30), he was already gone. It's early in the semester and there wasn't a ton of stuff to really talk about (well, sometimes it comes about anyways and the meeting still lasts a while :)
Anyways, we will definitely plan on meeting with Dr. Wolff next Thursday and on a more regular schedule from then on.
Cheers. Happy President's Day.
Thursday, February 7, 2013
Week of 2/4
Well, the final semester of the undergrad career has started. Fall and its project planning is past. We weren't too productive in J-Term; Grady did do some hardware research and coding and Tyler did some reading up on C++. On Monday, Tyler designed a rough GUI layout with tabs and I did some stubbing of the non-view classes (e.g. Models and Controllers).
Tyler and I met on Tuesday to go over what we did and made a few additions. We also made a new Github repo for our actual production of the project.
We met today to determine our semester meeting times and were successful in finding times that would work, allowing us to meet about 3 times during the week. So hopefully this will work out nicely.
Hopefully we can make some good progress and begin implementation; and hopefully we can pick up some of the slack for not doing much in J-Term.
Tyler and I met on Tuesday to go over what we did and made a few additions. We also made a new Github repo for our actual production of the project.
We met today to determine our semester meeting times and were successful in finding times that would work, allowing us to meet about 3 times during the week. So hopefully this will work out nicely.
Hopefully we can make some good progress and begin implementation; and hopefully we can pick up some of the slack for not doing much in J-Term.
Subscribe to:
Posts (Atom)