Here is some streaming video being masked by the depth image of the kinect. I use OpenCV's inRangeS to find a range to create a mask from the depth image, then cvCopy to copy it over the video image.
One strange thing I've noticed is the video image is down and to the left of the mask. If you look on the right edge you can see another anomoly which is the boundry of the depth mask. Apparently the depth image is 632 X 480 vs 640 X 480..
The offset seems to be ~32 pixels as a crow flies - which means I have to adjust the mask 25 pixels left and 25 pixels down...
I'm Wondering :
1. Is this for all Kinects or just mine... I suspect its all because of the way it works & 632 resolution, but I really have not found much info on the subject.
2. How the heck am I going to move the mask over given the set of OpenCV functions?
The Results
Looks better !, the alignment falls off as the target gets closer to the edges of the 640x480 screen, but pretty good when centered, which is what I'm gonna work with for object identification.
Without doing any kind of border interpolation or anything like that, you’ll have to live with a smaller image if you want an image that only contains true info from both the normal camera and IR depth mapper. The largest correlated image you can get will be (632 - 25) X (480 - 25), i.e. 607 X 455… what awful numbers =)
Anyway, that assumes that the scaling between the two images is the same, and that they really are offset by 25, 25 pixels.
There are probably more elegant ways to process the two images before you combine them (remap()?), but the first method that comes to mind is using getRectSubPix(). Create two empty 607 X 455 images, rip out the appropriate pixels from the two sources, then cvCopy() to combine the sub pixel images. That way you get your offset and cropping taken care of in one step per source image.
After scratching my head last night, I decided I’ll try this: Which is close to what you suggested except it only needs 1 empty image, and 1 copy
1. create a completely black image the same dimensions of the RGB image - 640 X 480 2. use cvSetImageROI on the depth data - if 25 x 25 is correct - I would cvSetImageROI(kinectMask, cvRect(25, 0, 615 - 8 , 455)); … the (-8) to get rid of the irritating band on the right
3. then cvSetImageROI on the black target - cvSetImageROI(black, 0, 25, 615 - 8, 455) 4. do a single copy from the depth to the black blank cvCopy(depth, black);
