Couple of months ago we decide to participate in Intel's perceptual computing challenge (http://perceptualchallenge.intel.com). The idea was to extend driver cockpit application to control Veterobot (http://veterobot.org) with gestures and voice recognition. We are using detected thumb position in 3D space. Position along horizontal axis is used as a steering signal and distance to the camera (depth) to control acceleration (forward and backward).
It was a fun exercise but some more tuning is still required to achieve smooth control experience. So it is work in progress.
Edit: somehow, I can not see the attached video in the blog post, so here is the link: http://www.youtube.com/watch?v=5AkzF8kIeC0
Gesture recognition technology is indeed interesting area, however I am not sure whether it makes sense for this kind of robot control. The reason is - after about 30 minutes of intensive testing my shoulders were really tired. It is necessary to keep hands straight in front of the camera and it is not very convenient during the long period of time. And honestly, I can achieve pretty much the same effect with mouse.
So for me it was a nice exercise to learn and play with promising Intel’s technology but I am not convinced yet that it is well suitable for this particular application scenario. Intel’s perceptual SDK also provides rather simple way to track the head movement and I was thinking about using it instead of hands tracking. Than I realize that if I want, for example, turn left or right, then I need to rotate the head, but at this moment I can not see the monitor anymore which is also not perfect for control . So there is still room for improvements.
I can imagine what you mean. But gesture does not have to with thumb. Why not train the robot to process gestures like arms up, arms to the side, one arm to the side…
That would be more recognizable i guess since arms are bigger than thumbs and the change of the picture before the background is also bigger.
Anyway, sometimes you try something and then you realize that there is a better way but what you have learned might help you later. So it’s not wasted time
> But gesture does not have to with thumb. Why not train the robot to process gestures like > arms up, arms to the side, one arm to the side…
I am not sure that it will be improvement compared to the mouse or or joystick. Also, based on my experience with Intel’s perceptual SDK, the recognition rate even for the thumb is quite good. So it is less the question of recognition, but more of general usability of gestures to control the robot.
> Anyway, sometimes you try something and then you realize that there is a better way but > what you have learned might help you later. So it’s not wasted time
I am totally agree with you here! At the very least, now I can mention gesture recognition technology in my resume which is already positive result of these efforts
> Why not train the robot to process gestures like arms up, arms to the side, one arm to the side…
Wait, I probably did not understand what you mean and was answering too fast. Do you actually mean to do recognition on the robot itself, i.e. using on-board computer? In my case, gesture recognition was made on the PC, i.e. where the driver console application was running.
If yes, then I think it is indeed coulde be useful in case where there is no second computer with control software to drive the robot. However, it will require the driver to be in the field of view of the camera, which is quite a limitation. So here is also plenty of problems to think about…
Well, yes. For an application like field operation the robot could do some tasks on specific gestures. I could imagine to gesture the robot from a remote location (but still in camera sight) to bring me my toolbox or in private sector gesture him to bring me a beer while I am lying in the garden.
Haha, a note like that in your resume is quite useful
. So there is still room for improvements.