Lipsyncing is the process of making an animatronic costume or model mimic the actions of speaking. I use the term mimic as it is currently impossible to make an artifical tougue or larynx that will duplicate speech. So we have to use a speaker with a recorded voice and have an animatronics track to make the movements.
Real Time Lipsyncing is where you do not have the luxury of being able to record tracks and add the lipmovements. It has to be done on the fly and often whilst interacting ad-hoc with an audience.
This is my concept of how it can be done usingoff the shelf technologoes and mininmal computing power.
First off I will be making and wearing a lipshield. This is a flexable silicon mask that is glued to my lips. It will have 8 embedded IR SMD leds set up as 2 circuits (A-B-C-D) and (E-F-G-H). A WII IR Camera will be positioned so that it can see all LED's (A fisheye lens will be needed)
The microcontroller (Picaxe or Arduino) will strob the 2 sets of LEDS and thus beable to read the 8 point in 2 sets of 4 co-ords.
Point A is important as it will indicate how much the mouth is opened. The other points can be used to adjust servo's around the characters lips so to simulate the speech pattern.
Where the character has a tongue that can be extended we can use a IR Emitter/Detector module focused at the blue circle. Whilst the other LEDS are turn off this LED can be illuminated and so via reflection from the tongues surface give an approximation of how far the tougue is extended (more refection more tongue). Also by using Point A it can work out if the mouth is opened and thus stop false reading.
All parts are off the shelf and I will be using Chris the Carpenters WII Camera module and possibly a PicAxe 28x2. The IR detector is here http://littlebirdelectronics.com/products/qre1113-line-sensor-breakout-analog
Many thanks for following my ramblings
Dave
p.s. - The animatronic jaw can be seen here http://www.marcwolf.org/gallery/default.aspx?aid=4
This is a companion discussion topic for the original entry at https://community.robotshop.com/robots/show/realtime-lipsyncing-for-animatronics
