Ping Pong Ball Juggling Robot

Let's Make Robots Robots
1

This entry is about the quest to get a machine to juggle a ping pong ball. 

I started thinking about a ping pong ball juggler 2 years ago. And it was around that time that I built the first one of them. it uses LED - Photo transistor pairs to "see" when and approximately where the ball comes down. I used 4 micro servos to move a wooden plate.  It's the machine in the first YouTube video.

The next major update came when I decided it might be interesting to try to track the balls position by using the noise the ball makes when it hits the plate. I installed 4 mics and switched a FlipFlop on as soon as the sound level got over some threshold. The FlipFlop's value then gets read by a micro controller as fast as possible to then compute the time difference between the sound waves hitting the mic on the left in comparison to the mic on the right. The same goes for the front and back mics. This way 2 dimensional data about the ball position is gathered. It worked quite well. The machine on this stage is the one in the second YouTube video.

After that I did other things for a long time. But some months ago I wanted to rebuild the machine from scratch. This entry is about this rebuilt version. The 2 videos on the top are just meant for a quick history on how it started. So, what changed? I no longer use servos but Nema17 stepper motors for the movements. I bought some belt drive components and ball bearings. I used those to convert the axial motion of the steppers into linear motion. It's the same setup as in an inkjet printer. 

The main controller of this project is a Raspberry Pi 2. It has a Raspberry Pi camera module NoIR connected to it. Because it is NoIR, there's no IR filter on the thing. I originally planned to use infrared light to brighten up the ball in front of the camera. But it didn't work well. The IR elements I used were way to weak. The infrared noise in the background (sunlight etc.) would overweight the infrared light getting reflected from the ball. Another problem was, that the IR signal gets picked up on the red-channel of the camera. This means that even if there is no infrared noise from sunlight, the infrared light's intensity still has to surpass the normal red light's intensity to be recognizable. So that idea got dropped. I ended up using the green-channel on the camera with green light from a RGB Power LED pointed at the ping pong ball. Sure enough, normal white light would've done the job as well, but I just happened to have a RGB LED laying around so I ended up with green.

Another important thing is this: You might be asking yourself "Why did you use a light source anyway? Couldn't you just use OpenCV and get the thing to recognize the white ball with normal lightning conditions?". The answer to this is: Probably not. At least not with a Raspberry Pi 2. The reason being that we need 90 fps real time image processing to track that ball. The image processing has to be done fast. I didn't really use OpenCV for anything at this Point in time but my guess is, that it wouldn't work well, because I don't think I could get the processing rate up to 90 fps with OpenCV. Maybe I am wrong about this, but it was my gut feeling, so I went on to program a really simple algorithm that could take care of the image data really fast. The algorithm looks directly at the data stream, so no conversion to a different datatype (e.g. NumPy) is needed. This is one reason this algorithm is quite fast. The other reason is, of course, that it is a dumb algorithm. All it does is look for a clump of bright green pixels on a fairly black background and then compute the center of it.

After the algorithm extracted 3D coordinates out of the images it examines that data and, in the case the ball is going downwards and gets quite near to the racket, it uses that specific data set in a PID controller to get some useful correction values on the racket as the ball is moving towards it (the machine tries to get the ball into the center of the racket). The movement values are then sent to the Arduino Uno over the serial port. The Arduino Uno runs grbl (a program for running stepper motors very efficiently) and all it does most of the time is waiting for some new instructions from it's master; The Raspberry Pi.

Some lines about the tilting mechanism: The tilt in the racket is produced by different heights of the 3 belts on which the racket is mounted. The joint "belt to racket" is a rather elastic one. This is why a tilted racket can be accomplished without ruining the joints or other parts of the machine. 

It's worth mentioning that the thing isn't working perfectly at all. A 2 hour juggling session is definite not one of the things this machine is capable of doing. It is able to juggle the ball for about 2 mins. Then it makes some misjudgment and the ball drops. So it isn't great. But then again: It only uses one single camera to compute 3D coordinates of a flying object. using 2 cameras would improve the performance. But it would also make the thing more complicated.

Here are some pics of the machine:

The build of the Juggling Robot

Juggling Robot with all of it's components

 

Raspberry Pi camera module plus RBG Power LED

Raspberry Pi camera module plus RBG Power LED

 

Camera module view from back

Camera module view from back

 

The ball bearings mounted on the top

The ball bearings mounted on the top

 

The connection racket - belt drive

The connection racket - belt drive

 

The base built

The base build

 

Nema17 Stepper Motor

Nema17 Stepper Motor

 

And here some gifs made from image data of the Raspberry Pi camera module:

This gif was made to check if the ball in the image data looks crisp enough

This gif was made in the very early stages of this project. It was made to check if the ball in the image data looks crisp enough to allow an image processing algorithm to extract the ball position from it.

 

One ball Juggling

Here is some image data of actual ball juggling. The red light was put there to make the racket movements visible. The grid represents points at which the algorithm checks for pixel brigthness. If the value on a gridpoint is bright enough the algorithm computes the center of the clump of bright pixels at that specific position (this data is used as x and y coordinates). The z coordinate is extracted from the width of the diameter of the ball.

Here's the Raspberry Pi 2 Python code:
https://github.com/T-Kuhn/PingPongPi

And Here's the superb grbl code (the thing running on the Arduino Uno):
https://github.com/gnea/grbl

It Juggles a Ping Pong Ball with a Racket

  • CPU: arduino uno, Raspberry Pi 2
  • Operating system: Linus
  • Programming language: C++
  • Sensors / input devices: Raspberry Pi camera module
  • Target environment: indoor
This is a companion discussion topic for the original entry at https://community.robotshop.com/robots/show/ping-pong-ball-juggling-robot
2

Ping Pong Ball

30.12.2016

What an interesting and well written article…excellent wok, indeed!

One thing: https://github.com/grbl/grbl   give a 404 Error.

3

So does the racket pivot to

So does the racket pivot to try to keep the ball centered? Is there just enough play in the belts to allow each belt drive to be at a different height?

4

pivot

Yes it tries to keep the ball centered (I really should have mentioned that somewhere… Will put it in there!). And yes, just as you suspected, there is enough play in the belt itself and in the joint “belt to racket”. So, different heights result in a tilted racket. See picture “The connection racket - belt drive”. The connection is actually made by using leftovers from the belt. Because the material seemed to have just about the right amount of elasticity. :slight_smile:

5

Blind Implementation of Juggler

Cool project. Here’s a similar blind implementation: (http://www.idsc.ethz.ch/research-dandrea/research-projects/juggling-machines.html)

6

thx

Oh, Thank you Bonzadog.

I corrected that link!

7

Awesome!

Very impressive work and a great read.
Perhaps if you plan to expand on it, you could record the data and train a neural net?

Might be able to improve on your 2 mintes eventually (not that 2 minutes isn’t impressive enough as it is!)

8

neural nets

Thanks for the comment!
That would be sweet. Never done neural nets. Want to learn more about them. Is there a neural net library you can recommend for python?

9

Blind Juggler

Hi ShuAux,
I read about that one. Thanks for the link!

10

Keras

Sure, i’d recommend using Keras, lots of tutorials online for it.
It uses tensorflow to do all of the hard work, but makes it really easy to get stuff up and running:

You can see docs for keras here:

https://keras.io/

You can find lots of great stuff for neural nets, deep learning and installing/working with keras here:

http://www.pyimagesearch.com/category/deep-learning-2/ 

 

Hope it helps, looking forward to what you come up with next!

11

Keras

Thank you very much!

12

Well done!

Nicely done!

13

Interesting

Good work and clean presentation. I will look forward to your work :slight_smile:

14

That is a hard problem to

That is a hard problem to solve.  It is amazing you solved this with just a Pi and an Arduino.  Very cool!  

Thank you for including your less successful attempts.  That was interesting to see the evolution of your thought processes on how best to approach this.  Unless you are sick of this, your next project could be an automated ping pong player.  You have already done a lot of the hard stuff on it with the vision piece, etc.  

Regards,

Bill

 

15

Very cool! That is a hard

Very cool!  That is a hard problem to solve, and I am very impressed with how it works.  This is something that is hard for a human to do, and to get a machine to reliably do it is very cool.  

I also like how you showed us the previous versions since it showed the evolution in your thought.  Maybe you are thinking of making a machine that will play ping pong against you.  Do some machine learning with it and before you know it, it will be unbeatable!

Regards,

Bill

16

Hey Bill!

Hey Bill!

Thanks for the comment! It was a bit tricky. But getting it to play against a human is a hole other level of trickiness. That’s a task for hight speed insustrial robot arms. I guess. ^^

But I like what you said about how it’s hard even for a human to juggle a ping pong ball for a while. I thought about this a lot. I sometimes do small presentations about some of the machines I build. And usually people like that ping pong juggler the best (the old version though. Didn’t do any presentations with that new thing 'til now ^^). I think people like it because what the machines trying to do is easily understandable and it’s not too easy of a task to be boring.

17

Saw your project featured on Hackaday

That’s always cool.

18

Super cool! And maybe some more info

Hi! First of all I must say your project is amazing! Those intermediary versions were already cool, but the final one is just incredible! I’m studying engineering and would love to replicate your work, so could you answer me a few questions?

Would you mind giving some more specifications about the hardware you used? Specially about the stepper motors as there are numerous possibilities and I am kind of new in this subject.

Also, I saw that you used a cnc shield with an arduino. Why didn’t you chose a cnc shield directly whi the raspberry pi?

And finally, would you have any advice on what to improve or do differently?

Thanks very much!

19

The stepper motors are NEMA

The stepper motors are NEMA 17 motors with a torque of roughly 0.25 Nm. It was barely enough to get the racket moving / accelerating fast enough to get that ball bouncing. I had to run them at 32 Volts (near the upper limit of the stepper drivers.)

I used the grbl code on the arduino (https://github.com/gnea/grbl). That’s why I went with the arduino raspberry pi conbination. I really like that grbl project. It basically turns your arduino into a full fledged CNC controller.

My advice is to not use my code. You could use it as reference of course. But working with it as a base will bring you no joy. Your first goal will most likely be to get those steppers moving. Then there’s the mechanical part. you need to get a build which supports the motors and belts. Just use whatever you find. Then, as soon as the basic up / down / tilt motions are possible, you can start with the image processing and pid controlling and whatever you have.