You might consider using PID control for tracking, I found it to be very complimentary to OpenCV input - it’s also nice in that PID allows easy extensibility to control other actuators besides servos… that is if you ever decided to re-use the code with a motor driven pan / tilt kit.
The short of it is… it’s a clever function which tries to follow chaotic input. If you think of tracking as trying to follow 2 chaotic inputs (the X & Y axis) - then you use 2 PID methods to follow them - it becomes smooth tracking…
There are several constants in the formula you can tweak to suite your specific system and desired attributes -
Here are some examples of people trying different values with MRL