Introduction
This article is the follow up to my previous post on PWM generation with GPIO using Xenomai real-time Linux extension. Despite clear improvements with Xenomai, the results were not good enough for us. In particular, under the heavy system load, PWMs were generated with jitter which was high enough that our servo motor starts shaking. So we decide to continue our search for the better solutions now in the kernel space.
This article represents the results we obtained after implementing Xenomai RTDM kernel module. It also compares performance of the kernel-based solution with our previous user-space based approach.
Problem
Our previous solution performs well if the system load is low to moderate. However, under heavy load, we were unable to keep the servo motor stable. We believe, that one of the main reasons for it was increased jitter which results in the in-precise pulse length. To check this hypothesis, we connect the oscilloscope to the signal (GPIO) pin to see what is really going on there. The first attached video (jitter-user-space) illustrates what we have seen.
We start our xenomai-based user-space application with no system load and after about 5 seconds run tar jcf lib.tar.bz2 /usr/lib command which leads to the top showing about 95% CPU usage. The distance between two horizontal lines around the falling edge of the signal corresponds to 50uSecs (as also shown in the bottom of the scope's screen). So the very rough estimation of the jitter (just visually) is about 40uSec. So with the whole pulse length of 1500uS (50% duty or middle servo position) it would be little less then 1%. This result roughly correlate with observable servo motor movements described in the previous post.
Solution - Xenomai RTDM kernel module
It is well known, that moving into the kernel space could provide much better timing. We were trying to avoid this step as long as possible (because it is easier to develop and debug in the user space), but based on the results mentioned above it looks like we can not improve timing any more from the user space. So we decide to write kernel module. Xenomai offers convenient API to write real-time kernel space drivers which is called RTDM - Real-Time Driver Model (please see introduction paper and API documentation for more details).
The result of these efforts (source code) could be found on git-hub.This repository contains the kernel driver and user-space test application. There are some instructions about compilation and how to run the test available in the README file. If you decide to compile and run everything yourself, we recommend to read the post about how we set-up our development and cross-compilation environment.
The core part of the driver is the 20mSec periodic task (function) which is toggling the GPIO pin in the endless loop:
const int which = (int)arg;
// Toggling the pins
for(;;) {
//set_data_out has offset 0x94
iowrite32(0x40000000, gpio + 0x6094);
if(0 != rtdm_task_sleep(up_period[which]))
rtdm_printk("PWM: rtdm_task_sleep() returns error\n");
//clear_data_out has offset 0x90
iowrite32(0x40000000, gpio + 0x6090);
// wait until the next pulse should start (20mS interval)
if(0 != rtdm_task_wait_period())
rtdm_printk("PWM: rtdm_task_wait_period() returns error\n");
}
This loop essentially sets the pin to 1, then waits requested amount of time (pulse width length), then sets the pin to 0 and waits until the next period starts.
Running driver-based solution under the same heavy load conditions shows considerable improvements compared to the user-space solution. The second attached video (jitter-rtdm) illustrates the performance of this solution.
Similar to the case with user-space application, there are two lines around falling edge with 50uS distance between them. Here we add the system load after about 8-th second on the video. The jitter is smaller then in the previous test and as a result, servo motor is also much more stable.
Still not perfect
Unfortunately, developed kernel module does not solve the problem completely. Despite clear improvement, servo is still shaking. Much less than before, but still shaking under the heavy CPU load. So we conclude that there might be two potential sources of the problem.
The first one, is rather obvious - the jitter observed in the video above. But taking in account, the simple generation loop mentioned above, we do not see how it could be improved. So the question is - did we reach the limit of Xenomai+Linux abilities to provide real-time behavior? If yes, then it is rather sad news. BeagleBoard xM we are using for this experiment is rather powerful computer running at 800MHz. Yet we still can not control servo motor as precise as the one can easily do with micro-controller. It is clear that there is much more going on with real OS then with single application on micro-controller. Yet our hope was (and still) that 800MHz with real-time supervisor (Xenomai) running Linux kernel as a separate preempt-able task could provide better performance (much smaller jitter).
Looking for further possible reasons for shaking servos, we notice somewhat strange behavior which we could not explain. To check what we are generating, we decide to display 20mS (similar as our signal) square wave generated by the oscilloscope near our PWM signal. We also instruct the scope to synchronize on the rising edge of this reference square wave. Our expectation was that we would see our PWM signal staying still with respect to the test square wave (because they have the same period of 20mS). That is why, we were surprised to see the behavior illustrated on the third attached video (moving-signal).
Here, the yellow (bottom) signal is the square wave generate by the scope. The blue one (up) is our PWM signal which is drifting to the right. Currently, we do not know what might be the reason for such behavior. The only idea is that rtdm_task_wait_period()function does not work precise enough and the process is systematically waked up later (hence the drift). If yes, the again, the question is did we reach the limit of Xenomai+Linux abilities to provide real-time behavior?
Of course, there is a chance that we just misunderstood some programming concept and there is a bug in our code :-)
Anyway, it seams that we are not done with this problem yet and there still open questions which can not answer right now. That is why, we would highly appreciate any hints and suggestions about what might be wrong with our implementation and how to improve it. So your comments are very welcome!
UPDATE: please see the following discussion thread for the promising idea how to fix the jitter problem.
https://www.youtube.com/watch?v=NOSbGWT_tyo
