Hi,
I’m trying to control the LSS motors via the Lynxmotion lss adaptor board.
When I send commands (e.g. #1D400T800) through the lynxmotion terminal the motors seem to work fine.
However, when I try to do the same thing via any other serial terminal program, the motors don’t seem to work. I made sure, the baud rate, parity, stop bit and bit size were set correctly (e.g. 115200, none, one, 8).
I tried various different serial terminal softwares such as coolterm, serial port monitor and serialportWriterReader but to no luck.
What am I missing here? why is it only working for the lynxmotion software and not for other serial programs?
I assume you mean the terminal at the bottom of the LSS Config (Wiki here)?
Please note that all commands are of the format #{id}{command}[value]\r. That \r at the end is required for the command to register. The LSS Config terminal always sends one but in most serial communication software you have to configure this. For example, the Arduino serial monitor allows selection of ending in this way (choices are nothing, \n, \r or \r\n):
Thank you very much for the reply. I actually figured this out the hard way. But you are right, the carriage return was indeed the problem. It seems the software I was using may have sent “back slash” and “r” as two separate ASCII values.
And Btw, your python driver code on git was quite helpful Hope you would implement a C++ version soon. Or a ROS driver would be very useful to many people I think. Cheers!
Oh no! Sorry I didn’t answer sooner. Then again, if you are doing robotics, these kind of situations (where a small thing stumps you) will be common place…
Would you mind letting us know what software you were using? That would probably be quite helpful for other users of the LSS in case this find this answer in the future.
Yay! It is always neat to know people are using it. Thanks!
Hadn’t actually considered this since people doing robotics in C++, but I’ll add it to the list. Can’t really promise a timeline yet, but at least I’m aware of the need, now.
We have something coming for ROS shortly. If you are curious you can have a look at how this is going over here. If you want you can watch the topic so you get notifications when it is updated:
I’ve tried the windows powershell, Serial port monitor and Free serial analyzer. The last two are opensource programs. What worked was ubuntu “screen”. It was a bit confusing at first but got the job done. Don’t even need to send the carriage return char for that.
I have one last question if you don’t mind. How fast is the serial communication is usually at 115200? I’ve sent serial messages to read the position, temperature, voltage and current values and the whole process seem to take about 60 ms to complete (for both reading values sent by the motors and to write request commands to the motors). Would you by any chance know how long it would usually take a single read/write process (e.g. read the voltage) to complete at 115200? I’m using the library given at https://github.com/wjwwood/serial
Sounds good! I quite like screen if I need to do some quick serial port stuff. It has many interesting features, too! Go check the man page!
Even if it is not really your last one, I don’t mind either way…
Lets break this down:
Well, it is 115200 bits per second, or 11520 characters per second when using 8n1 settings (1 start bit, 8 data bits, no parity, 1 stop bits = 10 bits per character). Therefore, one character takes 1/11520 seconds to transfer, or ~86.81 µs. In my case it was ~84 µs:
Therefore, sending something like #3QD\r would take 5 x 86.81 µs or roughly 434 µs. Of course, raw transfer speeds are not enough. You need to also include processing time (on the LSS) before a reply is sent back. Here’s a screenshot from my logic analyzer software of a LSS @ 115.2 kbaud responding to a QD command:
As you can see in the image, the total duration of the transaction from when the last character ( \r ) is received in the LSS (from the PC) to when the response is fully transferred for the QD command was about 827 µs (purple). The processing time for the QD command inside the LSS was roughly 316 µs (green) and the response time (i.e.: outputting the values physically on the bus) took just a bit over half a ms, or ~510 µs (orange), yielding us a length per character of ~86 µs which falls right in the expected range for communications at 115.2 kbaud rate.
Now, if this is your bottleneck, the best you can do is increase the baud rate. The LSS and LSS Adapter Board were tested at up to 500 kbps with 36 servos on the bus at the same time and the bus completely saturated at 100% with 0 packet loss. Of course, running at 500 kbps will reduce transmission time of one character by a factor of about 4.34, which is quite significant. In the example above, the orange section would take about 118 µs instead of 510 µs. As you probably figured out, though, there’s no real way to reduce the processing time on the LSS, so that ~316 µs will stick around no matter the baud rate.
Now that we’ve established the communication speeds, lets dive into the real reason(s) why the transfers with the LSS might actually much slower (or seem to be):
Your main issue here will be FIFO buffer & driver latency. Most USB interfaces will use a FIFO to allow for efficient communication at high speed. The alternative would be to generate an interrupt for every single character; doing so would lead to your CPU doing effectively nothing at all due to interrupt processing overhead while communications are on going. This gets worse the faster the communication speed (in the case of UART, the baud rate).
To get around this issue, a FIFO buffer is used. The characters are received in the interface and sent to a FIFO buffer, typically a simple circular buffer. Once that buffer reaches a certain per-determined limit (often half full, or 3/4 or full; where full is a number of bytes. That size depends on hardware and drivers) an interrupt is triggered for the drivers. These will read the buffer’s content and pass them along to the OS’ handling routine which will eventually trigger an event in the library/code/etc. that you are using to handle serial port communications.
As you can imagine, all of this is quite a lot of abstractions and not exactly lighting fast. Also, since this is a regular OS and not a real-time one, there are no hard limits on total duration of those operations and how much jitter (variable delays) can happen from other processes (a modern OS runs dozens if not hundreds/thousands of threads at once).
For example, when I run my code in a profiler against a LSS bus running at 115200 baud rate, I often get delays of ~15-20 ms before replies are sent back to my code running in a .NET app. But, sometimes, other stuff is schedule at the same time and the response is receive nearly a full 100 ms later. And, unfortunately, there is nothing I can do about it! Neither plan for it or prevent it…
The point that is important to mention is that a FIFO buffer in UART hardware will have a size and a certain trigger limit (half, full, etc.) for when to generate an interrupt. If your communication doesn’t fill that buffer, such as my reply above of *3QD2\r which is only 6 characters, the hardware will not trigger an interrupt until the trigger size is met or the bus is idle long enough to timeout. Depending on the hardware interface (ex: FT232R) and the drivers installed and other configurations, this can be quite long if it is optimized for large transfers (very typical, default use case) instead of small, numerous quick packets (what we do with the LSS).
Also, whatever library you use (and programming framework, compiler/interpreter and its optimizations) will affect how fast you can process those messages coming from the OS telling you that data is available.
Hopefully this gives you a better idea of why communications could be much slower than expected, such as 60 ms to read one value vs 827 µs for the actually processing and transmission of the response.
A good option to alleviate these issues if you are stuck on a non-real-time OS (such as Windows) is to package your queries to one(1) servo together so the servo can respond to them all at once. This has two benefits:
The servo will process all the queries quickly and you won’t have to wait for each response to send the next query, therefore limiting how much you wait for each data piece.
This will most likely cause the FIFO buffer to be full quickly, giving you access to the data sooner instead of waiting for an internal timeout on the UART hardware.
Here’s a quick example of gathering some telemetry from a servo:
Instead of sending #3QD\r, wait for a reply then send #3QV\r, etc. you could construct a string and send it all at once, such as: #3QD\r#3QV\r#3QT\r#3QC\r#3Q\r.
By doing this your servo will receive all of this in one large burst and respond as fast as possible. This will allow you to dodge all those delays due to hardware timeouts and interrupt delays down to just one for each! Much improved, right?
Try this out and let me know if you see any improvements!
Sincerely,
(1): If you need to query two or more servos, you’ll still need to wait for each one to respond before querying the next one. If you don’t you will most likely get bus contention, where two servos try to use the bus at the same time. That’s A Bad Idea ™!
@scharette Mate, you are an awesome dev! That reply was very comprehensive. And you are right, it did improve the performance very well! Thank you very much!
Here are some of my findings. Hope it will be useful for whoever might run into a similar problem in the future.
The bottle-neck is the FIFO buffers and the latency that comes with them (i.e. buffer time-out). The latency to empty the read/write buffers takes forever. Depending on how many bytes are in the write buffer at the time, the FIFO driver waits a fixed amount of time before pushing the data to the serial bus that’s connected to the motor.
Sending the commands in a single string (instead of calling the write function multiple times) can therefore reduce the time greatly. However, depending on the size of the string, some latency may still get added.
I’ve read somewhere the FIFO buffer is 16 bytes, so I tried padding the string with random characters to match that length and force the buffer to push the data without waiting for the buffer time-out, but this didn’t seem to work. I guess there may have been something else in the buffer that messes up the timing (couldn’t find a way to show this).
However, ubuntu comes with a package called “setserial” which can be used to mess around with the buffer latency.
You can use “$ setserial /dev/ttyUSBx low_latency” to minimise the time it takes to read/write from the buffer. In my work I sent a string to read the position, temperature, voltage and current in one long string message as was suggested by @scharette . I read the whole output from the motors with 5-7 ms on average. I saw that between read values there was about a 0.9 ms delay. This corresponds with the information provided by @scharette where it takes about 827 µs for the motors to process and send the data over the bus.
There maybe a way to change the read/write latency in windows as well, but I have not investigated this.
In conclusion it takes about 1-2 ms on average to request and receive a given sensor data (i.e. temperature, voltage, current, etc.) with 115200 baudrate provided the serial read/write buffer latency is minimised.
@Eranda43 oh! I totally forgot to respond to this amazing reply you placed! My bad!
I’m glad this was helpful! We are certainly very interested in improving all aspect of the LSS products, so feel free to get back to us with any other ideas/comments/etc.! We’ll do our best to either clarify stuff or improve it as much as we can.
Thanks for that! I’m sure the rest of the community will appreciate your efforts!
Here are some ideas concerning what you found out about all of this:
I’ve only really had the issue (on Windows) with reading from the buffer, not writing to it (and therefore to the LSS bus). That being said, it may be handled differently on various Linux platforms.
Yes, there can always be some extra latency just from the fact that if you are not running a RTOS then anything can happen, delaying a reply from the hardware/drivers.
And, of course, even if your packet exceeds the size of the buffer, the extra bytes may be delayed until the next buffer timeout (though that was not my experience with PC TX > LSS RX).
Another option (at least for PC TX) might be to simply tell the UART drivers to flush out the data (instead of waiting for exact size/timeout).
As for replies, it may be possible to add a configuration to the LSS firmware to add padding to replies. When running at 500k baud (~20 µs / character) the delay from buffer timeout/drivers/OS HAL is far, far longer (sometimes dozens of ms) compared to adding a few characters to each reply. It would have to be tested, but that may be one way to circumvent the issue. Since whitespace is ignored in the LSS protocol that would be an easy way to add padding (such as at the end before the footer; a carriage return).
With a lot of hardware (and proper drivers) you can reconfigure the buffering to be different. By default, they are usually set to 8-16 bytes to cater to high-data rates instead of low latency (optimally every byte would trigger an interrupt!). So that should be an interesting thing to look into. I know many of the FTDI chips (such as the one used in the LSS Adapter Board) have configurations that can be changed to lower latency.
I assume that package is probably changing the chip’s/drivers settings to improve the latency. Since the data rate for LSS is quite low but we want high latency smaller buffers are definitely a good way to go. And if you don’t have much running simultaneously anyway and multi cores than it is a no-brainer to do so.
I’ll have to look into that package and see what I can find out about it. I personally mostly run Arch or arch-based systems (such as Manjaro).
Well, other than changing settings in the chip/its drivers, I’m not sure if there is anything else to do. To be investigated too!
This seems like it would be quite acceptable for most cases.
We do have some customers using the LSS bus with a RPi4. The big advantage there is that the RPi4 has multiple physical UART buses available (of course, you’ll need a level shifter / voltage translator such as RB-Spa-879 to use them with the LSS). You can change the FIFO size on some of those buses. If I remember correctly, some can be set to 1 byte and then all that’s left is to give it high priority and you’ll be near real-time. Also, since there are multiple buses available, you can set each UART for a group of LSS to keep everything fast.
@cmackenzie has been doing some great work on this recently. You can read some of the details about it here. Be careful, it is a long topic full of very interesting discussions… you may get sucked in so make sure you have a handy snack nearby!
Hope you would implement a C++ version soon. Or a ROS driver would be very useful to many people I think. Cheers!
Then again, if you are doing robotics, these kind of situations (where a small thing stumps you) will be common place… 
