Lynxmotion SES V2 Hexapod Robot

Thanks @dialfonzo (@cbenson)

I will play more later with the new stuff. That is hopefully one can do a query and have the servos respond to a query and at the same time process other commands such as moves could come in handy…

That is if I could do something like: Serial1.print("#254QV0QT0QC0QS0QP0\r");
Followed by one or more sets of move requests and all of the servos will continue to respond to the posted query while they are doing the move request that could be useful.

Note: In practice I probably would not ask for all of the Q stuff above, especially if we are still needing to use EM=0 type moves. But would probably want some simple things like: Voltage, Temp, error state…

But before I go through the process of incorporating the new stuff, I thought I would first see if the new firmware version improved some of the communication issues we were seeing before.

So I updated all of the servos firmware. Then ran my sketch which updated all of their baud rates to 500000 as figured that was about half the new range mentioned. Note: My guess is I need to reset my logical 0 positions for the servos as I think all of these settings may have been lost when I updated the firmware… I know the baud rate was…

Then using my LSSTestServos sketch with the I command (maybe need to do c command first)

But the i command just cycles moving all 6 legs through like 7 positions:

  {0, -900, -600},       //Low
  {100, -450, -300},      //mid Low
  {200,  0,  0},          //Med
  {0,  450, 450},         //High
  { -100,  0,  0},        //Med
  { -200, -450, -300},    //mid Low
  {0, -900, -600}       //Low

Note: the servos were setup up the GYRE on the different legs are set to handle left versus right leg.

So it just cycles through these moves using different options, like I believe I default to timed moves.
And running this test for a short period of time, I typically have several of the servos go nuts and end up trying to go to strange locations. Typically several servos will error out and get into the blinking state.

So my strong guess is that the serial communication is still unreliable at this speed, which does not totally surprise me as running 18 servos over reasonably long wires, that are daisy chained (in a star sort of configuration) in a noisy environment…

Will try again at lower speeds.

I’m actually experiencing similar issues and doing more tests to narrow it.
This doesn’t come from the last update, I’ve made it happen back to 367.

Not always easy to reproduce but i found that if i’m sending a second movement command in the middle of the last movement it’s conflicting more.

More tests…

Thanks @dialfonzo

Not surprising, I have run into this with all of the versions of the firmware up till now.

I am trying to remember if it was @cmackenzie, that also mentioned earlier on, that he had some percentage (like 10%?) of packets corrupted?

As I mentioned in previous post, My guess is a lot of this is due to running high speed communications over reasonably long unshielded wires, that go through multiple connections, in a electrically and probably RF noisy environment.

Hardware wise, I am not sure what we can do about it? That is assuming that the UARTS in all of the Servos as well as in the Host processor are working very closely to the defined Baud rate… In theory I believe the Teensy should be spot on at 500k. But there is drift and the like in processor timers… So could be off some. Ditto for servos. Maybe servo cables could be shorter, or rerouted or made shielded or…

I am not sure at this point Not sure what I will try next…
May try changing the output of servo movements to be all in one logical command, but not sure if that will help.

So far it looks like the code will need to stay in EM=0 mode, which if you output maybe 50-100 positions per second, maybe having some percentage of them not being received as sent may not be as noticeable…

Still have not tested yet to see if servos can process move commands while waiting to output their results from a Group query command… That is while they are waiting for their slot time to come up, what happens if I send it a move to degrees command?
If they can not happen at same time, then need to see, how much of a query you can do and still output new moves at a high enough speed.

Hope to try some of this soon, but right now off working on other projects.

@dialfonzo @cbenson - Follow on to previous post:

I pushed up another test sketch into my test github project:

It tries to move all 18 servos by small moves, until you hit some character to stop it…

I have the sketch output the moves with just the one trailing CR.

I also have the ability to do query at same time: Serial1.print("#254QV0QT0\r");

The code can be configured to try it before you output the move or after you output the move to see if you can get overlap…

And as you see there is overlap:

But some of the data is being corrupted.

Note: I get some wild moves as mentioned… I left it configured to do query request first to start getting output back while outputting the move command… This is with or without doing the query…

Now back to other project

@kurte - @dialfonzo

Just gave @kure’s new test sketch a shot with the 3 servo setup that I have. With one difference, I am running at 921k for a baud. Was just curious about if the leg servos would go off in strange positions. Well…

Worked for several cycles went 1 or 2 of the servos went to a strange position, it recovered on the next move command ran a few more times went strange position again. After this I shut down the test. I didn’t have a LA attached unfortunately - in the middle of some other testing.

1 Like

@kurte @cyberpalin Appreciate the testing and looking into it.

@kurte Just need some clarification…

Breaking down the commands:

#254SLOTCOUNT18 // are you using all 18 servos?
#1SLOT0
#1G1
#2SLOT1
#2G429496 // What is this intended to do? Gyre is normally associated with -1 or 1.
#3SLOT2
#3G1
#4SLOT3
#4G429494967295 // same issue as above
#5SLOT4
#5G1
#6SLOT5
#6G4294967295 // same issue as above
#7SLOT6
#7G1
#8SLOT7

  • For group commands, the \r is only needed at the end. Might be a possible cause of the issue.
  • If the weird Gyre values are removed, and no \r after each (just one at the end), does that help?
  • Normally slots only need to be assigned once at the start.

@cbenson @dialfonzo @cyberpalin

Sorry that trace is a little misleading here:

The trace is showing the trailing edge of the last part of the code that I call to make sure I have all of the servos found and configured:

void FindServos(void) {

  g_count_servos_found = 0;
  int32_t pos;
  Serial.println("\nSearch for all servos");

  // Initialize to no servos...
  for (int i = 0; i < COUNT_SERVOS; i++) {
    myLSS.setServoID(i);
    pos = myLSS.getPosition();
    if (myLSS.getLastCommStatus() == LSS_CommStatus_ReadSuccess) {
      g_ids[g_count_servos_found++] = i;
      Serial.print("    ");
      Serial.print(i, DEC);
      Serial.print(" - ");
      Serial.println(pos, DEC);
    }
  }
  // Now lets update the slots.
  Serial1.printf("#254SLOTCOUNT%u\r", g_count_servos_found);
  for (int i = 0; i < g_count_servos_found; i++) {
    LSS_SERIAL.printf("#%uSLOT%d\r", g_ids[i], i);
    LSS_SERIAL.printf("#%uG%u\r",  g_ids[i], servo_gyre[g_ids[i]]);

  }
  Serial.println("  Done Slots and gyre updated");
}

The code that actually does the loop, with moving servos and query for voltage and temp:

void MoveAllServos(void) {
  // first move all to center and on
  AllServosCenter();

  static int MIN_SERVO_POS = -300;
  static int MAX_SERVO_POS = 300;
  int servo_angle = 0;
  int servo_increment = 50;

  int voltages[COUNT_SERVOS];
  int temps[COUNT_SERVOS];
  uint32_t move_time = 250;
  Serial.println("Move All servos: Enter any key to exit");
  while (Serial.read() != -1);

  while (!Serial.available()) {
    elapsedMillis em = 0;
    servo_angle += servo_increment;
    if (servo_angle >= MAX_SERVO_POS) servo_increment = -50;
    if (servo_angle <= MIN_SERVO_POS) servo_increment = 50;
#if WHEN_TO_CHEW == 2
    // Lets ask for Voltage and Temp
    Serial1.print("#254QV0QT0\r");
#endif

    for (int j = 0; j < g_count_servos_found; j++) {
      Serial1.printf("#%uD%dT%u", g_ids[j], servo_angle, move_time);
    }
    Serial1.print("\r"); // output terminator to execute the command.
    char cmd_str[10];
    uint8_t responses_remaining = g_count_servos_found * 2;
    uint8_t servo_index = 0;
#if WHEN_TO_CHEW == 1
    // Lets ask for Voltage and Temp
    Serial1.print("#254QV0QT0\r");
#endif
#if WHEN_TO_CHEW > 0
    while (responses_remaining && (em < 2 * move_time)) {
      uint8_t servo_id;
      int16_t servo_value = LSS_Read_Servo_s16(servo_id, cmd_str);
      if (servo_id != g_ids[servo_index]) {
        for (servo_index = 0; servo_index < g_count_servos_found; servo_index++) {
          if (servo_id == g_ids[servo_index]) break;
        }
      }
      if (servo_index < g_count_servos_found) {
        if (strcmp(cmd_str, "QV") == 0) voltages[servo_index] = servo_value;
        else if (strcmp(cmd_str, "QT") == 0) temps[servo_index] = servo_value;
        responses_remaining--;
        servo_index++;
        if (servo_index == g_count_servos_found) servo_index = 0;
      }
    }
#endif
    Serial.printf("servo_angle:%d QT: %u RR:%u\n", servo_angle, (uint32_t)em, responses_remaining);
    while (em < (2 * move_time)) ;
  }
}

Notice that the \r is only output after I write out all of the new positions.

But regardless: It should not matter if I tell each servo one at a time to start a move or tell all of the servos to start at once, I would believe that a goal would be that none of the messages should be garbled.

In the one I circled, my gut tells me there is a firmware issue. If you look at the overview trace, in most cases the blue traces (RX) look like it is responding one servo slot at a time, with both the QV and QT responses for that one servo… But in the circled area, the shorter blue one to the right of the circled only output the QV area and not the QT value. And then next thing corrupted.

My gut tells me that Servo started it’s response, and then became busy, maybe needing to compute and output the changes to the servo, and then it waited until next free time to start the rest of the response, but then the next servo also decided it was it’s time to output…

And I am seeing messages being corrupted or misinterpreted both on TX and RX pins…

RX mentioned above… TX - I am assuming TX error instead of servos on their own moving to wrong locations…

My next quick and dirty on this test case is to report how many of the responses to the commands
Appear to be wrong. Like: did not start with *, did not have a proper servo number, did not have one of the query command strings correct…

Note: the current library has very little support for this. That is you can ask for the next response to returned by calling something like:

	static char * genericRead_Blocking_str(uint8_t id, const char * cmd);

But this code will fail if you pass in the wrong ID or the CMD is not what you passed in… So you have to know the exact order things will be returned and if your off, no real good way to know how to guess then what the next things you should pass in to try to get the next one…

Again note: the sketch is up on the github project. Will update at some time soon with changes I mentioned. Hopefully you should be able to replicate this with your own setup.

Kurt

EDIT: Should mention, that for each run, at the end I reset the servos and the like and scan for them and set up slots for each run, as to hopefully be able to fully recover for each pass through… Hope that makes sense.

@cbenson @dialfonzo @cyberpalin - and all.

I pushed up an updated version of my Walk and Chew gum sketch…

It puts a gap in after searching for servos, setting up the slots and Gyre… Yes could only do this once, but most passes more than one servo end up in error states…

Also put in some code that tries to show some validation of data. Not 100% correct
but it also toggles IO pins.
IO 2 - for stuff reading in string, like characters received before *. Is there a number after the star…

IO 3 - is trying to extract the CMD and number off of the string. Is there a valid command. Like < 10 characters before a number… Then checks to see if a number after this… Probably bugs in this… but does give you cluses.
Then IO4 looks at the command string is it one of the two we asked for…

But does give you clues…

Showing multiple loops through:

Note: not everything shown as error is an error. Some of the RED lines is it is timing out as thought I would receive more responses, and timed out… Could increase this time in my code, but again just hopefully giving you some feedback that you can run locally and see if it is repeatable and if your firmware developer needs to make fixes…

Closer in one loop:

Again one interesting thing is each time you see a shorter output on top blue line, the next group will be bad.
Closeup:

Unless someone has some suggestions, I believe this is about all can do at this point.

Hi KurtE - We have a theory that i tested here.
Making more Query at the same time allows more time for the reply per servos, can you test that out ? (adding more queries)

Here at 921 600 and using your example but added two more queries:

1 Like

Thanks @dialfonzo (and @cbenson ) for trying it out. Will try your change at some point, right now working on other projects, like working extending the File System support on Teensy (FS/File classes) to support Dates and Times…

In cases like this, I keep asking myself what is the proper goal when you find what appears to be a repeatable issue. Often times my answer will be different depending on things like. Is my goal to get the get the project done, or is it to try to solve the problem. Often times it is somewhere in between.

For me, as I have nothing pressing for me to release something, my instinct is to leave the example in a state that the firmware developer can hopefully identify the issue and hopefully find a proper fix. Yes maybe you can add additional stuff to the query and maybe mask the issue, maybe happen with 4 items queried, by may with 5 or … But gut tells me, it may depend on what other things are going on within the Servo firmware at the same time.

And maybe there should be several more examples like this, to test, like convert this slot type requests into a long query string and see if that makes a difference…

But the real question I have to ask myself, how much does the changes in this firmware release apply to what I was trying to do, and as such how much time to spend on it.

That is the main parts of the Hexapod code dealing with servos, is probably 90+% writes and a few percentages doing query.

The one part that I set that applies is that the Servo position (degree) command we should be able to start all of the servos at the same time… I have not verified this yet.

For me, I still have two fundamental issues, which this release does not appear to address at all:

a) Reliable communications: The test sketch is sending out move commands twice per second and it usually does not take more than a few iterations before one or more servos obviously moves wrong. Note this is repeatable without the query involved as well.

This is saying either:

  1. The data is being corrupted, probably due to longer distances, voltage changes, EMI, …
  2. The servo just screws up and maybe forgets something like Gyre or offset or …
    I am guessing 1)

b) EM=1 is not usable for most programs that do any multiple step moves… i.e. the start, coast, stop behavior of each command, there was some interesting posts on this thread about it, but so far it sounds like no changes to address this have been made.

So that leaves us still needing to use EM=0 mode. Which I find frustrating that in this mode if I tell a servo to some position, it may or may not. You may have to tell it something like 3-4 times to actually go there before it believes you…Personally I don’t understand the reason for this, unless it has to do with a workaround for unreliable communications?

So then if your code needs to output 50+ moves per second, which implies a 20ms cycle, how much data can you query during each cycle, or does it make sense to allow the query to span multiple servo cycles…

Kurt

This becomes the priority.

1 Like

Our firmware guru(s) are looking into finding and correcting the issue. More to come.

The new group command is working awesome! I can push the limit to 100Hz control loop with 18 servos!! I am querying position, current (amps) and velocity (PCV), and then sending position. On the humanoid I had to split the 18 servos across 3 separate buses just to get near 30Hz so big improvement!

Here is a logic capture zoomed out with ~11 loop iterations. I am using group command and SLOT timing. The top blue is the PCV query, each red block is all 18 servos responding in sequence.

Here is a zoomed in view of one loop iteration. You can see one blue block which is requesting PCV using #254QD0QC0QS and then another command afterwards setting each servo to Limp (but could be a target position). You then see each servo responding in the red blocks below in their respective slot.

I’m not getting any errors event at 100Hz but I’ve since backed down to 50Hz just because I dont need that high an update rate. I rewrote the Ros2 controls plugin which is why this took so long. Since the servos are doing all the work of timing replies all the complexity of my LSS library is gone (trying to get fast updates) so my cpu usage on the ros2 controls plugin dropped from about 40% down to 1%. Now I just set a 50 or 100Hz timer in software, each tick I send the broadcast and action and read the serial buffers for the previous reply results.

This is on a new firmware release with a fix that was causing some servos to reply to late but it seems stable now.

I’m updating some of my other Ros nodes so hoping to have some actual video to show this week.

3 Likes

That seems to be great news! Really appreciate the thorough testing! We’ve been trying to replicate on our end and suspect that the older generation servos may be causing our issue.

I have the robot moving through direct manipulation of the limbs and base in RViz2, and I have it working with trajectories as well. So I am working in python now to compute walking algorithms and send them to the robot like Trajectory Builder does in this video.

4 Likes

@cmackenzie Excellent to see an update here! Guessing you’ve already informed @madmax ? If not…
Very nice model - looks like the real thing, and don’t see any / much lag between the virtual model and the real robot.
So as you said at the end of the video… on to the walking?

1 Like

Fantastic!! That is amazing… super cool ! I have the c code for walking @cmackenzie that I had used on mine. I am very much tempted to see if that can be ported. I will message you regarding this. I think we should have a catch up soon.

1 Like

a06fe237110e6da70fefe36b99f3c681

@kurte - Ok i know what i did wrong that that time. Last Friday i had another modification to do and update on Arduino library manager. Seems like the last time i didn’t updated the tag inside the library so it was not picked up.

It is now, as well as my change.