Lynxmotion SES V2 Hexapod Robot

@cbenson @dialfonzo @cyberpalin

Sorry that trace is a little misleading here:

The trace is showing the trailing edge of the last part of the code that I call to make sure I have all of the servos found and configured:

void FindServos(void) {

  g_count_servos_found = 0;
  int32_t pos;
  Serial.println("\nSearch for all servos");

  // Initialize to no servos...
  for (int i = 0; i < COUNT_SERVOS; i++) {
    myLSS.setServoID(i);
    pos = myLSS.getPosition();
    if (myLSS.getLastCommStatus() == LSS_CommStatus_ReadSuccess) {
      g_ids[g_count_servos_found++] = i;
      Serial.print("    ");
      Serial.print(i, DEC);
      Serial.print(" - ");
      Serial.println(pos, DEC);
    }
  }
  // Now lets update the slots.
  Serial1.printf("#254SLOTCOUNT%u\r", g_count_servos_found);
  for (int i = 0; i < g_count_servos_found; i++) {
    LSS_SERIAL.printf("#%uSLOT%d\r", g_ids[i], i);
    LSS_SERIAL.printf("#%uG%u\r",  g_ids[i], servo_gyre[g_ids[i]]);

  }
  Serial.println("  Done Slots and gyre updated");
}

The code that actually does the loop, with moving servos and query for voltage and temp:

void MoveAllServos(void) {
  // first move all to center and on
  AllServosCenter();

  static int MIN_SERVO_POS = -300;
  static int MAX_SERVO_POS = 300;
  int servo_angle = 0;
  int servo_increment = 50;

  int voltages[COUNT_SERVOS];
  int temps[COUNT_SERVOS];
  uint32_t move_time = 250;
  Serial.println("Move All servos: Enter any key to exit");
  while (Serial.read() != -1);

  while (!Serial.available()) {
    elapsedMillis em = 0;
    servo_angle += servo_increment;
    if (servo_angle >= MAX_SERVO_POS) servo_increment = -50;
    if (servo_angle <= MIN_SERVO_POS) servo_increment = 50;
#if WHEN_TO_CHEW == 2
    // Lets ask for Voltage and Temp
    Serial1.print("#254QV0QT0\r");
#endif

    for (int j = 0; j < g_count_servos_found; j++) {
      Serial1.printf("#%uD%dT%u", g_ids[j], servo_angle, move_time);
    }
    Serial1.print("\r"); // output terminator to execute the command.
    char cmd_str[10];
    uint8_t responses_remaining = g_count_servos_found * 2;
    uint8_t servo_index = 0;
#if WHEN_TO_CHEW == 1
    // Lets ask for Voltage and Temp
    Serial1.print("#254QV0QT0\r");
#endif
#if WHEN_TO_CHEW > 0
    while (responses_remaining && (em < 2 * move_time)) {
      uint8_t servo_id;
      int16_t servo_value = LSS_Read_Servo_s16(servo_id, cmd_str);
      if (servo_id != g_ids[servo_index]) {
        for (servo_index = 0; servo_index < g_count_servos_found; servo_index++) {
          if (servo_id == g_ids[servo_index]) break;
        }
      }
      if (servo_index < g_count_servos_found) {
        if (strcmp(cmd_str, "QV") == 0) voltages[servo_index] = servo_value;
        else if (strcmp(cmd_str, "QT") == 0) temps[servo_index] = servo_value;
        responses_remaining--;
        servo_index++;
        if (servo_index == g_count_servos_found) servo_index = 0;
      }
    }
#endif
    Serial.printf("servo_angle:%d QT: %u RR:%u\n", servo_angle, (uint32_t)em, responses_remaining);
    while (em < (2 * move_time)) ;
  }
}

Notice that the \r is only output after I write out all of the new positions.

But regardless: It should not matter if I tell each servo one at a time to start a move or tell all of the servos to start at once, I would believe that a goal would be that none of the messages should be garbled.

In the one I circled, my gut tells me there is a firmware issue. If you look at the overview trace, in most cases the blue traces (RX) look like it is responding one servo slot at a time, with both the QV and QT responses for that one servo… But in the circled area, the shorter blue one to the right of the circled only output the QV area and not the QT value. And then next thing corrupted.

My gut tells me that Servo started it’s response, and then became busy, maybe needing to compute and output the changes to the servo, and then it waited until next free time to start the rest of the response, but then the next servo also decided it was it’s time to output…

And I am seeing messages being corrupted or misinterpreted both on TX and RX pins…

RX mentioned above… TX - I am assuming TX error instead of servos on their own moving to wrong locations…

My next quick and dirty on this test case is to report how many of the responses to the commands
Appear to be wrong. Like: did not start with *, did not have a proper servo number, did not have one of the query command strings correct…

Note: the current library has very little support for this. That is you can ask for the next response to returned by calling something like:

	static char * genericRead_Blocking_str(uint8_t id, const char * cmd);

But this code will fail if you pass in the wrong ID or the CMD is not what you passed in… So you have to know the exact order things will be returned and if your off, no real good way to know how to guess then what the next things you should pass in to try to get the next one…

Again note: the sketch is up on the github project. Will update at some time soon with changes I mentioned. Hopefully you should be able to replicate this with your own setup.

Kurt

EDIT: Should mention, that for each run, at the end I reset the servos and the like and scan for them and set up slots for each run, as to hopefully be able to fully recover for each pass through… Hope that makes sense.

@cbenson @dialfonzo @cyberpalin - and all.

I pushed up an updated version of my Walk and Chew gum sketch…

It puts a gap in after searching for servos, setting up the slots and Gyre… Yes could only do this once, but most passes more than one servo end up in error states…

Also put in some code that tries to show some validation of data. Not 100% correct
but it also toggles IO pins.
IO 2 - for stuff reading in string, like characters received before *. Is there a number after the star…

IO 3 - is trying to extract the CMD and number off of the string. Is there a valid command. Like < 10 characters before a number… Then checks to see if a number after this… Probably bugs in this… but does give you cluses.
Then IO4 looks at the command string is it one of the two we asked for…

But does give you clues…

Showing multiple loops through:

Note: not everything shown as error is an error. Some of the RED lines is it is timing out as thought I would receive more responses, and timed out… Could increase this time in my code, but again just hopefully giving you some feedback that you can run locally and see if it is repeatable and if your firmware developer needs to make fixes…

Closer in one loop:

Again one interesting thing is each time you see a shorter output on top blue line, the next group will be bad.
Closeup:

Unless someone has some suggestions, I believe this is about all can do at this point.

Hi KurtE - We have a theory that i tested here.
Making more Query at the same time allows more time for the reply per servos, can you test that out ? (adding more queries)

Here at 921 600 and using your example but added two more queries:

1 Like

Thanks @dialfonzo (and @cbenson ) for trying it out. Will try your change at some point, right now working on other projects, like working extending the File System support on Teensy (FS/File classes) to support Dates and Times…

In cases like this, I keep asking myself what is the proper goal when you find what appears to be a repeatable issue. Often times my answer will be different depending on things like. Is my goal to get the get the project done, or is it to try to solve the problem. Often times it is somewhere in between.

For me, as I have nothing pressing for me to release something, my instinct is to leave the example in a state that the firmware developer can hopefully identify the issue and hopefully find a proper fix. Yes maybe you can add additional stuff to the query and maybe mask the issue, maybe happen with 4 items queried, by may with 5 or … But gut tells me, it may depend on what other things are going on within the Servo firmware at the same time.

And maybe there should be several more examples like this, to test, like convert this slot type requests into a long query string and see if that makes a difference…

But the real question I have to ask myself, how much does the changes in this firmware release apply to what I was trying to do, and as such how much time to spend on it.

That is the main parts of the Hexapod code dealing with servos, is probably 90+% writes and a few percentages doing query.

The one part that I set that applies is that the Servo position (degree) command we should be able to start all of the servos at the same time… I have not verified this yet.

For me, I still have two fundamental issues, which this release does not appear to address at all:

a) Reliable communications: The test sketch is sending out move commands twice per second and it usually does not take more than a few iterations before one or more servos obviously moves wrong. Note this is repeatable without the query involved as well.

This is saying either:

  1. The data is being corrupted, probably due to longer distances, voltage changes, EMI, …
  2. The servo just screws up and maybe forgets something like Gyre or offset or …
    I am guessing 1)

b) EM=1 is not usable for most programs that do any multiple step moves… i.e. the start, coast, stop behavior of each command, there was some interesting posts on this thread about it, but so far it sounds like no changes to address this have been made.

So that leaves us still needing to use EM=0 mode. Which I find frustrating that in this mode if I tell a servo to some position, it may or may not. You may have to tell it something like 3-4 times to actually go there before it believes you…Personally I don’t understand the reason for this, unless it has to do with a workaround for unreliable communications?

So then if your code needs to output 50+ moves per second, which implies a 20ms cycle, how much data can you query during each cycle, or does it make sense to allow the query to span multiple servo cycles…

Kurt

This becomes the priority.

1 Like

Our firmware guru(s) are looking into finding and correcting the issue. More to come.

The new group command is working awesome! I can push the limit to 100Hz control loop with 18 servos!! I am querying position, current (amps) and velocity (PCV), and then sending position. On the humanoid I had to split the 18 servos across 3 separate buses just to get near 30Hz so big improvement!

Here is a logic capture zoomed out with ~11 loop iterations. I am using group command and SLOT timing. The top blue is the PCV query, each red block is all 18 servos responding in sequence.

Here is a zoomed in view of one loop iteration. You can see one blue block which is requesting PCV using #254QD0QC0QS and then another command afterwards setting each servo to Limp (but could be a target position). You then see each servo responding in the red blocks below in their respective slot.

I’m not getting any errors event at 100Hz but I’ve since backed down to 50Hz just because I dont need that high an update rate. I rewrote the Ros2 controls plugin which is why this took so long. Since the servos are doing all the work of timing replies all the complexity of my LSS library is gone (trying to get fast updates) so my cpu usage on the ros2 controls plugin dropped from about 40% down to 1%. Now I just set a 50 or 100Hz timer in software, each tick I send the broadcast and action and read the serial buffers for the previous reply results.

This is on a new firmware release with a fix that was causing some servos to reply to late but it seems stable now.

I’m updating some of my other Ros nodes so hoping to have some actual video to show this week.

3 Likes

That seems to be great news! Really appreciate the thorough testing! We’ve been trying to replicate on our end and suspect that the older generation servos may be causing our issue.

I have the robot moving through direct manipulation of the limbs and base in RViz2, and I have it working with trajectories as well. So I am working in python now to compute walking algorithms and send them to the robot like Trajectory Builder does in this video.

4 Likes

@cmackenzie Excellent to see an update here! Guessing you’ve already informed @madmax ? If not…
Very nice model - looks like the real thing, and don’t see any / much lag between the virtual model and the real robot.
So as you said at the end of the video… on to the walking?

1 Like

Fantastic!! That is amazing… super cool ! I have the c code for walking @cmackenzie that I had used on mine. I am very much tempted to see if that can be ported. I will message you regarding this. I think we should have a catch up soon.

1 Like

a06fe237110e6da70fefe36b99f3c681

@kurte - Ok i know what i did wrong that that time. Last Friday i had another modification to do and update on Arduino library manager. Seems like the last time i didn’t updated the tag inside the library so it was not picked up.

It is now, as well as my change.

Sure is quiet up here ( :cricket:)

I hope everyone is doing well and that you all have a nice (safe) Holidays!

2 Likes

@kurte - It is certainly quite here.

Hope you are all safe and with that new Variant around… stay safe.

2 Likes

Ditto - happy holidays and stay safe :slight_smile:

3 Likes

Hello All,

Hope everyone is keeping safe. Wishing you all a very happy new year!!

3 Likes

Happy Holidays & New Year! @cmackenzie has been advancing with ROS 2 and has/will involve @madmax. The latest firmware update (under Experimental) seems to be working well. Hopefully video updates will be posted soon.

1 Like

A lot of code changes lately and the walking is looking much smoother! Yes, smooth but slow…I’ll turn the speed up soon enough.

3 Likes

Happy New Year all, hope you are all doing well and healthy.

Any updates on this?

Was a guru able to fix the packet reliability issue?

Nice work on the ROS2 stuff.

Are you using any of the timed moves and the like or are you sending out servo packets as fast as possible?

Again happy new year.