Sporadic "Ipo jump detected" errors

[bschmidt]

Hello,

My customer has a machine with CORE X7/ctrlX WORKS 2.02 with 28 real axes and 2 virtual axes. Most of the axes are using MB_GearInVel. Generally it is working ok, but sometimes they have a strange issue when they are starting the machine. The startup sequence involves:

1. MC_Power.Enable = TRUE (drive status Ab-->AF, PLCOpen state DISABLED-->STANDSTILL)

2. When MC_Power.Status = TRUE, then MB_GearInVel.Execute = TRUE (PLCOpen state DISABLED-->SYNCHRONIZED_MOTION)

Usually this works fine and they can run the machine. But occasionally they will get errors on random axes. The error is usually "Ipo jump detected." See below:

Then they Reset and try again and the error(s) moves to different axes. This will keep happening - the error moves around to different axes. When power is cycled on the X7 everthing is working again. 

To isolate the issue we put a delay between MC_Power and MB_GearInVel. In that case we are getting STANDSTILL_PENDING on some axes. So MC_Power does not see drive going to AF. I guess it's the same root cause as the other error. It seems there is some disconnect or timing problem between Motion and EtherCAT even though everything is looking fine in the Scheduler and ctrlX Dashboard.

I will try to get a diagnostic log with "Developer view" enabled the next time this happens.

Best regards,

Brian

[bschmidt]

Here is another screenshot of a typical error that is seen on one or more axes and moves around. Attached is the diagnostic log.

[CodeShepherd]

By checking the log files we found that first an axis warning is triggered:
091F2006, 0C560102, Axis limit exceeded, Deceleration limit exceeded
Then an error:
9,10E+01, 0C560102, Commanded dynamic parameters exceed axis limits, Deceleration limit exceeded, AddInfo: limit: 5.000 m/s^2, commanded: 10.000 m/s^2

Afterwards it seems a command is executed that is not allowed in that state. Please check your programm to prevent commanding violating the state machine:
- General axis command error, Invalid state, AddInfo: cmd: AxsCmdGenericProxy, axsState: DISABLED
- General axis command error, Invalid state, AddInfo: cmd: Power, axsState: SYNCHRONIZED_MOTION

And at last some seconds later the mentioned error is shown:
091F2002, 0C570100, General axis command error, Invalid state

As a response you could get the ipo jump error what we will have a look for:
091F2040, 0C560270, System error in the Motion system, Internal error occurred, AddInfo: Diagnosis codes have not been set

If it is possible to reproduce the problem please switch on motion traces and create a system report in case of the error (see how to).

[bschmidt]

Hello,

Thank you for the analysis. I will have them increase the deceleration limit to make sure that is not causing problems. Today the customer gave me a System Report (attached) but I think he only enabled the traces after the faults occurred. I think the traces should be enabled before the fault occurs.

I notice something else in the log files. There are warnings that licenses related to Motion failed to be acquired, even though these licenses exist. I'm not sure if this is a real problem or just a nuisance message. As far as I know they are not getting errors about missing licenses. 

Best regards,

Brian