CtrlX-Core, Ethercat falls stuck to Init from time to time

[MassimoG]

Goodmorning,

I am reaching out to seek your assistance regarding a recurring technical issue we’ve been experiencing. 

Every day, our customer encounters an error that appears to happen randomly, making it difficult to identify the cause.

The error manifests as Ethercat Master switch to Init and not able to switch to OP any longer.

This happens with no apperent cause. Sometimes, drives standstill no torque, no power and no user input.

At this point, we wish we had more information about devices and master itself, 

Diagnosis is made difficult due to the need to avoid interrupting the client's production.

This made it challenging to gather more data during the error occurrences as long as the customer needs to reboot to start again.

None relevant advisory of problems comes form PLC functionBlocks to diagnose Ethercat master and slaves or from the CtrlX log.

We added SynchUnits in the intent to narrow down the problem with no success.  

To provide some context, 

our architecture consists of 49 nodes ( see Fig 1 ) : mostly RTA Drivers + 3 X Hub Odot + ADF_Web  

CtrlX-Works : 1.20

CtrlX-Core : V1.20

App EthercatMaster : V1.20.1

( See Fig.3 ) 

When possible, we’ve conducted a few tests to gather some clues and disovered that :

• When scanning the network, all the devices are scanned with no problems making exception for the 3 x Odot Hubs.
  This case only Hubs are being detected. No children detected as yellow warning displays.
  ESI files are presents. It takes an additional scan to fetch Odot's children modules.
  This is the reason why our customer at first commissining manually added every and each device on the CtrlX-IO configuration.
  Doing it this way, Ethercat Master takes more than 1 minute to switch to OP
• Things are different when we remove the last device ( our case Tastiera 3 - en => keyboard ).
  Odot's children scan works at first shot.
  Switching to OP was much faster ( about 30 sec. ) despite up to 2 minutes for the last case.
  This is very reassuring as the problem hasn't shown up for 2 days now.
  The bad part is that "Tastiera 3" is required. So this can only be a temporary solution. 

We have planned to : 

• Make some tests to pinpoint the root of the problem by switching positions and replacing devices.
• Prepare a backup CtrlX-Core updated to 1.20.13 to replace the actual one limitating the impact on production.
• Collect more informations as long as we've enabled Ethercat tracing for all severities. ( Fig .2 )  

Despite these efforts, it is not easy investigate as long as the machine is in production and each suggested test is deemed irrelevant.

I would greatly appreciate your opinions and any suggestions you might have on how to further investigate or resolve this issue.

In attachment, 

230020_CtrlX_Master.zip for both EthercatI/O and PLC Project
ODOT.zip for ESI file and Odot Hub documentation

Thank you for your time and assistance. I look forward to your insights!

Best regards,

Massimo Grattieri

[cc2go · Accepted solution]

Hello Massimo, 
to identify the "bad" one the Slave Diag Info and Slave Statistics might help to get more info. How many EtherCAT Master errors are counted?
Since you identified one EC Slave, do you contacted this manufacture? And may a EtherCAT capture might help then.

What you can try too, to configure the "bad" one to an own sync unit and check the Slave Diag Info again. 

[Dias · Accepted solution]

Hello Massimo,

this Functionblock can be helpful for analysing which slave has a problem and make an automatic switching to OP after fixing the problem:

Get Ethercat diagnosis from ctrlX CORE into PLC (boschrexroth.com)

[MassimoG]

Thank a lot for your information.
Our customer has not had the chance to go on field and putting into practice your advices due to production needs. 
Regards,
Massimo