CORE X3 Not IP Forwarding

[AutomateSHANE]

I have been unable to access the web interface of an EtherCAT slave device connected to the ECAT master of X3 using EoE. This has worked before, so I know everything was set up correctly.

As a sanity check, I decided to see if could forward between the two Ethernet interfaces of the X3 (XF10 and XF51). Both are set to static, unique subnets, and IP forwarding is enabled. Gateway addresses are properly assigned. No matter how I set up the routing in Windows, I cannot get all the way through to a node on the other subnet. At best, I can make the first hop (through XF10) and ping the other interface (XF51). Via the CORE web interface, I can ping the device connected to XF51. For some reason, it won't route all the way through.

PC at 179.254.10.250 --> XF10 @ 179.254.10.1 --> XF51 @ 10.3.5.100 --> Device @ 10.3.5.1

FWIW, I have already been in contact with the top BR engineer in my region about this. He replicated my setup and does not have any issues. We are both scratching our heads. Given that I have encountered another strange issue this week, I'm left to assume that my control is bugged, following the last 1.20 app package update. Any ideas before I reimage and start fresh?

[CodeShepherd]

I assume the ctrlX OS app versions of both devices are the same. As a test a backup could be created of the working one and transferred to the not working one to be sure settings are really identical. 

There could be routing settings different between the two setups. Were both ctrlX CORE tested on different PCs? to make sure there are no different settings.

Are any apps like firewall installed, that could interfere the network communication?

[AutomateSHANE]

Hi @CodeShepherd, I think you misunderstood. I am only talking about one ctrlX CORE. The CORE will not forward traffic between its ports, despite being configured to do so. I want to access an internal web interface of an EtherCAT slave device, using EoE. That looks like:

PC --> CORE XF10 --> XF50 --> ECAT Slave

This wasn't working, so I thought to first check if I can route through the two Ethernet interfaces. I tried:

PC --> XF10 --> XF51 --> Test device

That doesn't work either. Again, one of your folks already tested this exact setup with his CORE and could not reproduce the issue. It works fine for him, but not me. I don't recall having to change any firewall settings the last time I did this.

[Sgilk]

Hi @AutomateSHANE ,

Is the gateway address on Device @ 10.3.5.1 set to XF51 @ 10.3.5.100? If it is not, you will need a SNAT rule configured in the firewall application. 

See this thread for a similar discussion.

[AutomateSHANE]

Hi @Sgilk,

Yes, the gateway address is set @ 10.3.5.100. It still does not work.

I do not have the firewall app installed on ctrlX. I don't believe it is necessary.

[Sgilk]

@AutomateSHANE ,

You're right that firewall is not necessary in this case, unless you don't want to set routing rules on your PC or gateways on the XF51 devices. You could use DNAT and SNAT respectively to avoid this.

I just verified connectivity using the setup you provided above (with different addresses)

PC at 192.168.1.6 --> XF10 @ 192.168.1.100 --> XF51 @ 192.168.2.1 --> Device @ 192.168.2.99

1. Route rule on PC 

   route add 192.168.2.0 mask 255.255.255.0 192.168.1.100​

2. Gateway on device set to XF51 address (192.168.2.1)
3. IP forwarding enabled on XF10 & XF51

[AutomateSHANE]

@Sgilk ,

 I don't want to use DNAT and SNAT if it is not necessary. I want the setup to be as simple and easy to document as possible so that the rest of my team can do it as needed. You have verified that everything I have done should work. You're the second one, in fact. But....it doesn't work.

I had hoped that somebody would want to take a deeper look into this to understand why it is not working. If there is a problem with my X3 or the installed software, it would be good to know. At this point, I only know of 2 options going forward:

1. Re-flash a clean system image to the CORE and take the time to re-install all apps and settings, then hope for the best.
2. Take the CORE to the nearest church and submerge it in holy water to cure it of any demons within.

I will try those, in that order, but I really don't want to do either.

[Sgilk]

@AutomateSHANE ,

The reason I mention the firewall rules is because you can transfer the configuration on the CORE and avoid setting routing rules on each PC you want to network.

1.  Can you do a route print on the PC you are attempting to route through the CORE to verify the route has been created properly? 

   route print​

2. Can you do a route trace on a message from PC to XF51 device? Ex. for device at 192.168.2.99 below. 

   tracert 192.168.2.99​

[AutomateSHANE]

@Sgilk ,

Yes, I have done this enough times to legally be declared insane.

[Sgilk]

@AutomateSHANE ,

Could you possibly try this with a different device internal to XF51? The route looks correct. If you can ping XF51, I believe the problem is likely with the gateway configuration on the internal device.

You could also try and add a SNAT firewall rule, to confirm.

[AutomateSHANE]

I've tried it now on not just 1, but 2 other devices. That makes 3 total devices, not counting the ECAT slave. I don't think the gateway configuration is the problem.

I don't have a license for the firewall app, so I cannot add a SNAT rule.

[Sgilk]

As a test, could you please swap the device and XF51 IP addresses? (XF51 10.3.5.1, device 10.3.5.100)

[AutomateSHANE]

Addresses swapped. Tested with two devices. Nothing new. Still doesn't work.

[Sgilk]

If you want to do a factory reset like you mentioned already, that wouldn't hurt. You can take a backup so you don't need to reinstall and reconfigure all of your apps.

If you want to send screenshots of your network interface configuration on the ctrlX CORE, I could review those as well. Nothing immediately stands out as incorrect with your setup as described.

Do you have SSH access on this ctrlX CORE? That opens up some troubleshooting possibilities.

[AutomateSHANE]

I activated SSH.

[Sgilk]

In addition to activating SSH, you need to upload a user assertion. See this how-to on the topic.

One other thought I had is that your PC could be actively denying the return packets from a different subnet. Is it possible you have some antivirus or firewall rules preventing the response? Could be worth testing another device on the request side.

[AutomateSHANE]

It's not because of my PC. I had already ruled that out once, but I just did it again. Using a PC that isn't on a domain, and with Windows firewall turned off, I set up the addresses and routing and...no change. It too can see the other interface of the CORE, but not the device on that subnet. 

[AutomateSHANE]

@Sgilk ,

After a lengthy process, I finally have a user assertion. It is uploaded and SSH active. What now?

[Sgilk]

Hi @AutomateSHANE ,

Can you please list your system app versions? I just booted up a different CORE than the one I had been testing originally and seem to be experiencing the same issue. The CORE experiencing the issue is using release XCR-V-0120.12, while the CORE I originally tested on is using release XCR-V-0120.4.

I reviewed the release notes and there is mention of a networking related vulnerability patch in this version. I am wondering if this could be the culprit.

Bug 749665: Vulnerability HTTP/2 Rapid Reset closed
Description
The HTTP/2 Rapid Reset vulnerability allows an attacker to carry out a "Denial of Service" attack. For further details, please refer to CVE-2023-44487.

Bugfix
The affected components have been updated and the vulnerability has been eliminated.

[AutomateSHANE]

@Sgilk ,

Unfortunately, it is too late for that. I had done a factory reset on my CORE because I couldn't waste any more time. After the reset, I restored from my backup but the problem still persisted. I reflashed, using the 1.20.12 systemimage and re-installed all the apps using the latest downloadable apps package. Forwarding has worked just as it should since then. If I had to make an educated guess on which version I was at previously, I would say it was the same as the one you originally tested, XCR-V-0120.4