HMI freezes after Update

[rexxon]

Hello everyone,

I wanted to raise some issues that I have been experiencing with the WebIQ software on the ctrlX after I updated from version 2.11.3 to 2.13. Specifically, I have the following problems:

1. The HMI is freezing and becoming unresponsive and inputting data on the HMI is no longer possible.
2. The timing of this issue cannot be determined as it occurs unpredictably.
3. Additionally, the WebIQ Runtime Manager cannot be accessed and does not load after the HMI has frozen.
4. Restarting the ctrlX seems to resolve the issue temporarily, but it keeps reoccurring at unpredictable intervals.

Furthermore, I have noticed in the Log (attached to this post) that there seems to be an issue with an incompatible data type (rPEQReadCurrent), which is a normal REAL data type that I have deleted and re-added to the project multiple times. Other variables of the same data type do not seem to be experiencing this issue. This issuesalso only seems to occur from time to time and not regularly? 

I would greatly appreciate any insights or suggestions on how to resolve these issues. Thank you in advance for your help!

Cheers, Sam

Ps. Is there any other way to start WebIQ Projects after an update, rather from the web-backend in the designer version 2.13?

[webiq-sk]

[2023-03-16 13:23:09.822015] [notification | ctrlX] Attempting to reconnect to 'opc.tcp://ctrlx-CORE:4840'.
[2023-03-16 13:23:09.834762] [ error | ctrlX] Connection error.
[2023-03-16 13:23:09.834762] [ error | ctrlX] service_type(): OpenSecureChannel
[2023-03-16 13:23:09.834762] [ error | ctrlX] error(): BadCommunicationError
[2023-03-16 13:23:09.834762] [ error | ctrlX] client_side_error(): no

This shows the following:
1. WebIQ cannot connect to the ctrlX OPC-UA server on port 4840 and is returned a BadCommunicationError (which is a standard OPC-UA error code).
2. Furthermore the error message shows that this is not a client-side error, i.e. is is caused by the OPC-UA server and not by WebIQ according to the OPC-UA stack - which error is logged in the OPC-UA server log at the exact same time?

I assume by "freezing" you mean that all WebIQ input widgets will be intentionally locked when there is an error with the item - like this one where the OPC-UA connection is not possible or breaks. That is a feature and not a bug, because it prevents you from changing values that would not be able to be transmitted anyway.

The underlying issue here is obviously the OPC-UA connection to the ctrlX OPC-UA server.

Questions:
1. What happens if you reload the URL of the HMI in the browser?
2. Which error exactly are you getting (->screenshot) when you are unable to access the Runtime Manager?

Regarding the other questions: please create separate posts rather then including multiple issues and questions in a single post in the future as this makes it easier for other users to find answers to their questions.

A1:
[2023-03-16 13:07:35.329682] [ error | data-recorder] Exception while attempting to process value.
[2023-03-16 13:07:35.329682] [ error | data-recorder] item(): rPEQReadCurrent
[2023-03-16 13:07:35.329682] [ error | data-recorder] what(): Attempting to record item value with incompatible type.
--> To analyze this we require the HMI and a screenshot of UA Expert showing all details of that item in the PLC

A2: Starting projects
You can start and stop projects also through the Runtime Manager since WebIQ 2.13

[rexxon]

Hey, many thanks for the quick response! You are right, by "freezing" I mean, that the HMI Items are kinda locked. No Iputs are accepted and Outputs are not updated. I will focus on the main question in the following:

"1. What happens if you reload the URL of the HMI in the browser?"

-> The URL of the HMI is not available and the browser keeps loading. This happens on the HMI PC as well as on another PC.

"2. Which error exactly are you getting (->screenshot) when you are unable to access the Runtime Manager?"

-> Also the webpage of the Runtime Manager just keeps loading. The rest of the ctrlX works just fine. I can even control the inputs through the PLC Engineering Tool.

I added the second questions regarding the REAL variable, because I thought, this might be connected to the freezing issue.

Do you have any idea, how to fix this issue?

Cheers, Sam

[webiq-sk]

Thank you for answering the questions. Here are a few more:

1. Are you using the PLC Engineering Tool from the same system where the browser is running on which shows the error?

2. What is the URL of the HMI and of the Runtime Manager you are using?

3. Which error is logged in the OPC-UA server log when the connection issue occurs? (asked before)

4. Does the issue also occur if you temporarily disable a firewall and antivirus software on the system you are accessing it from?

[rexxon]

Hey again,

"1. Are you using the PLC Engineering Tool from the same system where the browser is running on which shows the error?"

-> Usually not. The PLC Engineering Tool runs on a Windows-PC and the HMI runs on a Unix system. After the freezing, I tried to connect to the HMI with both of them and also a Android Smartphone, butneither of them worked. 

"2. What is the URL of the HMI and of the Runtime Manager you are using?"

HMI: [ctrlX-URL]:10123/projekt_name/ and Runtime Manager: [ctrlX-URL]:10123/.webui/

"3. Which error is logged in the OPC-UA server log when the connection issue occurs? (asked before)"

So far, I only get the following warning: "OPC UA Server warning User has not the required permission OPC UA StatusCode: 0x80210000". But after this warning is shown, the HMI still works fine. Today I rediscovered the issue after about 1,5 h after starting the ctrlX and HMI. During that time, I’ve updated the PLC program, successfully checked a few times if the HMI is still running but suddenly it stopped accepting inputs. It still updates the items and alarms on the HMI though and shows no new warnings in the ctrlX OPC UA Log. Furthermore, although the items are still updated, I can not reload the HMI URL, images are not loaded after switching tabs on the HMI, the Designer cannot connect to the server and the WEBUI also timeouts while loading.

"4. Does the issue also occur if you temporarily disable a firewall and antivirus software on the system you are accessing it from?"

Unfortunately, not.

[webiq-sk]

1. Can you please specify exactly whether you can connect with the PLC engineering tool to ctrlX at the same moment you cannot connect to WebIQ anymore from an Android phone?
Also, are you using a wired connection from the engineering PC to ctrlX, but a WiFi connection to your smartphone? Might a breaking WiFi connection simply be the reason?

2. Can you please specify the full URLs?

3. Do you see any error on the browser console (F12) when the issue with updating items occurs before you try to reload the HMI and it fails?

4. If I understand you correctly the issue does NOT occur when you disable all firewall and antivirus software on your system?

[rexxon]

"1. Can you please specify exactly whether you can connect with the PLC engineering tool to ctrlX at the same moment you cannot connect to WebIQ anymore from an Android phone?
Also, are you using a wired connection from the engineering PC to ctrlX, but a WiFi connection to your smartphone? Might a breaking WiFi connection simply be the reason?"

 I can connect to PLC Engineering Tool while I can’t access (or rather reloading) the HMI or the WebIQ Runtime Manager. Connecting to the ctrlX Web-Backend while I can’t access the HMI or the WebIQ Runtime Manager is also possible. I am using a wireless connection for the PLC Engineering Tool, wired connection on the HMI PC and of course wireless with the smartphone. All of these devices can’t (re)load the HMI or change any variables while its frozen.

"2. Can you please specify the full URLs?"

 HMI: http://192.168.177.100:10123/peq_test/ and Runtime Manager: http://192.168.177.100:10123/.webui/

"3. Do you see any error on the browser console (F12) when the issue with updating items occurs before you try to reload the HMI and it fails?"

Attached you can find the console log, after the HMI became unresponsive. I updated the HMI around 9:30 this morning and checked every hour without updating the HMI or PLC. It froze again and if you look up the timestamps in the console log, it might have happened around 13:15? Nothing gets added to the log, if I try to update an item after the HMI is frozen.

"4. If I understand you correctly the issue does NOT occur when you disable all firewall and antivirus software on your system?"

Please excuse me if I did not express myself correctly earlier. Deactivating the firewall does not change the fact that the variables cannot be changed after the HMI is frozen.

[webiq-sk]

Thank you for providing the information. You initially wrote this: "Restarting the ctrlX seems to resolve the issue temporarily, but it keeps reoccurring at unpredictable intervals."

I can't help you with ctrlX management, but I assume there is some kind of system and process manager which allows you to see which processes are running and if there is enough space on the disk. Can you please observe the system when the error occurs if you see any issues, RAM and CPU usage there?
-> #1 does the RAM, disk (free space and usage) and CPU load increase during the issue?

The issue with the "hanging browser" when trying to access the HMI when the issue occurs is not known to us, but it can be caused by proxy servers or other security software which intercepts, forwards and manipulates requests. Is there any proxy on your network? That might even be a transparent proxy.
-> #2 is there a proxy on your network?

Another question: does it happen with a single HMI only? Does the error also occur with simply HMIs like our demo HMI? That would be very interesting to know to narrow down on the issue.
-> #3 Does it happen with a very simple HMI that does not use OPC-UA as well?
-> #4 If it does not happen with a very simple HMI does it happen with a simple HMI that uses a few items from the same OPC-UA server?

[rexxon]

Many thanks for hanging in with me!

“#1 does the RAM, disk (free space and usage) and CPU load increase during the issue?”

As far as I can see, there is no unusual load in the system. Here you can find two screenshots from the ressources tab in the ctrlX settings after the issue has occoured:

“#2 is there a proxy on your network?”

No, there is no proxy in our network. The issue also persists for a wired connection directly to the ctrlX.

“#3 Does it happen with a very simple HMI that does not use OPC-UA as well?”

SUprisingly yes. I just checked it today for your “Design Demo” from your website. On the “Design Demo” I am still able to switch the views from the menu but the gauges have stopped working:

“#4 If it does not happen with a very simple HMI does it happen with a simple HMI that uses a few items from the same OPC-UA server?”

I have nevertheless checked it for a very simpe HMI with only 6 variables. Same Issue:-/

The /.webui/ link also stopped working as usual.

[webiq-sk]

Thanks for testing!

What we can deduct from that:

1. The (Websocket) connection between WebIQ Server and the HMI does not break directly (otherwise you would see a corresponding message)
2. As the Design Demo also shows this issue which only uses internal items and no OPC-UA this means that there seems to be an issue with the communication between your webbrowser and the HMI.

As this interestingly also happens from the smartphone I would usually assume it's a misconfigured transparent proxy on the network (we had this issue recently where a user's proxy did not handle Websocket connections properly - but this led to a breaking on the connection after 10 seconds) - or it is a yet unknown issue with WebIQ Server itself.

Let's dig a little deeper:
1. When the issue occurs please execute this command on the Windows console of an affected system: "netstat -an" and send the output
2. Please also execute this command: ping -n 25 192.168.177.100 and send the output
3. Please execute the command: pathping 192.16.177.100 and send the output
4. Please check the Windows Event Viewer for any issues at the time the issue occurs:

• Press the Windows key + R on your keyboard to open the run window
• In the run dialog box, type in eventvwr and click OK
• In the Event Viewer window, expand the Windows Logs menu
• Under the Windows Logs menu, you'll notice different categories of event logs—application, security, setup, system, and forwarded events
• Click on one of the event logs to check and view the events recorded under it

Please provide screenshots of any errors happening at that time.

[rexxon]

I have carried out all the commands you asked for and actually received a list of 12 entries in pathping. For data protection reasons, I have sent this to you in a DM.

Ping is <1 ms ("Reply from 192.168.177.100: Bytes=32 Time<1ms TTL=64") and no packet loss. The event viewer also shows no relevant errors or warnings.
Today I was able to narrow down the time of the freeze to about 5 minutes. Previously I was working in Designer without any problems and after a short phone call nothing worked again...

I hope this information helps. Otherwise, I'm always happy to provide more information. I am very curious to find out what the problem is in the end. Before the update, everything was still fine...

One more thing just came to my mind: When I loaded the HMI from the previous WebIQ version onto the updated ctrlX, all numerical displays suddenly had the maximum number of decimal places. Previously, I always had the number wrongly set to "auto", but perhaps it is still a relevant indication for the solution?

[webiq-sk]

Update: we are currently investigating this issue with Bosch Rexroth.

[Olivier_G]

Dear Gents, 

Can you give us some update regarding this issue? 
I currently have a the problem in a packing line and the production is stopped every 3 hours.
Could we downgrade to the version 2.11.3 (ctrlX version 1.20) and keep the application made in WebIQ version 2.13?

Regards.

[webiq-sk]

We are still investigating the cause of the issue and are in close contact with Bosch Rexroth on that.

It however looks like the issue somewhat occurs connected to a specific ctrlX version in connection with WebIQ. We haven't tested whether the issue also occurs with WebIQ 2.11.3 on the same ctrlX version so the issue might occur there as well.

You can neither downgrade an HMI nor can you load a WebIQ 2.13 HMI in WebIQ 2.11.3.

Which ctrlX software version are you using exactly?

[Olivier_G]

I use the ctrlX version 1.20 (RM2303).

[MrAdam1983]

i will add that I am also seeing this behavior on a 3x with 1.20 and WebIQ 2.13.0. good to see I am not alone and that it is being investigated. 

[HmiGuide]

WebIQ app for ctrlX will be released next week (31.May-02.June 2023) which should solve the problem.

[Istvan]

Hi all!

Is there any way to get a trial of this release today? We will have a 3 day customer/marketing event from tomorrow, where we have 3 instances running. It would be nice if I would not have to restart the exhibited machines in every hour.

Thank you in advance!

[EngJay]

I would also like to receive a release of the latest version ASAP, for tomorrow Wednesday.  We are releasing a product to the field that will need this fix to perform reliably.

Thanks!

[DdG]

I have the same issue.

(this post is to keep me updated on this topic)

[HmiGuide · Accepted solution]

WebIQ 2.14 is now alvailable in the store. Please give here a feedback, if your issue is solved or not.

[DdG]

I installed the update (server and designer) and updated my project the newer version.

If i try to open it with the WebStation in the kiosk the project is stuck in 'Connecting...'.

I have no problems in chromium, but I'd rather not use it because doesn't display in full screen mode and it refreshes the webpage if you swipe down.

(I added the log file just in case)

[webiq-sk]

This topic is a about the freezing server. 

@DdG Please do not double-post. You already posted to https://developer.community.boschrexroth.com/t5/ctrlX-HMI/Kiosk-mode-WebStation-won-t-load-WebIQ-pro... regarding the WebStation issue. Please let's keep this thread only regarding the freezing issue.

[rexxon]

Hey, many thanks for the update. On our machine it seemed to have solved the issue - at least for a testing time of around 7 hours. no freezes so far but i will keep you updated if anything bad happens.

cheerio