Data logging high refresh rate

[Davey]

So I succeeded before in using nodered - influxdb and IOT dashboard (grafana) together to log data. 
I have a customer who wants to add new data to the database every 250µs... The data would come from a 4-20mA sensor on the IO card of the ctrlX. 

The data doesn't need to be stored for long, one day is enough. So please advise what the possibilities are? 
Can I make the data layer faster if necessairy? What about node red? What about influx db? 

[Sgilk · Accepted solution]

Hi @Davey ,

The data needs to be updated in the database every 250µs, or the data needs to be collected at that rate? The first option is not going to be feasible. The datalayer, Node-RED and InfluxDB will not update at that rate. The second option is even going to be fairly difficult.

The solution I'd recommend is to buffer the data and push it in a bulk array to the database every few seconds. How you buffer the data is the next question...

What I/O card is being used? Would it be possible to buffer some data on the card?

What ctrlX CORE is being used? Even with the ctrlX CORE x7, I believe the current EtherCAT update rate is at 500µs.

See: Push oversampled arrays to InfluxDB 

[MauroRiboniMX]

Hello, 

There will be the possibility to have zero loss subscriptions from Telegraf i ask in which version and i reach you out guyz. 
Meanwhile @Davey proceed with the guide 🙂

[Davey]

I can test currently what the possibilities are with a CtrlX core X3 and a ctrlX IO card 4-20mA. I can also test in the near future with a XM22. But the customer asked about this possibility, before buying a ctrlX. I am still free to offer what i think is the best solution. I'm indeed guessing it almost makes more sense to buffer the data in maybe an XM for 1 second? If its only for reading IO and storing it that might make more sense. Then push this data to the ctrlX core to the database. 

In the example of oversampled arrays -> influxDB i think Step 1 is used for making a simulation of a signal, so that step can be ignored in my case? 

In step 2 an array is picked up from the data layer, each data point is assigned a different timestamp? I don't get how the timestamp would be correct in the way that it's programmed. 

[MauroRiboniMX]

Hello, 

I think that you can avoid the XM22. from 2.6 there will be the possibility to take the data directly from the datalayer RT and load the data inside influxdb automatically. 

Any point has a timestamp and is calculated taking inot account the sampletime. write me in private maybe we can make a demo together 😁

Mauro 

[Davey]

Hi Mauro, 

Thank you for your reply. I have informed our customer of our options. The customer prefers to check the alternative options. It's clear that we have enough options available to make this work. 

[MauroRiboniMX]

Hello, 

This is how it should work. The feature will be available from release 2.6. Give it a look. The target is to have the ethercat working at 250us.

[ThomasKostenzer]

Hi Mauro,

are there any news about the new release 2.6? Would like to use this great feature 🙂

And I would also like to use this feature in combination with OPC-UA.

Condition:

• CtrlX with running OPC-UA Server
• Separate IPC with InfluxDB and Telegraf, which acts as OPC-UA Client to fetch data from the OPC-UA-Server and writes it into the influxdb.

Is this approach a way to go?

Thomas

[MauroRiboniMX]

Hello, 

What do you want to Monitor? The lossless subscription is just working from Telegraf in the core to influxDB. 
I would avoid the external PC and do anything in the core. 

If you need to write it separatedly you can simply, from the ctrlX push the data in both servers: the ctrlX and the external server 🙂

The setup guide is here: https://developer.community.boschrexroth.com/t5/Store-and-How-to/IIoT-Use-Telegraf-app-to-sample-Rea...

[ThomasKostenzer]

Hello,

thanks for your answer.
We had some performance issues on the CTRLX itself, when running the telegraf especially when connecting directly to the datalayer.
Therefore our approach now is to run Telegraf on our HMI-IPC and use the OPC-UA-Standard to get the value changes.
Our HMI has to work hardware independently, therefore we connecting via OPC-UA instead of datalayer directly.
Is it possible to get the CTRLX-Telegraf. We would run it as a docker container. And is it possible to use LossLess over OPC-UA?

I saw this setup guide already, but without 2.6 I cannot test this. Can you please provide me the version 2.6?

[MauroRiboniMX]

No, is a datalayer specific function. ASK your boschrexroth sopport, he will organize that for you 😃