Data logging high refresh rate

[Davey]

So I succeeded before in using nodered - influxdb and IOT dashboard (grafana) together to log data. 
I have a customer who wants to add new data to the database every 250µs... The data would come from a 4-20mA sensor on the IO card of the ctrlX. 

The data doesn't need to be stored for long, one day is enough. So please advise what the possibilities are? 
Can I make the data layer faster if necessairy? What about node red? What about influx db? 

[Sgilk · Accepted solution]

Hi @Davey ,

The data needs to be updated in the database every 250µs, or the data needs to be collected at that rate? The first option is not going to be feasible. The datalayer, Node-RED and InfluxDB will not update at that rate. The second option is even going to be fairly difficult.

The solution I'd recommend is to buffer the data and push it in a bulk array to the database every few seconds. How you buffer the data is the next question...

What I/O card is being used? Would it be possible to buffer some data on the card?

What ctrlX CORE is being used? Even with the ctrlX CORE x7, I believe the current EtherCAT update rate is at 500µs.

See: Push oversampled arrays to InfluxDB 

[MauroRiboniMX]

Hello, 

There will be the possibility to have zero loss subscriptions from Telegraf i ask in which version and i reach you out guyz. 
Meanwhile @Davey proceed with the guide 🙂

[Davey]

I can test currently what the possibilities are with a CtrlX core X3 and a ctrlX IO card 4-20mA. I can also test in the near future with a XM22. But the customer asked about this possibility, before buying a ctrlX. I am still free to offer what i think is the best solution. I'm indeed guessing it almost makes more sense to buffer the data in maybe an XM for 1 second? If its only for reading IO and storing it that might make more sense. Then push this data to the ctrlX core to the database. 

In the example of oversampled arrays -> influxDB i think Step 1 is used for making a simulation of a signal, so that step can be ignored in my case? 

In step 2 an array is picked up from the data layer, each data point is assigned a different timestamp? I don't get how the timestamp would be correct in the way that it's programmed. 

[MauroRiboniMX]

Hello, 

I think that you can avoid the XM22. from 2.6 there will be the possibility to take the data directly from the datalayer RT and load the data inside influxdb automatically. 

Any point has a timestamp and is calculated taking inot account the sampletime. write me in private maybe we can make a demo together 😁

Mauro 

[Davey]

Hi Mauro, 

Thank you for your reply. I have informed our customer of our options. The customer prefers to check the alternative options. It's clear that we have enough options available to make this work. 

[MauroRiboniMX]

Hello, 

This is how it should work. The feature will be available from release 2.6. Give it a look. The target is to have the ethercat working at 250us.