Data exchange between two ctrlX Apps

[EchoHarm0ny]

Hi,

we are facing the challenge of designing the data exchange between two apps to be as performant as possible.

• The typical amount of data that needs to be exchanged per cycle is 30-60 Mb
• We are planning for 400-800 cycles per hour

We would like to learn more about the methods you would recommend for exchanging data between apps. What are the advantages and disadvantages of the different approaches?

• We found the following recommendation for the exchange via the file system:"The app must not write diagnostics or similar data cyclically to the internal solid-state memory, as this will damage the flash cells." https://boschrexroth.github.io/ctrlx-automation-sdk/appdevguide.html#622-integrated-storageflash-lif...Will this recommendation change in the case that a IPC with ctrlX will be used? Its sounds very specific for ctrlX CORE hardware.

• If the exchange with via the files system is viable, is there an example that shows using plugs and slots to define a shared directory for two apps to use?
  • Consider that these two apps may not be made by the same company, they may be not built at the same time.

• We also found an cpp example for the exchange via Datalayer shared memory, is this example also available in other programming languages?

[Sgilk]

@EchoHarm0ny ,

What kind of data are you exchanging between the applications?

• We found the following recommendation for the exchange via the file system:"The app must not write diagnostics or similar data cyclically to the internal solid-state memory, as this will damage the flash cells." https://boschrexroth.github.io/ctrlx-automation-sdk/appdevguide.html#622-integrated-storageflash-lif...Will this recommendation change in the case that a IPC with ctrlX will be used?Its sounds very specific for ctrlX CORE hardware.

This is going to be a problem with any storage media. The frequent read/write operations generate mechanical wear on the hardware. If you want to go this route, I'd recommend a removable media device.

• If the exchange with via the files system is viable, is there an example that shows using plugs and slots to define a shared directory for two apps to use?

The best approach here would be using removable storage media and the corresponding plug as shown above. It may also be possible to use the home interface but this plug does not auto-connect. A request to development would need to be made to possibly implement auto-connect for this plug. 

• We also found an cpp example for the exchange via Datalayer shared memory, is this example also available in other programming languages?

This would be my recommended solution, especially if the data is of a consistent format and simple types. There are not examples in other languages for shared memory. What language were you planning to use?

[EchoHarm0ny]

Hi @Sgilk, thank you for the explanation

Your questions in blue

What kind of data are you exchanging between the applications?

• Data acquired by a 3D camera. Our current solution outputs files.

The best approach here would be using removable storage media and the corresponding plug as shown above.

• How fast is the interface of a removable media on a ctrlX CORE, i read in the data sheet USB 2.0, is that correct?
• Is it possible to auto connect on removable media with two different snaps?

This would be my recommended solution, especially if the data is of a consistent format and simple types. There are not examples in other languages for shared memory. What language were you planning to use?

• A Python example for shared memory would be great.

I also read about an option to use RAM-Disk on Ubuntu systems, is that a feasible approach on a ctrlX Hardware? I assume its challenging create a RAM-DISK and then mounting it to two snaps.
The benefits of a RAM-Disk for us:
- We could continue writing files
- Very fast read and write operations
- Can sustain countless reads and writes

[EchoHarm0ny]

HI @Sgilk,

we looked into the datalayer shared memory example.

A few questions:

In the sample the two apps are running in the same snap. It is necessary for us that the data provider is a own snap and the data consumer is a seperate app. Is there a configuration in the snapcraft.yaml necessary that both apps can access the shared memory?

The MEM_SIZE of the shared memory is set to 100 Bytes, we increased it to fit our 60Mb requirements, the app is not starting after this and we receive an error: DL_OUT_OF_MEMORY
How much memory is available for the shared memory?

Is it possible to do a shared memory implementation without the datalayer?

[Sgilk]

@EchoHarm0ny ,

There shouldn't be any special snapcraft.yaml configuration required to share memory between applications. It should be the same as the sample.

I am not finding a documented limit on shared memory. Maybe @nickH could comment?

The shared memory functionality is provided through the datalayer service. I don't think we can support it otherwise.

Now that I understand more of the application, I think the removable media file sharing approach would be more suitable. This way you don't need to parse/encode your files as compatible types for the memory range.

[EchoHarm0ny]

Thank you,

we increased the MEM_SIZE in the example app main_owner
 (line 16), would be interesting to know the threshold.
Would be also intersting to know if shared memory without datalayer is possible.

Removable media via the sd card slot or the usb-c interface, correct?
I read in a data sheet that all ctrlX CORE have a usb 2.0 interface.
Which read/write speed can we expect for a ctrlX CORE x7?

Is it possible to auto connect on removable media with two different snaps?

[Sgilk]

Yes, you can use USB-C 2.0 or uSD.

• USB-C 2.0 should provide 10-25 Mbps read and 3-10 Mbps write
• uSD depends on the card type

The removable media and storage extension interfaces are both auto-connecting.

[CodeShepherd]

As an addition, apps can also exchange data via Unix sockets directly. The same mechanism is used internally in the ctrlX OS at several points.

[Sgilk]

@CodeShepherd ,

Do we have any documentation on this? I see the information on using the existing unix sockets (ie. licensing-service), and binding a webserver to a unix socket within a single application, but what about data exchange between apps? 

[CodeShepherd]

@Sgilk you found the correct documentation. The sockets used are examples how our system (consisting of apps) is exchanging data with other installed apps.

Also simply using an content interface in one app where the other app can connect to is possible. Just like our solution management is sharing the hosting folders for the active configuration and all apps can access this folder and its data.