SDK source / ZeroMQ communication

[LoicG]

Hello !

This post is related to the ctrlX DevelopR Challenge.

I'm trying to write a python library to interact with the controller in pure python (so without the SDK which is a python wrapper).
I also figured out the SDK only works on Linux systems (or Windows using the WSL) so that library could help to make the apps more portable.

As far I could see, the SDK use the ports 2069 and 2070.
Using nmap I figured out a ZeroMQ server is listening on these ports.
I tried to snif the packets using Wireshark but the communication is encrypted with TLS (I found some funny words in there ... Maybe an Easter egg ?!)

On GitHub, I didn't found the source code of the SDK.
Any chance for you to release it ?
Or could you briefly explain how the ZeroMQ communication works ? Which commands are sent for authentication and then to manipulate the datalayer ?

Is the SDK used to obfuscate this part ?

Thanks for reading,
Regards

Loïc

[tiramisu]

Hello,

Our motivation for not documenting the comunication layerout is not about obfuscation, but hiding complexity and maintaining compatibility. 

Anyhow, a few words about how the data layer communication works: 

• Communication is encrypted by curveMQ. The public key of communication to the server is available in datalayer/curvemq/publickey (read per HTTPS)
• For authentication a token is obtained per HTTPS and sent with each request.
• But please understand that don't want to make the internal message structure public. This has the following reasons:
  • The libraray provides a stable interface we can not guarantee on telegram level. This is for you a better option to build your application on a stable API instead of using the message level which might be extended or exchanged by something else, which would make your application incompatable.
  • In addition that are certain semantics and required behavioiurs (eg. keepalive, dead connection detection, ...) which also makes it hard to implement and support directly from other programming languages.

As I understand you want to have a portable python application between linux and windows. The Data Layer library is available for linux and windows. So perhaps we can find a convinient way to use these library in the different OS. Please can you elaborate what is the problem with python and datalayer SDK in windows, so that we can help you better?

regards

tiramisu

[DaEwe]

"There is no chance that we will make the protocol public."

Why not? Wouldn't it be nice to have more people working on client libs for the Data Layer? I found this post when I was looking for the same information as Loïc because the python SDK keeps crashing and I would like to fix that..

Cheers,

Daniel

[nickH]

Hello Daniel, 

What do you mean by "the python SDK keeps crashing"? 
Could you give us some details on your troubles?

Regards

Nick

[DaEwe]

Hi Nick,

currently I am experiencing issues when having a provider and a client in the same process. This is needed e.g. for automated testing: I want to check if my application is providing data to the data layer correctly. With the provider running, the data layer client receives a time out. Even if I start the client in another process (multiprocessing.Process) I run into connection errors.

Other issues are my provider crashing with a segfault after 30 seconds of providing data. 

If you like I can try to produce some minimal code examples..

With best 

Daniel

[nickH]

Hello Daniel, 

thank you for your answer!

Code examples would be great to understand and recreate the issues as a whole. 

Best regards

Nick 

[DaEwe · Accepted solution]

Okay, shame on me,  I can't reconstruct the error in my minimal example, client and provider run flawlessly in the same thread. 

I will dig a little bit deeper to find out whats going wrong in the testing framework..

With best,

Daniel

[CodeShepherd]

Moved posts to own topic:

Using VS2019 to get confidend with the datatayer