Transferring and Representing CSV Files from IndraControl to ctrlX CORE using FTP and InfluxDB

[sanX]

• Introduction
• Prerequisites
• Establish communication between both devices
• Configure InfluxDB
• Step 1: Install InfluxDB
• Step 2: Configure InfluxDB
• Step 3: Get API token
• Step 4: Create bucket
• Configure Node-RED
• Step 1: Install the Node-RED app
• Step 2: Provide internet connection to the ctrlX CORE
• Step 3: Start Node-RED and install the required nodes
• Step 4: Download Node-RED flow and import it
• Step 5: Connect ctrlX CORE with IndraControl
• Step 6: Configure nodes
• Step 7: Deploy
• Testing
• Data visualization
• Delete IndraControl csv file
• Delete ctrlX CORE csv files

Introduction

In this tutorial it will be shown how to configure Node-RED to be able to get CSV files stored in an IndraControl and represent the content of these files in InfluxDB. To be able to communicate with the IndraControl, the FTP communication protocol is used. For this purpose it is required to install the FTP nodes in Node-RED. On the other hand, with the use of function and csv nodes, the content of the csv files will be read and processed, so that the information can be sent to InfluxDB in the correct format. Finally, with the use of the InfluxDB node, data is sent and represented.

Prerequisites

• ctrlX CORE - Node-RED App 
• ctrlX CORE - InfluxDB App

Establish communication between both devices

• First, it is required to connect the IndraControl and the ctrlX CORE to be able to establish the FTP communication. For this purpose, please connect them as shown in the image:

Communication overview

• Secondly, it is required to configure the IP address between both devices. For that, connect to the ctrlX CORE with the Engineering computer. In the case, the user does not know how to achieve this, please check this tutorial. Once we are inside the ctrlX CORE web interface, it is required to follow the steps:

1. Access the "Settings" menu
2. Press "Network Interfaces"
3. Select the "XF51" interface
4. Select the "IPv4" tab
5. Enter an IP address in the same range as the IndraControl, but not with the same IP address because otherwise it will have conflicts. Additionally, enter the subnet mask.
6. Press the "Save" button and store the configuration

Connection configuration between ctrlX CORE and IndraControl

• Once the configuration has been saved, a "Connected" message should appear:

Successful connection check

• To make sure, the device is correctly connected, it is of interest to ping to the IndraControl from the ctrlX CORE. For that, return back to "Settings" and press over "Network Diagnostics"

Network diagnostics menu access

• In the "Network diagnostics" menu, enter the IndraControl IP address and check if data is transmitted and received as shown below:

Connection check between ctrlX CORE and IndraControl

• Finally, it is recommended to check that the IndraControl has the FTP communication protocol active.

1. Disconnect the IndraControl with the ctrlX CORE and connect the IndraControl with the Engineering computer. Once done that, configure the static IP address accordingly. 
2. Start IndraWorks and communicate with the IndraControl
3. Access the communication protocols menu: Right click over the device --> Communication --> Protocols

Access IndraControl communication protocols

4  Check that FTP is active as shown below:

Check FTP protocol

Configure InfluxDB

Step 1: Install InfluxDB

Download the app from this link and install it. In the case, it is the first time installing an app in the ctrlX CORE, please check the steps shown in this tutorial.

Step 2: Configure InfluxDB

• Define user settings

Provide a username, password, organization name, an initial bucket name, and press "Continue".
Note: A bucket is a named location where all our data is stored. 

InfluxDB initial configuration

• In the next window, select any of the options.

Step 3: Get API token

To be able to use the InfluxDB node in Node-RED it is required to get a bearer token from InfluxDB.

1. Hover the mouse over the Load data symbol in the InfluxDB homepage
2. Select "API Tokens" and the window as shown below will appear
3. Press over "Generate API Token"
4. Select the "All access API Token" option

Bearer token creation

• Provide a description, click over "Save" and a new window with the API token will appear

Bearer token creation

• Press over "Copy to clipboard" and keep the token in a safe place because later this will be required

Step 4: Create bucket

To store all the data sent from Node-RED to InfluxDB, a bucket es required to define. 

1. Hover with the mouse over the Data explorer symbol
2. Press over "Create bucket"
3. Provide a name. In this case, we will name this bucket "FTP"
4. Press the "Create" button

Bucket creation in InfluxDB

Configure Node-RED

Step 1: Install the Node-RED app

Download the app from this link and install it. In the case, it is the first time installing an app in the ctrlX CORE, please check the steps shown in this tutorial.

Step 2: Provide internet connection to the ctrlX CORE

To be able to install some nodes required to successfully connect per FTP to the IndraControl, it is necessary to connect to the Internet. For this purpose, connect the XF51 port to a router and make sure that in the Network Interfaces menu, the automatic settings configuration is enabled.

ctrlX CORE internet access configuration

• If the connection has been correctly established, a similar window as the previously shown should appear.

Note: It is required to have internet access in the ctrlX CORE and in the Engineering computer.

Step 3: Start Node-RED and install the required nodes

1. Press over the three horizontal lines
2. Select "Manage palette"

Access Node-RED palette Menu

• Install the FTP nodes:

1. In the palettes menu, press over "Install"
2. Write ftp and find the node-red-contrib-ftp package
3. Press over the "Install" button

FTP nodes installation

Once the nodes have been successfully installed, the following message should appear:

FTP nodes addition

• In addition to these nodes, it is also required to install the following ones:

1. node-red-contrib-influxdb
2. node-red-contrib-fs

Step 4: Download Node-RED flow and import it

At the end of this tutorial, there is a zip file which contains the json file required to upload to Node-RED.

1. Press over the three horizontal lines
2. Press over "Import"

Flow import in Node-RED

In the "Import nodes" menu follow the steps:

1. Press "Select a file to import"
2. Select downloaded and extracted json file and the following code should be visible
3. Import the flow

Flow import in Node-RED

• Once the flow has been imported, the following overview should be visible:

Node-RED flow overview

Step 5: Connect ctrlX CORE with IndraControl

As we previously used the XF51 port to connect the ctrlX CORE to the internet, now it is required to connect again both devices to be able to establish a FTP connection. For this purpose, please connect both devices as shown in the first image of this tutorial with their corresponding IP addresses.

Step 6: Configure nodes

Once the file has been imported, the flows need to be configured to be able to connect to the IndraControl.

1. FTP nodes:

List directories node access

• Double click over the node and a new window will be opened. In the filename, introduce the path of where the csv files are located in the FTP server. 

• On the other hand, provide the FTP server configuration by clicking over the pencil. With the FTP server IP address, port, username and password, the ctrlX CORE will be able to connect to the server. Once all the information has been introduced, press over "Update" and you will return back to the first window where it is required to click over "Done" to close it

FTP node configuration

• Once this information has been introduced, all the FTP nodes used in this flow will automatically be updated with this data. 

FTP nodes in the general flow

2. InfluxDB nodes:

InfluxDB node configuration access

• Double click over the node and a new window will be opened:

1. Check that the organization name matches with the one introduced when configuring InfluxDB
2. Check the bucket name 
3. Enter the InfluxDB configuration menu

InfluxDB node configuration

• In the new window, it is required to follow these steps:

1. Select version 2.0
2. Introduce the ctrlX CORE url with port 8086 per default
3. Introduce the previously stored API token
4. Press over the "Update" button to store the configuration

InfluxDB node configuration

• Once this node has been updated, the other three influxDB nodes will automatically be updated.

3. Return url node:

In this function node, the ctrlX CORE IP address is introduced and it is required to make sure that it matches with your ctrlX CORE address

ctrlX CORE IP address check

4. Delete files in Active node:

In this node it is required to introduce the ctrlX CORE credentials to be able to make HTTP requests

Credentials configuration in HTTP request node

Step 7: Deploy 

Once all the nodes have configured, we can deploy the flow by clicking over the "Deploy" button

Flow deployment

Testing 

Data visualization

Once the deploy button is clicked, if everything has correctly been configured, these debug messages will appear:

Check debugging nodes

• In these messages the user can read the csv values and their corresponding timestamps.

Now that data is sent to InfluxDB, we can check in the Data explorer if data is arriving:

1. Select the Data explorer button
2. Select the FTP bucket to visualize the data that has been stored there
3. Select which column of the csv file you want to visualize. Note: It is possible to select multiple columns
4. Press over "Custom" in the window period
5. Select 1ms to be able to read data at ms level if data is sent at this frequency
6. In "Aggregate function", select custom and do not select any function sot hat raw data is shown
7. Choose the data you want to visualize. In the case, the csv files have been generated in a defined hour, it is possible to configure this
8. Press over the "Submit" button to visualize the received data

Visualize received data in InfluxDB

Delete IndraControl csv file

In the case the user wants to delete all the csv files stored in the IndraControl, there is a button in the Node-RED flow which provides this capability.

Delete IndraControl csv files

• In the debug messages, it is possible to see all the files which will be deleted. In this case, 445 csv files will be deleted. Additionally, for every deleted file, a "File deleted" message will appear.

Delete ctrlX CORE csv files

As all the read csv files are also stored in the ctrlX CORE, the user may be interested on deleting these files. 

Note: In this case, the flow has been configured so that all the csv files are stored in active configuration folder of node-RED

csv files location in the ctrlX CORE

• To change the path where the folders have to be stored, open the "Check new objects" node and change the local path variable:

csv files path in the ctrlX CORE

• Keep in mind, that in the case the path is changed, it is required to modify the code in the following nodes:

1. "Check if file has been stored" node

Path check in Check if file has been stored node

2. "List files" node

Path check in list files node

3. "Return url" node

Path check in Return url node

• Once we have understood where the csv files are stored in the ctrlX CORE, we can test the button to delete the files by using the inject node:

Delete ctrlX CORE csv files

• In this case 38 files have been deleted

FTP_tutorial.zip