Gateway / Data Export

Data received from sensors may be exported from the Gateway to external systems. Data points received from sensors are stored to the Gateway’s file system and then exported to a server in one of the supported protocols or file formats. The Gateway is a ‘client push’ system – this reduces problems associated with Firewall and NAT configuration. The Settings > Data Export Page allows the export of data to be configured.

Data Export Settings

Status

Last Data Export: Show the elapsed time since the last export of data or ‘None’ if no data has been exported.

Files Queued for Export: Shows how many files are awaiting export. Completed files would be awaiting export if the first attempt to export the file had failed.

Settings

Data Export Type: The format the data will be exported in and the transport mechanism. Currently there are five options for data export:

  • RKDAP via HTTP(S)
  • CSV via FTP
  • RKDAP stored locally
  • CSV stored locally
  • Google Fusion Tables
Data Export TypeDescription
RKDAPExport data files will be in the ResourceKraft JSON format and will be exported using HTTP(S) as a transport mechanism.
CSV via FTPExport data file will be in CSV format and will be exported using FTP as a transport mechanism.
RKDAP stored locallyExport data file will be in the ResourceKraft JSON format but will be saved locally on the Gateway’s file-system. The files may be access via the Data > Data File Archive page.
CSV stored locallyExport data file will be in CSV format but will be saved locally on the Gateway’s file-system. The files may be access via the Data > Data File Archive page.
Google Fusion TablesData exported via Google Fusion Tables API. If export fails, data stored locally in CSV format.

Data Export Interval: How often the data will be exported. Default is 30 minutes. Data can be exported as frequently as every minute.

Max Data Points per Export: The maximum number of data points per exported file. If the number of data points accumulated in a file reaches this number the file will be closed and exported even if the ‘Data Export Interval’ has not elapsed. This is a useful parameter to set if a low bandwidth GPRS internet connection is being used, and it will limit the size of file that will be exported so the export has a better chance of succeeding.

RKDAP

HTTP(S) Server URL: URL to use for HTTP or HTTPS as advised by ResourceKraft or others. This field is validated as a URI with the following set of acceptable characters: [A-Za-z0-9.:/_-]

DAD-ID: Data Acquisition Device Identifier as assigned by ResourceKraft or others. Please note that the DAD-ID field must also be completed in the case of RKDAP stored locally export type. The DAD-ID field is validated with the following set of acceptable characters: [A-Za-z0-9.’ ‘_-]

An example of a file in RKDAP format is show here:

{
    "method":"dad_data",
    "ver":"1.1",
    "dad_id":"TEST_00055",
    "data":[         
{"id":"000D6F00010B528D_350","period":"2012-09-27T16:26:00","value":22.2},
{"id":"000D6F00010B7566_520","period":"2012-09-27T16:28:00","value":1.0}
    ]
}

In this example the DAD-ID is “TEST_00055”. There are two data points in this file. The first data point has a time stamp of “2012-09-27T16:26:00” and a value of “22.2”. The time stamp is encoded according to a subset of RFC 3339 and is in terms of GMT. The sensor data value is a floating point value.

This data point originates from data source with Export ID 000D6F00010B528D_350. The default Export ID is the node serial number combined with the sensor ID (see Sensor Settings section for more information). ResourceKraft need to be advised of the Export ID’s (a.k.a data source ID’s) so data feeds can be associated within ResourceKraft Advisor software.

CSV via FTP

Host: The FTP site host name or IP address.

Username: The FTP account user name.

Password: The FTP account password. The password can consist of letters [A-Z] and [a-z], numbers [0-9] and the following characters [. ! * ^ _ -]

Remote Directory: The remote directory to store files in. This is an optional field. Leading and trailing backslashes ( \ ) are also optional and will be ignored.

FTP Timeout: The timeout to use for all FTP transactions. Default value is 300000 milliseconds.

Note: The FTP server must be configured to accept PASSIVE mode connections to avoid problems with Firewalls on the client side.

An example of a file in CSV format is show here:

ExportID, Date, Value
000D6F00010B528D_350,2012-10-02T17:29:00,23.8
000D6F00010B755D_462,2012-10-02T17:15:00,0.0
000D6F00010B528D_350,2012-10-02T17:32:00,23.9
000D6F00010B528D_350,2012-10-02T17:35:00,24.1
000D6F00010B528D_350,2012-10-02T17:36:00,23.9
000D6F00010B528D_350,2012-10-02T17:37:00,23.5

In this example there are 6 data points. The first data point has a timestamp of “2012-10-02T17:29:00” and a value of “23.8”. The timestamps is encoded according to a subset of RFC 3339 and is in terms of UTC, with an offset as configured in the time and date page of the settings section. The sensor data value is a floating point value. This data point originates from a data source with Export ID: 000D6F00010B528D_350. The default Export ID is the node serial number combined with the sensor ID (see the ‘Sensor Settings’ section for more information). Export ID’s should be noted when configuring data feeds in any other software system.

For more information regarding the export protocol definitions contact: support@episensor.com

If a data export via FTP or HTTP(S) fails, the export file will be stored on the local file system. The number of files queued for export in this way will be reflected in the “Status” section of this page. Export of these files will be retried periodically, based on the data export interval.

Note: Export files saved when a connection to the server is unavailable will not be available for download via the web interface.

If communications fail during a data export, an error will be returned and the data export file will be queued in the local file system to be exported when communications have been restored.

Note: The file name of the CSV file will be DAT_XXXXXX_YYYYYY.csv where XXXXXX is the Gateway’s unique serial number and YYYYYY is a date/time stamp.

Google Fusion Tables

Google Client ID: The ‘Client ID’ string of the OAuth2 Client ID created in the Google account from the API Access Console. The Client ID is validated with the following set of acceptable characters: [A-Za-z0-9._-].

Google Client Secret: The client secret string of the OAuth2 Client ID created in the Google account from the API Access Console. The client secret is validated with the following set of acceptable characters: [A-Za-z0-9.!*^_-].

To configure Google Fusion Tables, see the section “How to Configure Data Export to Google Fusion Tables” at the end of this document.

Google Fusion Tables Settings

Next Page >>