Data Upload

RAMADDA provides a data upload API that enables record oriented data to be uploaded and appended to some file based on a sensor ID.
This service is centered around one or more groups, i.e., a group of data files. For each group you need to define the following properties in the file in your RAMADDA home directory:
upload.<GROUP>.file=file name template
upload.<GROUP>.header=header for when a file is created

e.g. for a group "group1":
upload.group1.header=#This is the header for the data\nd1,d2,d3

  • The .header is used when a file is first created. Use "\n" as a new line. RAMADDA will also write out a new line at the end of the header.
  • The .file property is a template that gives the name of the file to create. Use "{sensor}" for the sensor id and "{group}" for the group id
  • The .apikey is used to verify the API call to upload data
The above properties define a group "group1". To upload a record for the sensor "sensor1" simply call the API with:
https://server/data/upload?apikey=<your key>&group=group1&sensor=sensor1&data=7,8,9
This results in a new file called somedata_sensor1.csv under the directory /data/group1. The file will initially hold:
#This is the header for the data
If you wanted to upload multiple records then do:
https://server/data/upload?apikey=<your key>&group=group1&sensor=sensor1&data=7,8,9\n10,11,12
Data from another sensor would be uploaded as:
https://server/data/upload?apikey=<your key>&group=group1&sensor=sensor2&data=7,8,9\n10,11,12
Note: the return of the above API calls will be JSON of the form:
{"code":"ok","message":"data uploaded"}
If there is an error it will return, e.g.:
  {"code":"error","message":"No upload group found"}
Relaying Data
You can also specify one or more other RAMADDA servers that you can relay data to. When the first RAMADDA receives data it first writes the data to the file as specified above. If there are any relay servers specified then it also will pass on the data to each of those servers in turn.

To configure a relay server for a particular group just provide the RAMADDA URL in the properties file as:
upload.<GROUP>.relay.servers=<comma separated list of RAMADDA URLs>
By default the group and API key from the initial call is used. This can be overwritten with:
upload.<GROUP>.relay.apikeys=<comma separated list of keys>
upload.<GROUP>.relay.groups=<comma separated list of groups>  
where each key and group in the list correspond to the server. You can specify only one key/group which would be applicable to all of the servers.
Listing and Ingesting Files
If you are logged into RAMADDA as a site administrator you can view the files that have been uploaded by going to the url /repository/data/uploadlist on your server. To enable this you must specify the below property property so that RAMADDA can know of the different upload groups you have defined:
upload.groups=<comma separated list of groups>  

As an administrator you can ingest and make available these file in your RAMADDA as entries of a certain type. If they are CSV files then create a CSV File entry type. You can also define new entry types. There is incomplete documentation available here but there are logs of examples available on GitHub.

To create the entry you can do any of the following:
  • Create a harvester and have it periodically scan the containing directory. This is the best approach for most cases.
  • Create an individual entry with the full file path in the "Files on Server" tab in the create entry form
  • Create an individual entry with the directory of the files in the "Files on Server" tab in the create entry form. This will add all files as entries. However, any newly created files would have to be added again.
  • Create a Server Side File entry. This won't add the files as entries into RAMADDA's database but will make them accessible.