Data Upload
RAMADDA provides a data upload API that enables record oriented data to
be uploaded and appended to some file based on a sensor ID.
This service is centered around one or more groups, i.e., a group of
data files. For each group you need to define the following properties
in the repository.properties file in your RAMADDA home directory:
upload.<GROUP>.directory=/path/to/store/the/files
upload.<GROUP>.file=file name template
upload.<GROUP>.header=header for when a file is created
upload.<GROUP>.apikey=apikey
e.g. for a group "group1":
upload.group1.directory=/data/group1
upload.group1.file=somedata_{sensor}.csv
upload.group1.header=#This is the header for the data\nd1,d2,d3
upload.group1.apikey=mykey
- The .header is used when a file is first created. Use "\n" as a new line. RAMADDA
will also write out a new line at the end of the header.
- The .file property is a template that gives the name of the file to create. Use "{sensor}"
for the sensor id and "{group}" for the group id
- The .apikey is used to verify the API call to upload data
The above properties define a group "group1". To upload a record for the sensor "sensor1"
simply call the API with:
https://server/data/upload?apikey=<your key>&group=group1&sensor=sensor1&data=7,8,9
This results in a new file called somedata_sensor1.csv
under the directory /data/group1. The file will initially hold:
#This is the header for the data
d1,d2,d3
7,8,9
If you wanted to upload multiple records then do:
https://server/data/upload?apikey=<your key>&group=group1&sensor=sensor1&data=7,8,9\n10,11,12
Data from another sensor would be uploaded as:
https://server/data/upload?apikey=<your key>&group=group1&sensor=sensor2&data=7,8,9\n10,11,12
Note: the return of the above API calls will be JSON of the form:
{"code":"ok","message":"data uploaded"}
If there is an error it will return, e.g.:
{"code":"error","message":"No upload group found"}
You can also specify one or more other RAMADDA servers that you can relay data to.
When the first RAMADDA receives data it first writes the data to the file as specified above.
If there are any relay servers specified then it also will pass on the data to
each of those servers in turn.
To configure a relay server for a particular group just provide the RAMADDA URL in the properties file as:
upload.<GROUP>.relay.servers=<comma separated list of RAMADDA URLs>
e.g.:
upload.group1.relay.servers=https://ramadda.org/repository,https://geodesystems.com/repository
By default the group and API key from the initial call is used. This can be overwritten with:
upload.<GROUP>.relay.apikeys=<comma separated list of keys>
upload.<GROUP>.relay.groups=<comma separated list of groups>
where each key and group in the list correspond to the server. You can specify only one key/group
which would be applicable to all of the servers.
Listing and Ingesting Files
If you are logged into RAMADDA as a site administrator you can view the files that have been uploaded
by going to the url /repository/data/uploadlist on your server.
To enable this you must specify the below property property so that RAMADDA
can know of the different upload groups you have defined:
upload.groups=<comma separated list of groups>
e.g.:
upload.groups=group1,group2
As an administrator you can ingest and make available these file in your RAMADDA as entries of a certain type.
If they are CSV files then create a CSV File entry type. You can also define new entry types. There is
incomplete documentation available
here
but there are logs of examples available on
GitHub.
To create the entry you can do any of the following:
- Create a harvester and have it periodically scan
the containing directory. This is the best approach for most cases.
- Create an individual entry with the full file path in the "Files on Server" tab in the create entry form
- Create an individual entry with the directory of the files in the "Files on Server" tab in the create entry form.
This will add all files as entries. However, any newly created files would have to be added again.
- Create a Server Side File entry. This won't add
the files as entries into RAMADDA's database but will make them accessible.
