Data Model Developing point data readers for RAMADDA

Point Data Documentation

RAMADDA can provide rich support for structured CSV or text data files. First of all, download and install the pointtools.zip release. All of the examples below, once saved to disk, can be processed with:

    sh <install path>/pointtools/pointchecker.sh  value.csv

Here are some examples of different point data readers. If you have text formatted data that you want to ingest into RAMADDA you can either generate the data in a standard CSV text format for specify a set of metadata properties in an external properties file.

The "standard CSV" format has any number of "#" delimited comment and property lines at the beginning of the file followed by any number of data records. The properties are defined in the header with:

#comment
#property name=property value
#property name=property value
#
value1,value2,valueN
value1,value2,valueN
...

Here is a simple example with just a single column value: The only property that is required is the fields property - a comma separated list of field identifiers with a set of attributes contained within " and "".

    fieldname[attr1="value1" attr2="value2" ...]

Here is a simple example with 2 columns. The second column has a missing value defined. An alternative way to specify attributes of fields is with other named properties as shown below. You can specify a date/time by specifying its type="date". Use format="date format" to specify the date format. Here is an example with time and a single value. Here we have a file where the date and time are in different columns. The isdate and istime attributes specify that the time field is created from both of the columns. The dateformat specifies the format to use. If you have fields with the names yyyy (or year), month (or mm), day, hour, minute, second (or a subset of them) then RAMADDA will figure out the date/time of the records from the column values. If you have georeferenced data then specify latitude and longitude columns. Please, please, please, use decimal degrees east -180 to 180 and decimal degrees north -90 to 90. Here is a georeferenced time series: You can specify different coordinate reference systems with the crs property. For UTM coordinates specify an X and Y field and the utm zone and north/south flag: Here is data in UTM zone 58 South: Here is data in a WGS84 ellipsoid: You can also specify EPSG coordinate systems by setting the crs property to:

crs=epsg:<epgs code>

If you have text values then specify the type="string" Its often the case that a single file has site and lat/long data is implicit in a header, etc. For these cases we want to be able to access the site and location as we read the data. So, we define a fake field with a value="..." attribute. Likewise, you can specify the time value: You can also specify a pattern that is applied to the text in the header to extract out latitude, longitude, elevation, etc.

If you have text point files that are in some pre-existing format (i.e., you can't add "#" properties) then you can specfy Lets assume we have a simple CSV file with 4 columns - We can read this file with just a point.properties file. The key properties are delimiter, skiplines and fields.

You can upload an arbitrary CSV point file and its accompanying properties file. When you are logged in go to File->New Entry. Under the Point Data list choose Text Point Data. Specify the properties in the text field or upload the properties file. You can also define a new entry type in RAMADDA for your point data. Embed the properties in a properties tag. Install the types.xml file as a plugin.
exampletypes_.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<types>

    <type
     description="Test data"
     handler="org.ramadda.data.services.PointTypeHandler"
     name="type_point_test"
     super="type_point">

         <property name="record.file.class" value="org.ramadda.data.point.text.CsvFile"/>

         <property name="record.properties">
delimiter=
position.required=false
skiplines=1
dateformat=yyyy/MM/dd HH:mm:ss
fields=date isdate="true",time istime="true",value chartable="true" unit="some unit"
     </property>

      </type>

</types>





 

Data Model
 
Developing point data readers for RAMADDA