Developing point data readers for RAMADDA Time Series JSON Format

Point Data Documentation

The same point data access functionality that is available to an end user through the web forms is also available to external clients (e.g., Open Topography via the Point Data Access API.

There are 3 major service areas provided by the API: Collection query, Data access service and Job status.

RAMADDA supports querying on the point collections it holds and returning the results as an ATOM xml document. The URL to the ATOM feed is:

http//<hostname>/repository/search/type/point_collection?output=atom
or
http//<hostname>/repository/search/type/lidar_collection?output=atom

e.g.

The URL above returns the results as an Atom xml document that lists each of the matched LiDAR Collection entries in RAMADDA. For each collection it includes spatial, temporal and descriptive metadata as well as link elements that point to the different data access requests.

Search criteria can be added to the above URL to allow for spatial and metadata search.

• To search within spatial bounds use area_west, area_south, area_east and area_north arguments:
  https://nlas.unavco.org/repository/nlas/capability.xml?area_north=40&area_west=-120&area_east=-100&area_south=20

  These are the longitude and latitude bounds (in degrees) of the collections.

• To search for collections that hold waveform data use the metadata.attr1.nlas_haswaveform URL argument:
  https://nlas.unavco.org/repository/nlas/capability.xml?metadata.attr1.nlas_haswaveform=true

• To search based on LiDAR platform use the metadata.attr1.nlas_platform URL argument:
  https://nlas.unavco.org/repository/nlas/capability.xml?metadata.attr1.nlas_platform=High+Altitude+LiDAR

• To search based on text use the text URL argument:
  https://nlas.unavco.org/repository/nlas/capability.xml?text=lvis

An easy way to explore this is to go main NLAS search form, e.g.: https://nlas.unavco.org/repository/nlas/search and do the search interactively. The search URL will be shown in the location bar.

The Atom response contains various metadata elements for each collection.

The metadata includes geospatial tags using GEORSS tags:

<georss:polygon>lat1 lon1 lat2 lon2 ... latN lonN</georss:polygon>

<georss:box>44.0489574999292 -71.31482662748971 44.084895027353014 -71.2647717415648</georss:box>

Descriptive metadata is embedded in a metadata tag. The contained elements are DIF metadata and some custom NLAS specific elements:

<metadata>
  <Temporal_Coverage>
    <Start_Date>2011-08-10T15:04:56 +0000</Start_Date>
    <Stop_Date>2011-08-10T15:04:56 +0000</Stop_Date>
  </Temporal_Coverage>
  <HasWaveform>true</HasWaveform>
  <Source_Name>
    <Short_Name>High Altitude LiDAR</Short_Name>
    <Long_Name>LVIS Airborne LiDAR System</Long_Name>
  </Source_Name>
</metadata>

The link elements in the Atom response allow you to invoke different data access requests. The type attribute defines what type of product is to be generated.

<link  type="text/html"  
       href="https://nlas.unavco.org/repository/entry/show?entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5"  
       title="Web page" />
<link  type="points.latlonaltcsv"  
       href="https://nlas.unavco.org/repository/entry/show/LVIS Costa Rica.csv?entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5&output=points.product&product=points.latlonaltcsv&asynch=true"  
       title="Lat/Lon/Alt CSV" />
<link  type="points.las"  
       href="https://nlas.unavco.org/repository/entry/show/LVIS Costa Rica.las?entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5&output=points.product&product=points.las&asynch=true"  
       title="LAS" />
<link  type="points.asc"  
       href="https://nlas.unavco.org/repository/entry/show/LVIS Costa Rica.asc?entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5&output=points.product&product=points.asc&asynch=true"  
       title="ARC Ascii Grid" />
<link  type="points.kmz"  
       href="https://nlas.unavco.org/repository/entry/show/LVIS Costa Rica.kmz?entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5&output=points.product&product=points.kmz&asynch=true"  
       title="Google Earth KMZ" />
<link  type="points.count"  
       href="https://nlas.unavco.org/repository/entry/show/LVIS Costa Rica.xml?entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5&output=points.product&product=points.count&asynch=true"  
       title="Point Count" />
<link  type="thumbnail"  
       href="https://nlas.unavco.org/repository/metadata/view/hillshade.png?element=1&entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5&metadata.id=d9d5200b-e6b1-449b-8ca1-e57e0a76e135"  
      title="Thumbnail" />

To break the URL down we have:

The entry path:

    /repository/entry/show

The name used is not needed but is there for browser and wget saves:

    LVIS Costa Rica.las

This is the RAMADDA entryid for the LiDAR Collection entry:

    entryid=e85ebe95-65c8-4590-b933-dfd1453e7ce5

This just says to make a product:

    output=points.product

This is the product to make:

    product=points.las

However, one can specify any number of product url arguments, e.g.:

     ...product=points.asc&product=points.las...  

will generate both an ASCII ARC Grid and a LAS file.

This says to have the request be asynchronous. This results in a job status xml being returned. If asynch=false then, if there is just one product specified, the return of the URL fetch will be the actual product file.

    asynch=true 

You can also have any of the above URLS return a point count of how many points (approximately) will be returned:

   pointcount=true|false

If you also specify:

      response=xml

you will get the number of points wrapped in xml.

There are a number of other URL arguments that can be specified to do spatial subsetting and decimation.

To spatially subset use the same URL arguments used for the LiDAR Collection search:

area_west, area_south, area_east and area_north 

To decimate:

    lidar.skip=<skip factoro>

To specify a fixed number of points:

   numpoints=<approximate number of points>

To probabilistically sample points:

   probability=<probablity a point will be included 0.0-1.0>

If the job submission request has a

    response=xml

argument then the return from the job request and the job status urls will be in this xml format.

The xml is wrapped with <response>...<response> tags. The response tag has a code attribute for specifying whether an error took place or if the call was ok.

We have this outer response tag so it is easy to determine if there was an error and if so to retrieve the error message.

If there is some error then the response will be:

<response code="error">
    error message here
</response>

Else if there was no error then the response will be of the form:

<response code="ok">
... some xml response here ...
</response>

When submitting a job request the xml response will return 2 urls. One to show the job status and one to cancel the job. We use the type attribute to distinguish between the 2 urls:

<response  code="ok" >
<url  type="status" >
    <![CDATA[http://localhost:8080/repository/entry/show/The+Mac/NLAS/Data/LVIS+Costa+Rica?entryid=3f9e9a07-fc6e-4aa0-b2d7-7eefef03b027&output=points.results&job.id=8dfb5456-ec52-429b-a75d-0c74c43b6c56&response=xml]]>
</url>
<url  type="cancel" >
    <![CDATA[http://localhost:8080/repository/entry/show/The+Mac/NLAS/Data/LVIS+Costa+Rica?entryid=3f9e9a07-fc6e-4aa0-b2d7-7eefef03b027&output=points.results&job.id=8dfb5456-ec52-429b-a75d-0c74c43b6c56&response=xml&cancel=true]]>
</url>
</response>

The jop status URL returns the status of the job in an xml format. Hitting the job status url for a job that is still running gives the following xml:

<response code="ok">
    <job status="running"/>
</response>

If the job was cancelled then the status xml is:

<response code="ok">
    <job status="cancelled"/>
</response>

If the processing job has completed then we get:

<response code="ok">
    <job status="complete">
        <products>
            <url>url to retrieve product #1</url>
            <url>url to retrieve product #2</url>
            <url>...</url>
        </products>
    </job>
</response>

To put this all together then to access data you:

1. Query the LiDAR Collection to identify the desired collection.
2. From the Atom Link elements invoke the data processing request adding a response=xml url argument.
3. Get back the job invocation response xml. Extract out the job status URL.
4. Poll the job status URL until the job is complete.
5. Fetch the generated data and products via the given URLs.

 

Developing point data readers for RAMADDA

Time Series JSON Format