Now that you have your RAMADDA server installed you may ask how do I get go about populating it with data and other content.
We covered how to make folders and upload files (or archives of files) through the web interface. However, RAMADDA also supports
ways of providing access to the files that are already on the server machine.
RAMADDA provides a harvester facility
that allows an administrator to scan the local file system on the server
and add entries into the database for those files.
For example, say you have a large directory of data (and/or other) files (e.g., /project/data
that you want RAMADDA to provide access to.
You could simply create a Server Side File
view of the directory but this does not
actually add entries into the repository database for the files and you cannot add metadata, search etc., with those entries.
The File Harvester can walk the directory tree and add entries into the repository for the files it finds.
The first thing you need to do is to specify in the Site Administration area
that it is OK to serve up files from that directory tree. Go to the page in RAMADDA
and enter the directory path, e.g., /project/data:
Image 1: File System Access
Next, create a File Harvester by going to the Harvesters tab of the Admin pages and press the "New Harvester" button.
Specify a name (e.g., "Test"), leave the type as Local Files and hit create:
Image 2: Create a new File Harvester
This will take you to the Harvester Edit form. The minimal configuration you need to do here is to enter the file
system directory to scan and the RAMADDA folder to add the harvested entries to. In this example
we are harvesting the directory tree /project/data
and we are adding the entries into
the RAMADDA folder RAMADDA/Case Studies/Test
Image 3: Harvester Edit Form
When you have entered this information press "Change" which will apply the changes and then take you back
to the Harvester Edit Form. You can start this Harvester with the "Start" link in the upper right. Or you can
go to the main Harvesters page (the top Harvesters tab) and this will list your Harvester. From there you can start the
Harvester and monitor and/or stop its progress. When it is finished the Stop link will change to Start.
With the above settings the directory tree of your file system will be used to create the
folder hierarchy in RAMADDA. Every file the Harvester finds will result in a file entry. You can run
a Harvester any number of times and it will only add the new files that it has not seen before.
The File Harvester has many other settings described in more detail here.
If the server machine you have RAMADDA running on has a large collection of data files then you can
very easily provide RAMADDA access to those files with the "Local File View" entry type.
This provides a view of the file system. The Folders and Entries it creates are not in the database but
they provide a way to access the data through all of the RAMADDA services.
For example, on Unidata's motherlode server we have a large archive (10's of millions of files)
of real time data from Unidata's Internet Data Distribution system (IDD). This would be too large
to harvest into the database. Instead, on Unidata's RAMADDA server there is a Local File View entry:
that provides a synthetic view of this large directory tree.
You create one of these entry types through the File->New Entry menu. The create form looks like:
Image 4: Creating a Local File View Entry
You specify a directory path on the server machine (e.g,. /data/ldm/pub) and other configuration options (e.g., what to show, what not to show, etc.)
Note: for security reasons the local file path that is specified has to be a subdirectory of one of the directory
paths specified in the Site Administration page under Settings->Access->File System Access
You end up then with a view of that directory that looks like:
Image 5: Local File View Example