This project has moved. For the latest updates, please go here.

Metadata Fetcher Improvements

Jan 13, 2010 at 7:58 PM

Here's a Discussion about improvements to the metadata fetcher plugin.  This plugin harvests metadata about wateroneflow services so that searches can be made for data on those services.  This is an alternative to relying on HIS Central to assist in searches.  The plugin stores data in a local metadata cache database, with info on sites, variables, and series (e.g., time period and number of records). 

Currently, if the plugin encounters an error during harvesting, the harvest stops and nothing is written to the database.  Is this the best behavior.  Or would it be better if the plugin skipped a bad site and kept searching to populate the database with as much data as possible?  If the latter is desired, then how do we flag a service as having only been partially harvested?  Does that flag appear in the record for the service, for the site within the service, for the series, etc?  Should there then be a separate function that performs a harvest only for items that aren't yet completely harvested? 

Another thing we could do is add the ability to query the new HIS Server, e.g.,

http://icewater.usu.edu/HISServerCapabilities/

It looks like the GetWaterOneFlowServiceInfo method may give the metadata that we're trying to harvest.  If we add the ability to harvest from HIS Server like this, do we always trust the service, or do we query just for a list of wateroneflow services, and then harvest data for each service individually?

I envision this ability to query an HIS Server to be its own menu item for the plugin.

Thoughts?

Jan 14, 2010 at 3:24 PM

The HIS Server Capabilities web service will provide the list of WaterOneFlow services on the server.  To harvest the metadata for each service, you will have to hit each one individual (as you are doing now).  We plan on the HIS Server Capabilities service being part of the standard software stack for HIS Server and so we are open to feedback about the methods and what is returned.

Jan 14, 2010 at 4:21 PM

The metadata fetcher plugin can now get a list of wateroneflow services from HIS Server.

I still haven't come to a conclusion about the best way to handle web method request failures, and could use some guidance on that.