1

Closed

Time Series Data Import

description

General goal: Allow the user to import data from a text file or Excel file.

The import function should be accessible from : File -- Import and from the "Import" ribbon button on the table tab.

Supported file formats for import: text file (tab separated), csv (comma-separated values) Excel (XLS / XLSX), WaterML (only WaterML is currently supported)

In the case of text file, CSV or Excel, the following file structure should be supported:
There must be at least two columns (fields)
One of the fields is the Date / Time
The other fields are the data values

The import should be designed in the form of an "import wizard" with four steps:
Step 1 - Show a file open dialog with the supported formats: .csv, .txt, .xml, .xls, .xlsx, or "all files"

Step 2 - Specify file format options. In case of .txt, choose the file type "delimited" or "fixed width". If "delimited" is chosen, allow user to select the type of delimiter (comma, tab, semicolon, whitespace, other) In case of .xls or .xlsx, allow user to choose the name of the Excel sheet. In this screen a "Data preview" should be shown with the columns and the first few rows. When the user changes the format option the "Data Preview" should be refreshed accordingly.

Step 3 - Specify properties of imported fields.
The dialog has a "Specify Date/Time column" drop-down to allow the user to specify which column is the Date/Time column. note: the default date time column should be selected automatically by checking the first few entries in the file and finding which can be converted to date time

The dialog also has a data preview showing the name of fields and the first few rows. When the user clicks on a field, the input boxes for properties are shown:
* Import this column check-box
* Site - Select an existing site or specify a new site by entering the site name (required), site code (pre-populated as "Site1"), latitude (required), longitude (required), elevation (optional). Under the site selection there is a checkbox "Apply to all columns"

* Variable - Select an existing variable or specify a new variable by setting the variable name (required), variable code (pre-populated as "Variable1"), variable units, data type, value type, time support, time units and no data value. The variable name selection should be in form of a combo box. If there are existing variables in the current project, then the combo box should be populated with them. When the user selects an existing variable, the other properties of the variable are assigned.

* Details - when details is clicked, allow the user to specify the Source, Method and Quality Control Level attributes.

After all properties are specified, proceed with the data import by saving the series and data values to the current project database. During the import the following case can be encountered: There already exists a series with the identical site and variable in the database. In this case, show a dialog: "There is already a time series with site: [Site Name] and variable: [Variable Name]. How do you want to proceed? with options: "Merge with existing time series", "Overwrite existing time series", "Create a new time series".

Closed Sat at 10:21 PM by jirikadlec2
Items fixed in 1.4 stable release are marked as closed

comments

jirikadlec2 wrote Mar 5 at 4:43 PM

All requested features are implemented. Status changed to Fixed.

pamaishlin wrote Feb 28 at 5:42 PM

Thank you for addressing the need to extract offset metadata and values along with datavalues.

jirikadlec2 wrote Feb 27 at 11:27 PM

Please see the time_series_import_design.png attachment for the new changed design of the "Properties of imported fields" step. Clicking "Add" should launch the "Column Properties" dialog.

jirikadlec2 wrote Feb 27 at 10:41 PM

The "offset" option is needed for each data column. It should be added as a new tab to the Column Properties dialog with three properties: Offset Type (text), Offset Units (combo box to select from existing units and a New Unit.. button) and Offset Value (number)

By default the offset is unknown. If If the user specifies the offset properties, then the OffsetType should be saved to the DB and the corresponding OffsetTypeID should be saved with each DataValue from the column.

mogikanin wrote Feb 26 at 8:41 PM

About "offset value". This option is need for each column with data to import? Or it is global for all data from current file?

jirikadlec2 wrote Feb 25 at 4:36 AM

Testing latest changeset - a couple of new issues were found (tested with sample files caslav.dat and idaho_falls.csv)

1. SiteCode should be unique. If I select "New Site" and there already exists a site "Site1", then the default site code should be "Site2"
2. VariableCode should be unique. If I select "New Variable" and there already exists a variable "Variable1" then the default variable should be "Variable2"
3. Same applies for Method Code
4. I need to click "Refresh" in series selector to see my imported time series in the Selection Tool. Expected behavior is: Selection Tool should refresh automatically after import completes.
4. After I complete my import and try to show the graph, the graph shows a message "All data is censored, so there is no data to display" although censor code is "nc".
5. After I complete my import, click "Refresh" on SeriesSelector and check the map, no sites are shown in the map. Expected behavior is: A new point layer "Imported Data" with the imported series (1 point for each time series) should be added to the map.
6. After completing import, going to "Edit View" and selecting "Edit Series", a message appears "The selected series has no curve".

7. (this was in the initial specs): Also add an Import.. menu item to the File menu (use SimpleActionItem with Key=HeaderControl.ApplicationMenuKey)

8. In the variable tab, default NoDataValue should be -9999.

9. New requirement. In the "Format options" step, add an option: "Start import at row:" so that user can specify the first row with data values

10. In the "Properties of imported fields" step, change the text to "Right-click on a column to specify properties."

pamaishlin wrote Feb 23 at 4:18 PM

Please include 'offset value' as a critical descriptor for time series data such as soil moisture wherein sensors are buried as various depths.

Sign in to add a comment