A Web-based Tool for Translating Unstructured Data from Dataloggers into Standard Formats

Field data obtained from dataloggers often take the form of comma separated value (CSV) ASCII text files. While ASCII based data formats have some positive aspects, such as the ability to open them with a text editor or spreadsheet software and "see" the data effortlessly, there are many negative aspects, especially when viewing the situation through the lens of data interoperability and stewardship.

Issues regarding ASCII data and their integration, interoperability, and stewardship have become especially urgent for the NSF-funded next-generation Advanced Cooperative Arctic Data and Information Service (ACADIS) project. The goal of ACADIS is to allow scientists to more easily access, share, integrate and work with Arctic data spanning multiple disciplines. These goals become quite challenging when one considers the large number of ASCII datasets that are either currently part of ACADIS or are being routinely submitted to the project, as those ASCII data are stored in a multitude of layouts, and nearly all metadata reside in non-standard README files, completely disjointed from the actual data they describe.

This talk will introduce the Unidata Data Translation Tool, a web-based service that provides an easy, wizard-based way for data collectors to transform their datalogger generated ASCII output into Climate and Forecast (CF) compliant netCDF files, complete with metadata describing what data are contained in the file, the instruments used to collect the data, and other critical information that otherwise may be lost in one of many dreaded README files. The use of CF compliant netCDF files allows the use of standard services to achieve the many goals of the ACADIS project. However, with the understanding that the observational community appreciates the ease of use of ASCII files, methods for transforming the netCDF back into a user defined CSV or spreadsheet format are also introduced. We anticipate that the Data Translation Tool and the associated service will be of value to a broader community users who have similar needs for transforming the data they have collected or stored in non-standard formats.