IEDRO's high resolution Automated Chart Digitizing Process (ACDP) for the development of worldwide environmental databases

Introduction

This paper summarizes a new non-profit organization within the international data rescue community. It also describes their computer program recently developed to digitize scanned images of analog precipitation strip charts, providing the output at virtually any time interval. The program provides the digital output from a scanned image in 10 seconds per chart instead of the 10 to 15 minutes taken by manual methods using digitizing tablets and light tables.

The organization: The International Environmental Data Rescue Organization [IEDRO]

IEDRO is a new organization whose primary mission is to assist foreign countries to locate, rescue and digitize their deteriorating environmental data. The acronym is derived as follows:

International - any country in the worldwide community that needs help getting started with organizing, implementing and preventing their valuable data from deteriorating and being lost forever is eligible for assistance.

Environmental – includes not only meteorological, oceanographic, hydrological, and glacial data, but also other types of data that relate to the environment.

Data – is defined as conversion of any scripted and autographic records presently archived on deteriorating medium such as paper, photographic (microfilm, microfiche, glass slides, etc.), magnetic, etc.

Rescue – is to convert the data forms into digital images and secondly into a standard ASCII format that can be easily read by computers.

Organization – a Non-Profit 501(c)(3) group whose primary mission is to aid and assist in data rescue programs worldwide in any way possible.

Examples of existing data formats being evaluated by IEDRO for future rescue include relative humidity, temperature, wind movement and wind gust data, station pressure, solar radiation (pyroheliometer and incoming solar radiation (insolation)) data and sea ice coverage.

Current data rescue projects initiated by IEDRO are:

1. The manual keying efforts already ongoing in cooperation with NOAA's National Climatic Data Center's (NCDC's), Climate Data Modernization Program (CDMP) for: 7 countries in Africa (Kenya, Malawi, Mozambique, Niger, Tanzania, Senegal and Zambia,) 2 countries in South America (Chile, and Uruguay), and the Dominican Republic in the Caribbean.

2. Automated Daily Precipitation Charts from Malawi are being digitized in cooperation with The Malawi Meteorological Agency. These are continuous recording strip charts containing 24 hours of data. Thus far IEDRO has digitized over 1000 Malawi strip charts covering about three years with many more years to go. The archived data will be converted into 5-minute precipitation intervals.

How IEDRO Works

I. First IEDRO must be approached by the country and asked to consider their request.

II. The country must also furnish a written request for assistance that includes:

A. Data types to be rescued (surface, upper air, hydrologic, glacial, etc.)

B. Periods of Record and frequency of observation.

C. Current storage medium (paper, microfilm, microfiche, magnetic tapes, autographic strip charts, etc.) If possible furnish samples of the medium records and pictures.

D. Approximate volume (number of pages, reels of film, number of microfiche, etc.)

E. Current condition of the medium to be “rescued”.

III. IEDRO will evaluate the request and advise as to feasibility of rescuing the data.

IV. IEDRO will attempt to obtain funding before proceeding with data rescue

V. Although the paper summarizes the rescue of alphanumeric data, the focus is the IEDRO “Automated Chart Digitization Program (ACDP)” developed by Dr. Ed Root for use on environmental autographic strip charts. The general “artificial intelligence techniques used in this process will be briefly discussed in this paper.

Phase One – Data Acquisition & Raw Data Digitization

The charts are received at IEDRO Headquarters and scanned by a Fujitsu FI-6130 Sheetfed Scanner. The images are transferred to the IEDRO server. Each scanned chart is quality controlled, metadata and dates recorded. The original charts are returned to the country of origin. The scanned images are prepared in batches for digitizing with the exception of charts showing zero precipitation during the period. Those are placed in another file not requiring input to the digitizing program. Those charts indicating measurable precipitation are run through the ACDP with the digitized data output into an appropriate data file. IEDRO volunteers then examine and manually digitize problem charts…those which the ACDP was unable to digitize. The rejection may be due to handwriting across the trace, an interrupted trace due to the recorder running out of ink, etc. The output digitized data files are quality controlled for accuracy. This ends Phase One – Data Acquisition & Raw Data Digitization

Phase Two – Data Conversion & Archival Preparation

The output files are converted to an appropriate time interval of data (i.e. every 5 minutes) with files grouped on a daily basis (i.e. five minute increment data for one date assembled together). Additional output files grouped into “precipitation events” where beginning and end of precipitation events are tallied with the maximum precipitation total identified. The digitized data are then sent to the country of origin as well as to the NOAA National Climatic Data Center for archiving. A detailed description of the procedure is reserved for a more technical paper to follow in a subsequent AMS journal article