Reprocessing of 14 years of GTS data at CPTEC/INPE for distribution

The CPTEC is the Center for Weather Forecast and Climatic Analysis, a division of the INPE, the Brazilian National Institute for Space Research. The institution is a reference for space science, satellite imagery, and environmental studies in South America. Currently the politics of “free data and software” is being strengthened, as the INPExs administration has stated it as a priority for the following years. The institution is the largest distributor of satellite imagery and numerical model outputs in South America. The value of this reprocessing work on the archived conventional meteorological data is to provide a better database of observational data for distribution.

Since the middle of 1995, when the data-processing system of the CPTEC started in an operational way, meteorological data from the GTS, the Global Telecommunications System sponsored by the World Meteorological Organization, began to be received and archived in tapes. These data are mainly the surface and upper-air conventional observations from all the globe, compound by synoptic stations, airports, ships, aircrafts and radiosonde data. The purpose of the operational work is to feed the numerical weather forecast models, the weather forecasters and the researchers, among other uses that had multiplied with the years.

In order to be useful for the users, these data needs to be processed and decoded. The core of the data-processing system from CPTEC/INPE assembled in the first years were a set of FORTRAN programs from the ECMWF pre-processing system that have been modified for local needs. The processed data were converted to WMO's BUFR format and stored in an Oracle database.

Starting at 2003 some efforts were applied to improve the data-processing system. The adopted strategy was to add a second processing suite using decoders from the UNIDATA's GEMPAK package. With this was possible to do comparisons and verifications, and this contributed for improved results. Other initiatives were on the data transmission, that was improved replacing a set of scripts by tools from the UNIDATA/UCAR and from the German Meteorological Service (DWD): The LDM, Local Data Manager; and the AFD (Automatic File Distributor). With the use of LDM the data from the UNIDATA's IDD (Internet Data Distribution) could be received to complement the data received from the Brazilian GTS's Regional Transmission Headquarters (RTH) at the National Institute for Meteorology (INMET).

The reprocessing of our archived data was able to generate a increased number of observations. This is because during the operational processing always there are errors and data-loss that not always can be recovered at the time, and due to several other factors that can be better controlled during a late reprocessing than during the operational daily routine. However, the main reasons for the improvement of the results that we met should be credited to the availability of better decoders and more complete and accurate station tables.

With better decoders and station tables, still was lacking one piece to do the reprocessing work in a more effectively way: a database for the raw GTS data. This issue was addressed in the last two years, when our data-processing system was improved again with the installation of new machines and the addition of a locally-developed database that was designed for the archiving of GTS data. Although this database still be in an experimental form, it made the reprocessing work easier and more controlled. A final step was realized with another database to organize the station tables, improving and completing it.

For the reprocessing work we first recovered all the archived tapes and inserted the data on the database for the raw GTS data. With this step we not only eliminated duplications and wrong bulletins, as we could also control in a very effective way the completeness of our archive identifying periods with lost data or tapes. The second step was the extraction of the data in a form ready to feed the decoders, organizing them by the observation times. These data were processed using a set of modified GEMPAK's decoders to meet local necessities, and the results were made available to the users in GEMPAK format.