Data Management at the Other End of the Data Life Cycle

Over the past decade, NOAA has devoted time and expertise into developing a “What to Archive?” policy regarding environmental data. NOAA has also made significant progress answering the “How to Archive?” question. Less attention has been paid to assessing the criticality and usefulness of data once it has been archived. How do we define and manage the latter stages of data life cycle, when data falls into inactive (rarely-used) status? What data may be disposed of when the cost of archiving exceeds the value of retention? How can an archive avoid a situation in which the archive is legally required to retain a dataset that has no further scientific value?

NOAA's National Climatic Data Center (NCDC) is working with the National Archives and Records Administration to develop retention guidelines for all data held in the archive, in all formats. This involves the very difficult process of evaluating data holdings for their current and potential usefulness as we seek to better understand earth's climate. Storage advancements have enabled archives to store ever larger data volumes, but the task of maintaining expert understanding of all this data requires extensive resources. Compounding the problem is the loss of expertise as data sets age and providers and stewards are no longer available to support their understanding. Topics covered include strategies to reduce data holdings, such as product regeneration and data sampling in older datasets. The difficulties of anticipating future usefulness of datasets are also explored.