Extended Abstract (292K)P2.42
HDF5—A High Performance data Format for Earth Science
MuQun Yang, National Center for Supercomputing Applications and Univ. of Illinois, Champaign, IL; and R. E. McGrath and M. Folk
The Hierarchical Data Format (HDF) developed at the National Center for Supercomputing Application (NCSA) at University of Illinois at Urbana-Champaign is the primary standard file format for storing data from NASA's Earth Observing System (EOS). However, the original HDF format and library had inherent limitations. For example HDF Version 4 has difficulty supporting huge datasets and does not support parallel computing environments. Since 1999, HDF Version 5(HDF5) was developed to support contemporary high performance computing environments, including datasets larger than 2 Gigabytes, very large numbers of objects in a file, and alternative storage layouts. HDF5 provided chunking storage of the data, which could greatly improve the IO performance. HDF5 was also able to easily to connect with external compression packages and make the data storage more efficient and flexible. Moreover, HDF5 was designed to use Message Passing Interface Input and Output (MPI-I/O), which is capable of performing I/O efficiently in parallel computing environments. HDF5 is available on most important high performance computing platforms.
HDF5 is used in many atmospheric sciences and climate applications. For example, the National Polar-orbiting Operational Environmental Satellite System (NPOESS) will use HDF5 as its principle format. In this poster, we will demonstrate some library and tool features of HDF5.
For more information:
HDF Home Page: http://hdf.ncsa.uiuc.edu HDF Help Desk: hdfhelp@ncsa.uiuc.edu
Supplementary URL: http://hdf.ncsa.uiuc.edu/
Poster Session 2, IIPS Poster Session II
Wednesday, 12 January 2005, 2:30 PM-4:00 PM
Previous paper Next paper