Thursday, 13 January 2005: 9:00 AM
Networked synchronization of netCDF datasets
Poster PDF
(15.0 kB)
It is now universally expected that scientists in the geophysical data community will have remote access to scientific datasets. Many of these datasets are constantly changing as new information is gathered or generated. The most commonly used protocols for accessing these datasets are ftp, http, and OPeNDAP. All of these protocols assume that the client has uninterrupted, high speed access to the Internet.
In today's world of mobile computing, it can no longer be assumed that clients will have uninterrupted, high speed network access. Further, even if a client has high speed access, network latency in wireless networks can be much greater than in traditional wired connections. One alternative to traditional network based access to datasets is to mirror a remote dataset on a local machine.
This paper describes cdfsynch, an extension to the rsync algorithm that is optimized for efficient mirroring of netCDF datasets over a high latency, low bandwidth network link.
Supplementary URL: