J3.3 SmartMet Server – Providing MetOcean Data

Monday, 23 January 2017: 11:30 AM
611 (Washington State Convention Center )
Roope Tervo, Finnish Meteorological Institute, Helsinki, Finland; and M. Heiskanen

SmartMet Server is a data and product server for MetOcean Data used by Finnish Meteorological Institute (FMI). It provides high capacity and availability services for M2M and user interfaces. SmartMet Server has been optimized for on-demand queries and fast response.

Several output formats and interfaces as WMS, WFS 2.0, JSON, ASCII, HTML etc. are provided. WMS and WFS 2.0 services are European Union INSPIRE-directive compliant and have been basis of both FMI INSPIRE and Open Data portal since 2013.

Server architecture is modular, scalable and extensible. It consists of frontends, which work as a load balances and product cache, and backends, which provide data products.  The architecture is plugin-based. Engines provide shared access to the data and plugins provide APIs for requesting the products.

Accessing gridded data is based on memory-mapping. A grid data engine does the memory-mapping to NWP data and provides interpolated data for requested area and time to plugins. The server has also some on-demand post-processing capacity like DEM correction. Point data like weather observations are typically stored in and provided from databases. The server contains local databases for most used data to achieve better scalability.

The frontend product cache contains two phases mapping of products. ETags consisting of request url, product configuration and used data id works as a product hash, which are mapped to a buffer hash containing the actual product. This way for example empty radar images can all be mapped to as one cache entry. The cache is LRU cache where most used content is stored in server memory and rarely used content on local disk.

The most important plugins are WMS providing meteorological Web Map Service, WFS 2.0 providing time series data for locations, Download-plugin providing access to the binary data and Q3 providing service to process the data. The WMS plugin is based on SVG rendering model. It creates contours from data and writes them as SVG image, which can then be rendered to as bitmap image by librsvg-library.

The server has been operative since 2008. SmartMet Server was originally developed for FMI client services, and since 2013 it’s been basis of FMI Open Data Portal. At the moment operative client server provides over 30 million requests per day, about 200 requests per second as a baseline with peaks over 650 requests per second. The operative server contains approximately 2 TB of data with update interval varying from 1 minute to 6 hours. End user availability has been over 99,95 %. Average response times for point data are about 25 ms per request.

SmartMet server is going to be published as open source in 2016 under MIT license.

- Indicates paper has been withdrawn from meeting
- Indicates an Award Winner