Description:

Given the expansive breadth and depth of climate, weather and built and social environment geospatial data sources, a challenge of assessing physical climate risk is capturing and associating all of the data sources for descriptive or analytical insights. Geospatial data sources originate from a variety of data sources (GeoTIFF, netCDF, grid, ASCII, Shapefile, File Geodatabase, .csv, etc.) and require a variety of Python and other libraries to work with (GDAL/OGR, Rasterio, xarray, shapely, Fiona) for reading, writing, transforming, geoprocessing and analysis which adds friction to the path to data enrichment and insights. The Data Mesh is the concept of decentralized catalogued central metadata store with a common query interface that allows for teams to query data in-place from catalogued data sources without the need to duplicate data in a central data warehouse. On AWS, a Data Mesh may be constructed using the AWS Glue Data Catalog for registering data into the catalog, governance with AWS Lake Formation/Amazon DataZone and Amazon Athena for querying data and AWS Lambda and API Gateway can be used for querying and access. Registering or accessing the SpatioTemporal Asset Catalog (STAC) in the data catalogue serves as a first layer of querying for identifying raster files to access to query against. Registration or a common query interface such as pystac-client of the STAC catalog supplies the bounding boxes, temporal information and other metadata and helps point to the raster files that may be accessed with User Defined Functions (UDFs) in Amazon Athena. Using Amazon Athena UDFs for custom queries or querying the raster data model to invoke AWS Lambda queries for point samples and zonal statistics using geospatial Lambda Layers (GDAL, Rasterio). Using a common access query interface in a python wrapper for Athena and/or API, Data Mesh users can supply queries for single coordinate locations, collections of asset coordinates as well as zonal or regional queries to capture historical, near real-time and forward-looking climate and weather prediction data. The aim of this Data Mesh architecture and a simple access pattern python library is to reduce the complexity and time of collecting climate, weather and built and social environment geospatial data sources so that users can rapidly enrich location data with rich climate and weather data. The aim of the talk is to present the Infrastructure as Code (IaC) and Python library for querying the Data Mesh as well as key learnings and challenges developing the architecture and library.