Building Custom Neural Net Models to Classify Coastal Imagery

This talk describes the use of machine learning (ML) to classify types of shoreline areas based on aerial imagery. Our methodology lets us predict the Environmental Sensitivity Index (ESI) of shorelines within the images, specifically how sensitive a particular type of shoreline would be to an oil spill. We compared two different types of aerial photographs: oblique and orthorectified. For oblique shoreline photos, field of view (FOV) areas of each photo were spatially modeled and all ESI shoreline values that were within the modeled FOV area were joined to the photos. For orthorectified aerial photos, a grid was overlaid on the imagery and ESI shorelines; both were extracted for each grid cell and joined together.

To build our classification models, we used Google Cloud AutoML Vision which allows to build accurate custom models from a relatively small image dataset by leveraging transfer learning and Neural Architecture Search. We first experimented with a single-label version of the oblique image set with the label referred to a single primary shoreline type included in the image. The AutoML UI allows easy visual inspection of model training and evaluation results. From inspection of the false positives we were able to determine that this first model often predicted coastline types that were present in the image but did not match the single label. It seemed likely that this dataset was a better fit for a multi-label classification approach. AutoML also supports multi-label classification, and so we generated a multi-label version of the same dataset, and trained a new model. This multi-label model showed significantly higher accuracy than the first, the single-label model.

We also compared model accuracy across the two different types of image datasets. We found that prediction accuracy using oblique imagery models was higher than that for orthorectified imagery models. We attribute the oblique imagery models' higher performance to the larger geographic coverage of the oblique imagery, and the inclusion of vertical information in these images.