12JCSDA Deep Learning Methods for Damage Estimation using Integration of Building Data and Satellite Imagery

Hurricanes, a form of tropical cyclones, cause storm surges, high winds, heavy rains, and floods. The hazard events damage or destroy vehicles, buildings, bridges, and other infrastructure, turning loose debris into deadly flying projectiles. The frequency and intensity of extreme events such as hurricanes are increasing, so as the demand for the trained workforce to inspect and assess the extent of the building damage sustained by these events. The required time for damage estimation can be potentially reduced by implementation of automated or semi-automated decision support systems based on machine learning algorithms that employ, extract, process, and analyze the collected data. Data assimilation is an essential phase for developing machine learning algorithms to model the response variable for a given application.

Researchers and disaster management professionals from multiple local, state, and federal agencies collect a large quantity of useful and time sensitive data in a post hurricane situation. Due to the non-unified structure of responsibility between the institutions and agencies acting post disaster, much of this data is collected and stored in a nonuniform and scattered way. To combine and integrate all relevant sources and data from these multiple agencies with varied reporting mechanisms is a challenging task.

We have invested much effort to find the optimal platform and database type for the integration of these various data and sources. We have built an integrated database for hurricane Michael by identifying, extracting, and assimilating hazard, exposure, and reconnaissance data. We are extending the database by incorporating satellite imagery to facilitate the development of machine learning algorithms for damage estimation.

Machine learning (ML) methods for damage assessment can facilitate the disaster response by federal, state, and local governments and expedite insurance claim processing. ML algorithms rely on measured, observed, and collected data to model and predict the response variable. In turn, machine learning methods can be implemented for damage estimation using assimilated data. The objectives of this project are listed below.

1. Identify the relevant and publicly available satellite imagery pre- and post-disaster (hurricane Michael).
2. Extend and modify the data structure of the current integrated database to host the identified satellite data in part 1.
3. Augment the integrated database by pre- and post-disaster satellite imagery.
4. Annotate the satellite data for training phase.
5. Implement a deep learning methodology to estimate the building damage and classify it to no, moderate, and extensive damage.
6. Train and validate the implemented method.

The automation of the learning process from images and the classification of damage seen on the images is a challenging task. The development of an automated technique to classify the overall damage state can be performed using either a binary classifier or a multiclass classifier that assign a probability to each damage class (from no-damage to total) for each building.

An image recognition model will be trained to input reconnaissance images of the building and to output its damage ratio. In addition, the model may also output a confidence score related to how certain the damage ratio is for a building. In general, deep learning architectures suitable for image recognition are based on variations of convolutional neural networks (CNNs). The implemented CNN is applied to classify the contents of different images. Reconnaissance images from past hurricane events would be used to train the model. The value of this model is that even before the reconnaissance team inspects the buildings that are impacted by a hazard event, a quick estimation of the overall damage will be obtained using satellite imagery. We first assimilate the satellite imagery with the constructed database that contains reconnaissance, hazard, and exposure data. We then implement the deep learning method for damage estimation using the assimilated dataset. This method provides the following benefits.

• The catastrophe modelers will potentially be able to use it for damage estimation, and compare, validate, and calibrate their models to improve their loss predictions.
• Manufacturers will be able to use these models to test and improve mitigation techniques and study the behavior of buildings during hurricanes.
• Insurers will benefit from a better estimate of natural disasters footprints leading to better decision making.

The building data is assimilated within our integrated database for impacted area of hurricane Michael. Depicted in Figure 1, we see the geospatial building data obtained from National Structure Inventory (NSI) for the Area impacted by hurricane Michael. Two counties, Bay and Gulf counties were hit by hurricane Michael in 2018. To implement a deep learning method for classification, post-event images must be annotated/labeled to represent each specific class. Figure 2 shows drone images that are georeferenced and integrated in the database. Annotations for different damage classes using a binary classifier (damaged or not), or a multi-class classifier (from no damage to moderate to severe damage) are shown in this figure. Pre- and post-event satellite imagery can be effectively used for scaling up and expediting the process of damage estimation for a large area. Figure 3 shows pre- and post-hurricane satellite imagery taken from the impacted area. Buildings with detected roof damage are marked using red rectangles.