A Machine Learning Framework for Comprehensive Thermal Imagery Analysis: An Application for Urban Heat Studies

In a multi-faceted urban heat island study, harvesting and correlating ground-based and satellite-based data is essential for correlating land surface temperature (LST) and air temperature. Typically, ground-based data is of higher-resolution and covers a limited swath of surface area; satellite-based data envelops greater expanses and is of lower resolution. Thermal imagery is one such ground-based measurement: it has a field-of-view similar to a conventional RGB image, but every pixel corresponds to a distinct temperature value. Even still, thermal imagery, taken via FLIR infrared cameras, is limited by its high cost and coarse resolution. This study proposes a machine learning framework to provide two means of analysis: one, using LangSAM, a text-prompt stimulated model based on Meta’s Segment Anything (SAM) model, which creates precise segmentation edits, and GroundingDINO, which associates text strings with object bounding boxes, it allows for segmentation of corresponding visual (RGB) images based on four different feature types—pavement, cement, grass, and bare dirt; two, using a modified Pix2Pix cGAN, based on a Tensorflow backbone which converts conditional information into sketch-based guidance, it allows for image-to-image translation of those feature types’ segment masks after obtaining further dimensionality metadata. Using infrared imagery data from underground subway-based and above-ground city-based field campaigns, spatial-series plots were created across chosen stations and transect locations. Over 10,000 LST temperature values were ascertained for each feature type at each location; more temperature variation—and higher temperatures—were identified for cement during afternoon readings, specifically. Furthermore, unlike previous urban heat studies, this work pioneers deep learning to generate realistic synthetic thermal imagery with the segmented visual images providing novel training data. The framework-based methodology resolves problems previously encountered with a double neural-network system. First, it resolves scaling inconsistencies between individual thermal-and-visual pairs; unless fixed, FLIR infrared cameras typically have a field-of-view offset between each image in a pair; second, it conducts depth estimation on both RGB and infrared imagery; third, it modifies the Pix2Pix cGAN architecture to include a 3D convolutional layer, which allows for an additional input of .csv temperature values corresponding with individual pixels on the thermal image. Future work will utilize newer cGAN models, including a Pix2Pix-turbo model pioneered in 2021; CycleGANs, known for conducting image-to-image translation with unpaired imagery, could bolster efforts to translate urban thermal imagery across various seasons.