Improving Road Surface Classification with Co-Training Algorithms

Adverse winter weather conditions cause billions of dollars of damage yearly in the United States. Winter weather events often affect road safety and impede transportation. For this reason predicting the road-surface conditions is of high importance for our infrastructure and safety. In order to provide motorists with up-to-date information to make travel decisions with, the New York State Department of Transportation (NYSDOT) has cameras positioned along the state's roadways which take pictures of the road surface at regular intervals. From a machine learning perspective these images represent a large repository of data that can be used to automatically assess risks due to weather conditions on the road. By labeling a small portion of them and using the big unlabeled portion we train artificial neural networks that follow the DINO architecture in a semi-supervised way. Importantly to us, cameras that are located in nearby locations often observe lengths of road that are impacted by similar winter weather conditions. We use the similarity between these two "views" of the same road as the inputs to two different models, and then we apply "co-training" style self-supervised learning algorithms to train both of these models jointly, and then use their joint prediction. We find that individual models in the pair are more accurate than those trained using a single view, and the joint prediction of both models is more accurate than either model's individual prediction. In particular we use co-training and meta co-training and the predictive accuracy that we achieve on classification is 98.02% and 98.63% respectively.

Our choice of co-training style algorithms mitigates several problems for our data source. The simplest issue that commonly occurs is that a camera can be obstructed or nonoperational for a time, and still being able to use the model for the remaining view is preferable to not being able to make any prediction. Another important limitation is the number of labels required to make accurate predictions. Using the combination of self-supervised learning and co-training algorithms we are able to make accurate predictions with very few labels which is important because there are many locations around the state of New York in which it is important to monitor road surface conditions, and providing many expert labels for each is difficult. Finally, we find that calibrated model predictions are important to the performance of our co-training algorithms, and as a result we use ensembles as our models, which give calibrated uncertainties for their predictions. This is an important step toward building end-user trust. These properties and design decisions in creating our co-training algorithms highlight the benefits over more traditional deep learning approaches.

We study the performance of two different types of co-training algorithms from the machine learning literature: the original co-training algorithm proposed by Avrim Blum and Tom Mitchell, and a newer algorithm, called meta co-training, by Jay Rothenberger and Dimitrios Diochnos. Co-training style algorithms are semi-supervised; that is, they use unlabeled data to boost the performance of supervised models. More specifically, these algorithms make use of pseudo labels which are model predictions that are treated as labels for unlabeled points. In the case of co-training style algorithms the pseudo labels are generated by the model for one view, and then assigned to the corresponding instance of the other view. In this way one model helps the other model predict more accurately by making use of unlabeled data.

We performed an evaluation using a dataset of 4303 images each from two different cameras located within a quarter mile of each other. For each image taken by one camera we had one for the other which was captured within two and a half minutes of the other. We reserved 20% for a testing set, then the remaining data were used to create nineteen labeled dataset and unlabeled dataset pairs corresponding to discarding from 5% to 95% of the labels in 5% increments. In every case, after excluding the testing data (which is 20% of the original 4303 images), we used 25% of the remaining labeled data in order to train the two models and the remaining 75% of the labeled data was used for validation. These nineteen dataset pairs were designed to illustrate how the method's performance changes with access to more or less labeled data.

We first performed a set of experiments using the ResNet-50 architecture to evaluate the performance of co-training versus fully supervised training. For each labeled-unlabeled dataset pair we trained five pairs of ResNet-50 models using co-training. In all cases, we find that co-training increases the predictive accuracy with respect to the testing set with the performance trending upwards as the model is given access to more labeled data. We also evaluated the performance of an ensemble of multi-layer perceptron models trained on embeddings generated by a pre-trained DINO vision transformer. For each of the nineteen data dataset pairs we trained five ensembles of twenty five members using different random seeds for initialization. By using embeddings, we are able to achieve higher predictive accuracy compared to using images, with the performance gains most noticeable when working with limited labeled data. Using the ensemble models coupled with pre-trained embeddings increases testing set accuracy by at most 23.14% with the fewest number of labeled data and at least 0.26% with the greatest number of labeled data.

Using the ensemble of models that use the DINO representation as input that were found to be most effective in the co-training experiments, we performed a similar set of experiments to evaluate the relative performance of meta co-training. For each of the nineteen data dataset pairs we trained five ensembles using different random seeds for initialization. These ensembles had the same architecture as those for the co-training experiments as well as access to the exact same labeled-unlabeled dataset pairs. On all data dataset pairs meta co-training out-performed co-training.