Space-based lidar denoising methods and validation with focus on CALIOP

Photon limited noise severely impacts the ability of lidars to accurately retrieve geophysical variables such as aerosol and cloud extinction [1/km], and water vapor mixing ratios [g/kg]. For space-based lidars such as CALIOP and CATS the extinction accuracy requires high signal to noise ratio (SNR) in order for the residual noise to be proportionally less than the extinction signal. Currently the standard method to achieve higher SNRs for existing lidars is to down-sample the observations which has the consequence of introducing biases in the measured extinction coefficient (coeff.) due to non-uniform aerosol and cloud structures and non-linearity of the single scatter lidar equation. Future space-based lidars, regardless of architecture such as HSRL or DIAL, will have similar limitations due to noise since it is costly to increase the SNR via more performant optical hardware.

The lidar community identified the need to increase the SNR of photon limited lidars via advance statistical signal processing methods and machine learning methods to reduce the uncertainties of geophysical variables due to noise. The performance of various denoising methods have been explored such as optimal estimation, quantum recurrent neural networks, convolution neural networks, penalized maximum likelihood estimation, etc.

In this presentation we will share work that has been done using penalized maximum likelihood estimation methods, such as Poisson Total Variation (PTV) [1, 2] and proxbm3d [3], to denoise CALIOP measurements. These denoising methods, originally developed for denoising photon counting medical images, separates the extinction from the photon limited noise by making a distinction between stochastic processes of the photon and solar background noise and the coherent spatial-temporal extinction image structure. This separation between signal and noise is achieved through a theoretically supported mathematical optimization framework in which a cost function is minimized that includes a photon detector noise model and the single scatter lidar equation.

We will specifically share the strength and weaknesses of the denoising methods that we are developing in denoising CALIOP observations, how we validate the denoising accuracy of our methods via simulations and coincident HSRL2 measurements, and future plans of how we will resolve the weaknesses of our denoising methods. We will also share obstacles that we had to face when we applied our methods on real CALIOP data. Finally, we'll share constructive insights that will be relevant for denoising future space-based lidar observations.

[1] - Marais, Willem J., Robert E. Holz, Yu Hen Hu, Ralph E. Kuehn, Edwin E. Eloranta, and Rebecca M. Willett. "Approach to simultaneously denoise and invert backscatter and extinction from photon-limited atmospheric lidar observations." Applied optics 55, no. 29 (2016): 8316-8334.
[2] - Marais, Willem J., and Matthew Hayman. "Extending water vapor measurement capability of photon-limited differential absorption lidars through simultaneous denoising and inversion." Atmospheric Measurement Techniques 15, no. 17 (2022): 5159-5180.
[3] - Marais, Willem, and Rebecca Willett. "Proximal-gradient methods for poisson image reconstruction with bm3d-based regularization." In 2017 IEEE 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pp. 1-5. IEEE, 2017.