Object detection under bad-weather conditions is a fundamental computer vision task widely used in autonomous robot systems, including self-driving vehicles and autonomous drones. In recent years, vision systems based on RGB sensors, that operate in the visible (VIS) part of the spectra (with wavelengths between 400 nm and 700 nm) have become an essential element for autonomous navigation systems, but it can be affected by rain which limits the propagation of visible light due to the light scattering effects, thus drastically reducing the depth of visibility degrades the quality and clearness of image. In rainy environments, the interaction of light with raindrops causes absorption, reflection, and scattering. This prevents the light from crossing such scenarios and limits the visual depth. These light attenuation effects occur more frequently in the visible spectrum if compared with the near-infrared (NIR) radiation. Due to this fact, the imaging and ranging capabilities of RGB sensors in rain conditions are much lower than in scenarios without the presence of raindrops, fog, or smoke.
In longer wavelengths within the NIR spectra are used, elastic Mie scattering effects occur more frequently. In contrast, Rayleigh scattering effects almost disappear due to the difference between the radiation wavelengths and the sizes of the microscopic particles present in the environment. The atmosphere absorption feature diminishes significantly for this wavelength range.
DIFFUSION MODELS
The diffusion model is defined by a forward process that gradually degradation (the degradation can be randomized or deterministic) data xo~q(x) with noise over the course of T timesteps and through a restoration operator can perform restoration the data (see Fig.1).
Forward diffusion process
For each training data, we add gaussian noise to the sample in T steps, producing a sequence of noisy
samples x1,...,xT, the step sizes are controlled by a variance schedule the forward process is defined form the following Markov chain , with $z_{i}\sim N(0,I)$,$\left \{\beta_{i}\right \}_{i=1}^{N}$ pre-defined noise schedule, $\alpha_{i}=1-\beta_{i}$,and $\bar{\alpha}_{i}=\prod_{j=1}^{i} \alpha_{i}$.
Reverse diffusion process
The reverse process (see Fig.1) requires the estimation of probability density. This means estimating the $q(x_{t}|x_{t-1})$, when t=T, which implicates generating a data sample from isotropic Gaussian noise. Therefore, we shall have to train a neural network model that estimates the $p_{\theta }\left ( x_{0:T} \right )$ based on learned weights $\theta$ and the current state at time t, Eq.~\ref{eq:Reverse_DN}-\ref{eq:mean_DN}~\cite{diffusion_ddn}, where $\mu_{\Theta}$ is the parameterization of the mean~Eq.~\ref{eq:mean_DN}, and $\sum_{\theta}(x_{t},t)$ as variance function. For estimation of $\mu_{\theta}(x_{t},t)$, we apply different ways of training, in this work we use U-net as a neural network, trained to predict the noise $\epsilon$ from the earlier formulation of $q(x_{t}|x_{t-1})$.
Fig.1. Forward and reverse diffusion process of generating a sample by slowly adding and removing blur.
Experimental results
To test the NIR-SPI vision system's capabilities with active illumination in dry and raindrop-rich conditions, respectively, we developed a test bench that has a controlled system of illumination to simulate background light in outdoor conditions, accompanied by a system that can simulate the conditions of rain with raindrop sizes of respectively 2 mm (heavy rainy). In the test, we evaluate the improved image using a GAN network~\cite{GAN_CNN} and our diffusion model (see Fig. 2)
Fig 2. Example recontruction NIR-SPI: comparison Visible and NIR-SPI images obtained measuring distance 30 cm, 60 cm, and 1 m, under a scenario of half-cloudy (15 Klux) and heavy rainy. In the case of NIR-SPI applying GAN and diffusion model improves the image.
Comments