14 October 2022 Camera–Lidar sensor fusion for drivable area detection in winter weather using convolutional neural networks
Author Affiliations +
Abstract

Autonomous vehicles rely on perceiving the road environment through a perception pipeline fed by a variety of sensor modalities, including camera, lidar, radar, infrared, gated camera, and ultrasonic. The vehicle computer must perform sensor fusion reliably in various sensor signals degrading environments, which can be performed parametrically or using machine learning. We study sensor fusion of a forward-facing camera and a lidar sensor using convolutional neural networks (CNNs), for drivable area detection in winter driving, where the roads are partially covered with snow, tire tracks obscuring lane markers. Seven fusion models are developed and evaluated for their ability to classify pixels into two classes: drivable and nondrivable. A total of seven models are designed and tested, including camera only, lidar only, early fusion, (three types of) intermediate fusion, and late fusion. The models have accuracies between 84% and 89%, with runtimes between 23 and 61 ms. To select the best model from this group, we introduce a unique metric, named normalized accuracy runtime (NAR) score. A network in a group has a higher NAR score, the more accurate it is, and the shorter its runtime is. The models are evaluated on a winter driving DENSE dataset. Camera and lidar fog noise are synthesized to examine model robustness.

© 2022 Society of Photo-Optical Instrumentation Engineers (SPIE)
Nathir A. Rawashdeh, Jeremy P. Bos, and Nader J. Abu-Alrub "Camera–Lidar sensor fusion for drivable area detection in winter weather using convolutional neural networks," Optical Engineering 62(3), 031202 (14 October 2022). https://doi.org/10.1117/1.OE.62.3.031202
Received: 8 June 2022; Accepted: 25 July 2022; Published: 14 October 2022
Lens.org Logo
CITATIONS
Cited by 6 scholarly publications.
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Cameras

LIDAR

Data modeling

RGB color model

Sensor fusion

Sensors

Data fusion

Back to Top