Enhancing Autonomous Driving in Urban Scenarios: A Hybrid Approach with Reinforcement Learning and Classical Control

100%

sensors Enhancing Autonomous Driving in Urban Scenarios: A Hybrid Approach with Reinforcement Learning and Classical Control

Abstract: The use of Deep Learning algorithms in the domain of Decision Making for Autonomous Vehicles has garnered significant attention in the literature in recent years, showcasing considerable potential. Nevertheless, most of the solutions proposed by the scientific community encounter difficulties in real-world applications. This paper aims to provide a realistic implementation of a hybrid Decision Making module in an Autonomous Driving stack, integrating the learning capabilities from the experience of Deep Reinforcement Learning algorithms and the reliability of classical methodologies. Our Decision Making system is in charge of generating steering and velocity signals using the HD map information and sensors pre-processed data. This work encompasses the implementation of concatenated scenarios in simulated environments, and the integration of Autonomous Driving modules. Specifically, the authors address the Decision Making problem by employing a Partially Observable Markov Decision Process formulation and offer a solution through the use of Deep Reinforcement Learning algorithms. Furthermore, an additional control module to execute the decisions in a safe and comfortable way through a hybrid architecture is presented. The proposed architecture is validated in the CARLA simulator by navigating through multiple concatenated scenarios, outperforming the CARLA Autopilot in terms of completion time, while ensuring both safety and comfort.

One. Introduction

The rapid progression of Autonomous Driving has made significant inroads into both industry and academia. It has evolved into one of the most intensively researched fields, experiencing exponential growth in recent years. Despite this, most vehicles currently include only Advanced Driver Assistance Systems as a pathway to achieve full Autonomous Driving. However, the ongoing advancements in Autonomous Driving, coupled with an analysis conducted by McKinsey, indicate that Advanced Driver Assistance Systems and Autonomous Driving combined could potentially contribute a substantial United States dollar three hundred billion to United States dollar four hundred billion to the passenger car market by the year two thousand thirty-five. This insight underscores the profound significance of Autonomous Driving in shaping the future of the automotive industry. Without a doubt, these improvements in Autonomous Vehicles promise to reduce the number of accidents on the road. Over a million people die in traffic-related accidents each year. By removing human factors from vehicle control, Autonomous Driving could significantly enhance traffic safety.

In the field of Autonomous Driving, Decision Making is undoubtedly one of the most critical aspects influencing the vehicle's behaviour. The ability to determine the appropriate action based on the surrounding environment, in a manner that is both safe and efficient, is a fundamental concern in this domain. It is universally recognised that every Autonomous Driving system incorporates a Decision Making module, aimed at minimising human errors in driving tasks. This research is centred on these crucial aspects, to develop a framework for autonomous Decision Making in realistic scenarios within the Autonomous Driving context.

Deep Reinforcement Learning, a subset of Machine Learning, emerges as a promising candidate to help in the decision task. As delineated above, Autonomous Driving challenges within Decision Making offer a clear avenue for exploration. Deep Reinforcement Learning appears to be an ideal tool for the task, given its close alignment with the Partially Observable Markov Decision Process, its adaptability to diverse domains, and its applications in Decision Making engineering. At its core, Deep Reinforcement Learning combines Reinforcement Learning with Deep Neural Networks. By integrating the principles of Bellman's equation with the advanced capabilities of Neural Networks, Deep Reinforcement Learning can analyse environments and infer the optimal decision for a given set of inputs, surpassing the capabilities inherent to traditional Reinforcement Learning. This process occurs through repeated trial and error experiments as humans do, during which an agent incrementally acquires a desired behaviour by continuously interacting with its environment.

However, the application of Deep Reinforcement Learning techniques does not always satisfy the requirements of Autonomous Driving. As mentioned before, the main problem to be solved by Autonomous Vehicles is related to traffic safety, where Deep Reinforcement Learning approaches may have some difficulties due to their primary focus on optimising a task without considering comfort and safety. These techniques are divided into model-based and model-free. Model-based methods incorporate vehicle dynamics but frequently encounter concerns, such as high computational demands and the complexity of creating accurate environmental models, which compromise reliability in unpredictable scenarios like those found in Autonomous Vehicles. On the other hand, model-free methods seek to find the best solution to a specific problem, without taking into account the vehicle's dynamics. By a trial and error process, these methods learn a certain behaviour, which is not usually the safest nor the most comfortable, but that identifies high-level behaviours. This is why we propose a hybrid approach in this work that mixes the best of the Deep Reinforcement Learning and classical control approaches. While a model-free Deep Reinforcement Learning algorithm takes care of choosing the optimal high-level actions, some other classic modules are in charge of safe and comfortable movements defined for the actions, including the vehicle's dynamics.

This work develops a hybrid hierarchical Decision Making architecture which generates steering and velocity signals for navigating in urban scenarios. The adoption of hierarchical systems is widely acknowledged in the academic community; however, unlike hierarchical Deep Reinforcement Learning that presents practical implementation concerns, our proposal uses a classical control system for generating vehicle movement signals, which significantly improves driving smoothness and safety. This makes it possible to develop a realistic approach for driving in various complex urban scenarios, transcending the typical focus on singular use cases found in the existing literature based on Reinforcement Learning. The key contributions of the proposal are summarised as follows:

· Contribution One: This paper proposes a hybrid methodology that integrates multiple components: pre-processing of map information, high-level Decision Making facilitated by the Deep Reinforcement Learning module, and low-level control signals managed by a classic controller. Our approach not only solves individual complex urban scenarios but also handles concatenated scenarios. This contribution is an extension of the work previously published in the conference IV two thousand twenty-three.

· Contribution Two: In this work, a novel low-level controller is developed. This includes a Linear-quadratic regulator controller for trajectory tracking and a Model

Predictive Control controller for manoeuvre execution. The online integration of these two controllers results in a hybrid low-level control module that allows for the execution of high-level actions in a comfortable and safe manner.

· Contribution Three: This study also presents a Reinforcement Learning framework developed within the Car Learning to Act simulator to evaluate complete vehicle navigation with dynamics. Unique to this framework is the incorporation of evaluation metrics that extend beyond mere success rates to include the smoothness and comfort of the agent's trajectory.

Two. Related Works

Two point one. Transformer-Based Reinforcement Learning

Two point two. Attention-Based Deep Reinforcement Learning

Two point three. Combining Machine Learning and Rule-Based Algorithms

Two point four. Tactical Behaviour Planning

Two point five. Discussion

Three. Background

Three point one. Autonomous Driving

Three point two. Partially Observable Markov Decision Processes

Three point three. Deep Reinforcement Learning

Four. Methodology

Four point one. Strategy Level

Four point three. Operative Level

Five. Experiments

Five point one. POMDP Formulation

Five point one point two. Observation

Five point one point four. Reward

Five point two. Evaluation Metrics

Five point three. Concatenated Use Cases Scenario

Five point four. Results

Five point five. Influence of Realistic Sensors

Six. Conclusions and Future Works

Overview

The paper explores a hybrid architecture for Autonomous Driving that leverages both Deep Reinforcement Learning and classical control techniques. It aims to improve the safety and efficiency of decision-making in complex urban driving scenarios, validated through simulations.

Key Points

1A hybrid Decision Making module integrates Deep Reinforcement Learning with classical control methods
2The proposed system navigates various urban scenarios, outperforming existing solutions in terms of comfort and safety
3The framework includes a novel low-level controller for trajectory tracking and maneuver execution
4Simulated tests demonstrate improved performance in navigating complex driving environments
5The research highlights the importance of addressing traffic safety in Autonomous Vehicles through innovative techniques.

Details

Authors: Rodrigo Gutiérrez-Moreno, Luis M. Bergasa, Rafael Barea, Elena López-Guillén, Felipe Arango, Fabio Sánchez-García
Category: Technology and Engineering

PDF
Operating Systems
This document provides an overview of operating systems, including their functions, types, and the essential role they play in managing computer hardware and software resources. It outlines the basic operations of an operating system and explains how they serve as an interface between the user and computer hardware.
PDF
Force, Motion, and Energy: Comprehensive Guide to Mechanics and Electricity
This document provides a detailed overview of fundamental concepts in mechanics and electricity, covering topics such as the laws of motion, force, energy, and electrical circuits. It illustrates these principles with practical examples and problems, making it suitable for educational purposes.
PDF
Introduction to Word Processing
This document provides an overview of word processing, its historical development, and the primary functions of software like Microsoft Word. It discusses key features, objectives for users, and various applications in business, education, and creative writing.
PDF
Wheel-INS2: Multiple MEMS IMU-based Dead Reckoning System with Different Configurations for Wheeled Robots
This document presents a multiple IMUs-based dead reckoning solution for wheeled robots, improving navigation through the integration of IMUs mounted in various positions on the vehicle to enhance accuracy and robustness.
PDF
Wheel-INS: A Wheel-mounted MEMS IMU-based Dead Reckoning System
This document presents Wheel-INS, a dead reckoning system that utilizes a wheel-mounted MEMS IMU to enhance the accuracy of inertial navigation systems for wheeled robots, mitigating errors associated with conventional odometers.