Enhancing Autonomous Driving in Urban Scenarios: A Hybrid Approach with Reinforcement Learning and Classical Control

100%

sensors Enhancing Autonomous Driving in Urban Scenarios: A Hybrid Approach with Reinforcement Learning and Classical Control

Abstract: The use of Deep Learning algorithms in the domain of Decision Making for Autonomous Vehicles has garnered significant attention in the literature in recent years, showcasing considerable potential. Nevertheless, most of the solutions proposed by the scientific community encounter difficulties in real-world applications. This paper aims to provide a realistic implementation of a hybrid Decision Making module in an Autonomous Driving stack, integrating the learning capabilities from the experience of Deep Reinforcement Learning algorithms and the reliability of classical methodologies. Our Decision Making system is in charge of generating steering and velocity signals using the HD map information and sensors pre-processed data. This work encompasses the implementation of concatenated scenarios in simulated environments, and the integration of Autonomous Driving modules. Specifically, the authors address the Decision Making problem by employing a Partially Observable Markov Decision Process formulation and offer a solution through the use of Deep Reinforcement Learning algorithms. Furthermore, an additional control module to execute the decisions in a safe and comfortable way through a hybrid architecture is presented. The proposed architecture is validated in the CARLA simulator by navigating through multiple concatenated scenarios, outperforming the CARLA Autopilot in terms of completion time, while ensuring both safety and comfort.

One. Introduction

The rapid progression of Autonomous Driving has made significant inroads into both industry and academia. It has evolved into one of the most intensively researched fields, experiencing exponential growth in recent years. Despite this, most vehicles currently include only Advanced Driver Assistance Systems as a pathway to achieve full Autonomous Driving. However, the ongoing advancements in Autonomous Driving, coupled with an analysis conducted by McKinsey, indicate that Advanced Driver Assistance Systems and Autonomous Driving combined could potentially contribute a substantial United States dollar three hundred billion to United States dollar four hundred billion to the passenger car market by the year two thousand thirty-five. This insight underscores the profound significance of Autonomous Driving in shaping the future of the automotive industry. Without a doubt, these improvements in Autonomous Vehicles promise to reduce the number of accidents on the road. Over a million people die in traffic-related accidents each year. By removing human factors from vehicle control, Autonomous Driving could significantly enhance traffic safety.

In the field of Autonomous Driving, Decision Making is undoubtedly one of the most critical aspects influencing the vehicle's behaviour. The ability to determine the appropriate action based on the surrounding environment, in a manner that is both safe and efficient, is a fundamental concern in this domain. It is universally recognised that every Autonomous Driving system incorporates a Decision Making module, aimed at minimising human errors in driving tasks. This research is centred on these crucial aspects, to develop a framework for autonomous Decision Making in realistic scenarios within the Autonomous Driving context.

Deep Reinforcement Learning, a subset of Machine Learning, emerges as a promising candidate to help in the decision task. As delineated above, Autonomous Driving challenges within Decision Making offer a clear avenue for exploration. Deep Reinforcement Learning appears to be an ideal tool for the task, given its close alignment with the Partially Observable Markov Decision Process, its adaptability to diverse domains, and its applications in Decision Making engineering. At its core, Deep Reinforcement Learning combines Reinforcement Learning with Deep Neural Networks. By integrating the principles of Bellman's equation with the advanced capabilities of Neural Networks, Deep Reinforcement Learning can analyse environments and infer the optimal decision for a given set of inputs, surpassing the capabilities inherent to traditional Reinforcement Learning. This process occurs through repeated trial and error experiments as humans do, during which an agent incrementally acquires a desired behaviour by continuously interacting with its environment.

However, the application of Deep Reinforcement Learning techniques does not always satisfy the requirements of Autonomous Driving. As mentioned before, the main problem to be solved by Autonomous Vehicles is related to traffic safety, where Deep Reinforcement Learning approaches may have some difficulties due to their primary focus on optimising a task without considering comfort and safety. These techniques are divided into model-based and model-free. Model-based methods incorporate vehicle dynamics but frequently encounter concerns, such as high computational demands and the complexity of creating accurate environmental models, which compromise reliability in unpredictable scenarios like those found in Autonomous Vehicles. On the other hand, model-free methods seek to find the best solution to a specific problem, without taking into account the vehicle's dynamics. By a trial and error process, these methods learn a certain behaviour, which is not usually the safest nor the most comfortable, but that identifies high-level behaviours. This is why we propose a hybrid approach in this work that mixes the best of the Deep Reinforcement Learning and classical control approaches. While a model-free Deep Reinforcement Learning algorithm takes care of choosing the optimal high-level actions, some other classic modules are in charge of safe and comfortable movements defined for the actions, including the vehicle's dynamics.

This work develops a hybrid hierarchical Decision Making architecture which generates steering and velocity signals for navigating in urban scenarios. The adoption of hierarchical systems is widely acknowledged in the academic community; however, unlike hierarchical Deep Reinforcement Learning that presents practical implementation concerns, our proposal uses a classical control system for generating vehicle movement signals, which significantly improves driving smoothness and safety. This makes it possible to develop a realistic approach for driving in various complex urban scenarios, transcending the typical focus on singular use cases found in the existing literature based on Reinforcement Learning. The key contributions of the proposal are summarised as follows:

· Contribution One: This paper proposes a hybrid methodology that integrates multiple components: pre-processing of map information, high-level Decision Making facilitated by the Deep Reinforcement Learning module, and low-level control signals managed by a classic controller. Our approach not only solves individual complex urban scenarios but also handles concatenated scenarios. This contribution is an extension of the work previously published in the conference IV two thousand twenty-three.

· Contribution Two: In this work, a novel low-level controller is developed. This includes a Linear-quadratic regulator controller for trajectory tracking and a Model

Predictive Control controller for manoeuvre execution. The online integration of these two controllers results in a hybrid low-level control module that allows for the execution of high-level actions in a comfortable and safe manner.

· Contribution Three: This study also presents a Reinforcement Learning framework developed within the Car Learning to Act simulator to evaluate complete vehicle navigation with dynamics. Unique to this framework is the incorporation of evaluation metrics that extend beyond mere success rates to include the smoothness and comfort of the agent's trajectory.

Two. Related Works

Two point one. Transformer-Based Reinforcement Learning

Two point two. Attention-Based Deep Reinforcement Learning

Two point three. Combining Machine Learning and Rule-Based Algorithms

Two point four. Tactical Behaviour Planning

Two point five. Discussion

Three. Background

Three point one. Autonomous Driving

Three point two. Partially Observable Markov Decision Processes

Three point three. Deep Reinforcement Learning

Four. Methodology

Four point one. Strategy Level

Four point three. Operative Level

Five. Experiments

Five point one. POMDP Formulation

Five point one point two. Observation

Five point one point four. Reward

Five point two. Evaluation Metrics

Five point three. Concatenated Use Cases Scenario

Five point four. Results

Five point five. Influence of Realistic Sensors

Six. Conclusions and Future Works

Overview

The paper explores a hybrid architecture for Autonomous Driving that leverages both Deep Reinforcement Learning and classical control techniques. It aims to improve the safety and efficiency of decision-making in complex urban driving scenarios, validated through simulations.

Key Points

1A hybrid Decision Making module integrates Deep Reinforcement Learning with classical control methods
2The proposed system navigates various urban scenarios, outperforming existing solutions in terms of comfort and safety
3The framework includes a novel low-level controller for trajectory tracking and maneuver execution
4Simulated tests demonstrate improved performance in navigating complex driving environments
5The research highlights the importance of addressing traffic safety in Autonomous Vehicles through innovative techniques.

Details

Authors: Rodrigo Gutiérrez-Moreno, Luis M. Bergasa, Rafael Barea, Elena López-Guillén, Felipe Arango, Fabio Sánchez-García
Category: Technology and Engineering

PDF
Proficiency of records practitioners using artificial intelligence to enhance records management practices at the Gauteng Department of Education in South Africa
This study examines the proficiency of records practitioners at the Gauteng Department of Education in using artificial intelligence (AI) to improve records management practices. It employs a mixed-methods research approach to evaluate the existing skill levels and proposes a framework for upskilling these practitioners in AI utilization.
PDF
Intelligent Archive Management Based on Deep Learning Technology Driven by Artificial Intelligence
This research article presents a study on an intelligent archive management system (AMS) that employs deep learning (DL) technology to enhance the efficiency and accuracy of processing multimodal data. The proposed model addresses traditional AMS shortcomings such as data classification accuracy and system responsiveness.
PDF
Discharging Records Management Activities Using Artificial Intelligence at the Council for Scientific and Industrial Research, South Africa
This study investigates the application of artificial intelligence (AI) in enhancing records management activities at the Council for Scientific and Industrial Research (CSIR) in South Africa. It provides a framework for utilizing AI to improve efficiency in managing records throughout their life cycle.
PDF
Human-like systematic generalization through a meta-learning neural network
This article explores the capacity of neural networks to achieve human-like systematicity through a meta-learning approach that enhances compositional skills. It presents experimental findings comparing human and machine performance on systematic generalization tasks.
PDF
TBI-603 Network Security & Cyber Law Unit Notes
This document serves as comprehensive unit notes for a course on Network Security and Cyber Law, covering topics like network security practices, types of security attacks, and legal aspects of cybersecurity.