A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA
To enable autonomous driving in real-world environments that involve a diverse range of geographic variations and complex traffic regulations, it is essential to investigate Deep Reinforcement Learning (DRL) algorithms capable of policy learning in high-dimensional environments characterized by intr...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-06-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/12/6838 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1839654885854806016 |
|---|---|
| author | Yechan Park Woomin Jun Sungjin Lee |
| author_facet | Yechan Park Woomin Jun Sungjin Lee |
| author_sort | Yechan Park |
| collection | DOAJ |
| description | To enable autonomous driving in real-world environments that involve a diverse range of geographic variations and complex traffic regulations, it is essential to investigate Deep Reinforcement Learning (DRL) algorithms capable of policy learning in high-dimensional environments characterized by intricate state–action interactions. In particular, closed-loop experiments, which involve continuous interaction between an agent and their driving environment, serve as a critical framework for improving the practical applicability of DRL algorithms in autonomous driving systems. This study empirically analyzes the capabilities of several representative DRL algorithms—namely DDPG, SAC, TD3, PPO, TQC, and CrossQ—in handling various urban driving scenarios using the CARLA simulator within a closed-loop framework. To evaluate the adaptability of each algorithm to geographical variability and complex traffic laws, scenario-specific reward and penalty functions were carefully designed and incorporated. For a comprehensive performance assessment of the DRL algorithms, we defined several driving performance metrics, including Route Completion, Centerline Deviation Mean, Episode Reward Mean, and Success Rate, which collectively reflect the quality of the driving in terms of its completeness, stability, efficiency, and comfort. Experimental results demonstrate that TQC and SAC, both of which adopt off-policy learning and stochastic policies, achieve superior sample efficiency and learning performances. Notably, the presence of geographically variant features—such as traffic lights, intersections, and roundabouts—and their associated traffic rules within a given town pose significant challenges to driving performance, particularly in terms of Route Completion, Success Rate, and lane-keeping stability. In these challenging scenarios, the TQC algorithm achieved a Route Completion rate of 0.91, substantially outperforming the 0.23 rate observed with DDPG. This performance gap highlights the advantage of approaches like TQC and SAC, which address <i>Q</i>-value overestimation through statistical methods, in improving the robustness and effectiveness of autonomous driving in diverse urban environments. |
| format | Article |
| id | doaj-art-ece2cdc9974b46a498c52d0d00925d5b |
| institution | Matheson Library |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-ece2cdc9974b46a498c52d0d00925d5b2025-06-25T13:26:33ZengMDPI AGApplied Sciences2076-34172025-06-011512683810.3390/app15126838A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLAYechan Park0Woomin Jun1Sungjin Lee2Department of Artificial Intelligence, Konkuk University, Seoul 05029, Republic of KoreaKorea Electronics Technology Institute, Seongnam 13488, Republic of KoreaDepartment of Smart Automotive, Soonchunhyang University, Asan 31538, Republic of KoreaTo enable autonomous driving in real-world environments that involve a diverse range of geographic variations and complex traffic regulations, it is essential to investigate Deep Reinforcement Learning (DRL) algorithms capable of policy learning in high-dimensional environments characterized by intricate state–action interactions. In particular, closed-loop experiments, which involve continuous interaction between an agent and their driving environment, serve as a critical framework for improving the practical applicability of DRL algorithms in autonomous driving systems. This study empirically analyzes the capabilities of several representative DRL algorithms—namely DDPG, SAC, TD3, PPO, TQC, and CrossQ—in handling various urban driving scenarios using the CARLA simulator within a closed-loop framework. To evaluate the adaptability of each algorithm to geographical variability and complex traffic laws, scenario-specific reward and penalty functions were carefully designed and incorporated. For a comprehensive performance assessment of the DRL algorithms, we defined several driving performance metrics, including Route Completion, Centerline Deviation Mean, Episode Reward Mean, and Success Rate, which collectively reflect the quality of the driving in terms of its completeness, stability, efficiency, and comfort. Experimental results demonstrate that TQC and SAC, both of which adopt off-policy learning and stochastic policies, achieve superior sample efficiency and learning performances. Notably, the presence of geographically variant features—such as traffic lights, intersections, and roundabouts—and their associated traffic rules within a given town pose significant challenges to driving performance, particularly in terms of Route Completion, Success Rate, and lane-keeping stability. In these challenging scenarios, the TQC algorithm achieved a Route Completion rate of 0.91, substantially outperforming the 0.23 rate observed with DDPG. This performance gap highlights the advantage of approaches like TQC and SAC, which address <i>Q</i>-value overestimation through statistical methods, in improving the robustness and effectiveness of autonomous driving in diverse urban environments.https://www.mdpi.com/2076-3417/15/12/6838deep reinforcement learningTQCCrossQSACTD3DDPG |
| spellingShingle | Yechan Park Woomin Jun Sungjin Lee A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA Applied Sciences deep reinforcement learning TQC CrossQ SAC TD3 DDPG |
| title | A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA |
| title_full | A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA |
| title_fullStr | A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA |
| title_full_unstemmed | A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA |
| title_short | A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA |
| title_sort | comparative study of deep reinforcement learning algorithms for urban autonomous driving addressing the geographic and regulatory challenges in carla |
| topic | deep reinforcement learning TQC CrossQ SAC TD3 DDPG |
| url | https://www.mdpi.com/2076-3417/15/12/6838 |
| work_keys_str_mv | AT yechanpark acomparativestudyofdeepreinforcementlearningalgorithmsforurbanautonomousdrivingaddressingthegeographicandregulatorychallengesincarla AT woominjun acomparativestudyofdeepreinforcementlearningalgorithmsforurbanautonomousdrivingaddressingthegeographicandregulatorychallengesincarla AT sungjinlee acomparativestudyofdeepreinforcementlearningalgorithmsforurbanautonomousdrivingaddressingthegeographicandregulatorychallengesincarla AT yechanpark comparativestudyofdeepreinforcementlearningalgorithmsforurbanautonomousdrivingaddressingthegeographicandregulatorychallengesincarla AT woominjun comparativestudyofdeepreinforcementlearningalgorithmsforurbanautonomousdrivingaddressingthegeographicandregulatorychallengesincarla AT sungjinlee comparativestudyofdeepreinforcementlearningalgorithmsforurbanautonomousdrivingaddressingthegeographicandregulatorychallengesincarla |