2020

Han Wang, University of California, Berkeley
Zhang Jian, Southeast University
Shuyang Dong, Southeast University
Fang Tang, Southeast University

We propose the Delayed Reward's Multi-Agent Arterial Signal Control (DMAS) method . Reinforcement learning(RL) is widely utilized in the study of intersection signal control methods and exhibits excellent performance in improving traffic efficiency and reducing fuel consumption, mainly for individual intersections. However, to achieve the coordination in the multi-intersection arterial traffic signal control (ATSC) system, RL-based control methods must confront the training difficulty of multi-agent systems, like the curse of dimensionality problem, which is aggravated by the delayed-reward property( integral traffic flow state does not change instantly with the signal operation) of the multi-intersection artery. Based on the topological structure of the communication network of the ATSC system in the CV environment, we adopt the multi-agent actor-critic training regimen referring to MADDPG( 1 ), wherein single signal controllers play the role of actor agents, and the center controller operates as an valuator integrated with multiple critic agents. Considering the delayed reward property, we ameliorate the MADDPG by embedding the return decomposition module whereby DMAS transfers the delayed reward into immediate reward. We introduce a dynamic delayed reward prediction model to implement the contribution analysis of RUDDER( 2 ). The total delayed reward is assigned to each step according to the prediction value difference between adjacent steps. The simulation results show that the average stop times of artery vehicles decreases by 34.3%, the average waiting time decreases by 26.8%, the average delay decreases by 27.6%, and the time-space diagram demonstrates the superiority of DMAS on solving ATSC problems.

Qiangqiang Guo, University of Washington
Xuegang Ban, University of Washington

As the rise of connected vehicles (CV), it becomes practical to train learning-based methods to better control traffic signals by utilizing the CV data. Reinforcement learning (RL) has shown great promise in agent-environment interactive optimal control. Current RL based signal control methods usually assume 100% CV penetration to get the full knowledge of the environment and simplify the intersection scenario to reduce the state and action spaces, which make these methods not practical. This paper proposes a new RL-based signal control algorithm considering a real-world four-leg twelve-movement intersection and study the control performance under limited CV penetration. The positions, speeds, and accumulated waiting times are normalized and spatially discretized to represent the state in RL. The average waiting time and penalty of phase transitions are weighted up as the reward in RL. A two-stage deep Q-learning method is adopted to learn the Q-function. The numerical experiment results show that the proposed RL-based signal control algorithm could reduce the average queue length, average waiting time, and average time loss by up to 32.85%, 32.37%, and 19.14%, respectively. The relationship between CV penetration and RL-based control performance is also analyzed.

Yun Yuan, University of Utah
Xianfeng Yang, University of Maryland, College Park
Hao Wang, Shanghai Institute of Technology
Tian Zhao, University of Wisconsin, Milwaukee
Yang Liu, Nanjing Agricultural University

To contend the considerable externalities of traffic congestion, researchers and engineers have made great efforts to improve traffic-responsive algorithms for Adaptive Traffic Signal Control (ATSC). Deep reinforcement learning methods have been tested on simplified traffic light timing problem and show promising potentials in addressing the curse of dimensionality. However, previous studies ignore the limitation of sensors and heavily rely on simulators. To address these issues, this paper employs a Deep Q-learning Learning (DQL) method for the traffic signal timing control problem at isolated intersections. A Deep Q Network (DQN) is proposed to estimate the value function. To accelerate the training computation, a deep model is investigated to extract features from flawed data and predict the traffic status, which outperforms the artificial data simulators regarding the running time. This study tests the proposed methods on a real-world case with Automatic Number-Plate Recognition data. Experiments show the deep model can predict the traffic state with much shorter computational time and the deep Q-learning algorithm is 3.9% and 22.0% better than the SCOOT adaptive control system and the time-of-day plan by SYNCHRO, respectively.

Yaobang Gong, University of Central Florida
Mohamed Abdel-Aty, University of Central Florida
Jinghui Yuan, Oak Ridge National Laboratory
Qing Cai, Waymo

Adaptive traffic signals improve traffic efficiency, but their safety effects vary among different implementation strategies. To improve the traffic safety of signalized intersections, this study proposes a safety-oriented adaptive signal control algorithm to optimize traffic efficiency and safety at the same time. A multi-objective deep reinforcement learning framework is utilized as the backend algorithm to solve the discrete optimized problem. The proposed algorithm was trained and evaluated on a simulated isolated intersection built based on field traffic data. A real-time crash prediction model is calibrated to provide the safety measure. The performance of the algorithm was evaluated by the real-world signal timing provided by the local jurisdiction. The evaluation results showed that the algorithm improves both traffic efficiency and safety compared with the benchmark. Besides, compared with an adaptive traffic signal optimizing only traffic efficiency, the proposed algorithm did improve traffic safety with the negligible deterioration of traffic efficiency. To the best of the authors’ knowledge, the proposed algorithm is the first successful attempt in developing adaptive traffic signal system optimizing traffic safety.

Zhenning Li, University of Macau
Hao Yu, University of Hawai'i, Manoa
Guohui Zhang, University of Hawai'i, Manoa
Tianwei Ma, Texas A&M University, Corpus Christi
Jun Wang, East China Normal University

Inefficient traffic signal control methods may cause numerous problems, such as traffic congestion and waste of energy. Reinforcement learning (RL) is a trending data-driven approach for adaptive traffic signal control in complex urban traffic networks. Although the development of deep neural networks (DNN) further enhances its learning capability, there are still some challenges in applying deep RLs to transportation networks with multiple signalized intersections, including non-stationarity environment, exploration-exploitation dilemma, multi-agent training schemes, continuous action spaces, etc. In order to address these issues, this paper first proposes a multi-agent deep deterministic policy gradient (MADDPG) method by extending the actor-critic policy gradient algorithms. MADDPG has a centralized learning and decentralized execution paradigm in which critics use additional information to streamline the training process, while actors act on their own local observations. The model is evaluated via simulation on the Simulation of Urban MObility (SUMO) platform. Model comparison results show the efficiency of the proposed algorithm in controlling traffic lights.

Jorge A. Laval, Georgia Institute of Technology
Hao Zhou, University of South Florida

This paper uses supervised learning, random search and deep reinforcement learning (DRL) methods to control large signalized intersection networks. The control policy at each intersection is  parameterized as a deep neural network, to approximate the ``best'' signal setting  as a function of the state of its incoming and outgoing approaches.  The traffic model is Cellular Automaton rule 184, which has been shown to be a parameter-free representation of traffic flow, and which is probably the most efficient implementation of the Kinematic Wave model with triangular fundamental diagram.  We are interested  in the steady-state performance of the system, both spatially  and temporally: we consider a homogeneous grid network inscribed on a torus, which makes the network boundary-free,  and drivers choose random routes.   As a benchmark we use the longest-queue-first (LQF) greedy \citep{arel2010reinforcement} algorithm. We find that: \begin{enumerate*}[label=(\roman*)]     \item a policy trained with supervised learning with only two examples outperforms LQF,       \item random search is able to generate near-optimal policies,       \item the prevailing average network occupancy during training is the major determinant of the effectiveness of  DRL policies. When trained  under free-flow conditions one obtains DRL policies that are optimal for all traffic conditions, but this performance deteriorates as the occupancy during training increases. For occupancies $> 75\%$ during training, DRL policies perform very poorly for all traffic conditions, which means that DRL methods cannot learn under highly congested conditions. \end{enumerate*} We conjecture that DRL's inability to learn under congestion might be explained by a property of urban networks found here, whereby even a very bad policy produces an intersection throughput higher than downstream capacity. This means that the actual throughput tends to be independent of the policy.  Our findings imply that it is advisable for   current DRL methods in the literature to discard any congested data when training, and that doing this will improve their performance under all traffic conditions.    They also suggest that this inability to learn under congestion might be   alleviated by combining DRL for free-flow and supervised learning for congested conditions.

Danial Esaid, University of California, Riverside
Peng Hao, University of California, Riverside
Guoyuan Wu, University of California, Riverside
Fei Ye, University of California, Riverside
Kanok Boriboonsomsin, University of California, Riverside
Matthew Barth, University of California, Riverside

Taking advantage of communication and sensing technology, trajectory optimization at signalized intersection (i.e. Eco-Approach and Departure) based on information from vehicle-to-infrastructure (V2I) communications has been proven to be effective to improve vehicle energy efficiency while guaranteeing safety and mobility. However, existing approaches are either rule-based model or optimization model, which cannot achieve optimality and computational efficiency at the same time. In this paper, we propose and test a novel machine learning approach, Machine Learning Trajectory Planning Algorithm (MLTPA), to achive real time optimization by training a machine learning model to approximate the solution from a previously developed optimization-based method named graph-based trajectory planning algorithm (GBTPA). The proposed MLTPA reduces computation time from tens of seconds to a few milliseconds. The proposed approach also transforms GBTPA’s solution space from discretized to continuous. Simulation results illustrate that MLTPA can achieve average 12.80% improvement on energy savings over multiple simulation runs without compromising the vehicle average speed. The proposed method also has the potential to approximate other trajectory planning algorithms to achieve real time performance while ensuring optimality.

Lingyu Zheng, Tongji University
Bing Wu, Tongji University

Urban traffic control systems (UTCS) are deployed to a great number of urban cities despite they lack of feedback when adjusting the traffic signals. Development of reinforcement learning (RL) makes it possible to apply feedback to UTCS, and great efforts have been made on RL-based traffic control strategies. However, those studies are regardless of the traffic flow theory of the network and the road users’ perspectives on the performance of traffic. This study proposes a multi-agent reinforcement learning (MARL) based traffic control strategy, in which each intersection in a macroscopic fundamental diagram (MFD) region was controlled by one agent using level of services (LOS) and MFD-based parameters as reward. The proposed MARL strategy was evaluated by simulation in a 3x3 grid network compared with pretimed, actuated and MFD-based traffic control strategies. The evaluation results showed that in the low demand level, the performances of proposed MARL strategy and MFD-based traffic control strategy are similar, and both strategies outperform pretimed and actuated traffic control strategies in terms of average intersection queue length and average intersection waiting time. The proposed MARL strategy outperforms other three traffic control strategies in cases with high demand level.

Matthew Muresan, Miovision Technologies, Inc.
Guangyuan Pan, Linyi University
Liping Fu, University of Waterloo

This paper discusses a machine-learning traffic signal control method. A full-scale corridor is analysed and the transferability of using a model pre-trained on a single intersection is examined. Two controller designs are explored, a simple two-phase design and a full ring-and-barrier style controller. The full ring-and-barrier controller adapts many of the key features present in traditional controllers, such as protected-permissive left turns, so that they can be used in the reinforcement learning paradigm. This study is the first to propose a method that uses deep reinforcement learning (DRL) to implement a full ring-and-barrier style controller. The study also examines the feasibility of using transfer learning to pre-train a model on a single intersection and then fine-tune it for application in a complete environment. Training is done on a simple four lane intersection and the pre-trained model is then transferred for fine tuning to six controllers operating on a corridor modelled with field data obtained for University Ave in Waterloo, Ontario. The performance of the fully trained model is then compared to the existing signal plans in terms of the average delay and average queue length. Application of the ring-and-barrier design to this corridor was found to reduce delays by at least 5% and average queue lengths at intersections by 27%.

Weiran Yao, Carnegie Mellon University
Sean Qian, Carnegie Mellon University

The main question to address in this paper is recommend optimal signal timing plans under real-time incidents by incorporating domain knowledge developed for the traffic signal timing plans under real-time incidents, and learning from historical data of both traffic and implemented signals timing. The effectiveness of traffic incident management is often limited by the late response time and excessive workload of traffic operators. This paper proposes a novel decision-making framework that learns from both data and domain knowledge to recommend incident plans at least 30 minutes in advance, with the outputs from real-time traffic prediction. Specifically, considering the rare occurrences of engagement of contingency signal plans for incidents, we propose to decompose the end-to-end recommendation task into two hierarchical models -- real-time traffic prediction and plan association. We learn the connections between the two models through pairwise learning, which reinforces partial-order preferences observed from historical engagement records. We demonstrate the effectiveness of our approach by testing this framework on the traffic network in Cranberry Township in 2019. Results show that our recommendation system has a precision score of 96.75\% and recall of 87.5\% on the testing plan, and make recommendation of an average of 22.5 minutes lead time ahead of Waze alerts. The results suggest that our framework is capable of giving traffic operators significant time to access the condition and reacting appropriately.

Mosammat Tahnin Tariq, ITERIS, Inc.
Aidin Massahi, Eland Engineering, Inc
Rajib Saha, Virginia Department of Transportation
Mohammed Hadi, Florida International University

Events such as surges in demands or lane blockages can create queue spillbacks even during the off-peak periods resulting in delays and spillbacks upstream intersection(s). To address this issue, some transportation agencies have started implementing processes to change the signal timing in real-time based on traffic signal engineer/expert operator’s observations of incident and traffic conditions at the intersections upstream and downstream of the congested locations.  Decisions to change the signal timing are governed by many factors such as the queue length, conditions of the main and side streets, potential of spilling back to upstream intersections, the importance of upstream cross streets, the potential of the queue backing up to a freeway ramp. This paper investigates and assesses automating the process of updating the signal timing plans during non-recurrent conditions by capturing the history of the responses of the traffic signal engineers to non-recurrent conditions and utilizing this experience to train a machine learning model. A combination of Recursive Partitioning and Regression Decision Tree and fuzzy rule-based system is utilized in this study to deal with the vagueness and uncertainty of human decisions. Comparing the decisions made based on the resulting fuzzy rules by applying the methodology to previously recorded expert decisions for a project case-study indicates accurate recommendations for shifts in green time. The simulation results indicate that changing the green times based on the output of the fuzzy rules decrease the delays due to lane blockages or demand surge. Keywords: Non-Recurrent Congestion, Active Arterial Management, Incidents, Traffic Signal Control