Medium-Sized Network: AV Behaviors & Collaborative MARL with TorchRL

In this tutorial, we use a medium-sized network for agent navigation. The chosen origin and destination points are specified in this file and can be adjusted by users. We also define AV behaviors through the agents’ reward formulation and implement their learning process using the TorchRL library.

---

Network Overview

In these notebooks, we utilize the Cologne network within our simulator, SUMO. As an initial baseline, we employ malicious AV behavior, where all AVs share the same reward, making it an ideal setup for algorithms designed to solve collaborative tasks.

Users can customize TrafficEnvironment by consulting its current defaults.json, overriding the desired values in a dictionary, and passing that dictionary to the class.

Included Tutorials:

• VDN Tutorial. Uses Value Decomposition Networks (VDN) for decentralized MARL training.

• QMIX Tutorial. Implements QMIX (QMIX), which leverages a mixing network with a monotonicity constraint.

• MAPPO Tutorial. Uses Multi-Agent Proximal Policy Optimization (MAPPO), the multi-agent adaptation of PPO (PPO).

---

Cologne Network Visualization

---

Defining Automated Vehicles Behavior Through Reward Formulations

As described in the paper, the reward function enforces a selected behavior on the agent. For an agent k with behavioral parameters φₖ ∈ ℝ⁴, the reward is defined as:

\[ r_k = \varphi_{k1} \cdot T_{\text{own}, k} + \varphi_{k2} \cdot T_{\text{group}, k} + \varphi_{k3} \cdot T_{\text{other}, k} + \varphi_{k4} \cdot T_{\text{all}, k} \]

where Tₖ is a vector of travel time statistics provided to agent k, containing:

• Own Travel Time (\(T_{\text{own}, k}\)): The amount of time the agent has spent in traffic.

• Group Travel Time (\(T_{\text{group}, k}\)): The average travel time of agents in the same group (e.g., AVs for an AV agent).

• Other Group Travel Time (\(T_{\text{other}, k}\)): The average travel time of agents in other groups (e.g., humans for an AV agent).

• System-wide Travel Time (\(T_{\text{all}, k}\)): The average travel time of all agents in the traffic network.

---

Behavioral Strategies & Objective Weightings

Behavior	φ₁	φ₂	φ₃	φ₄	Interpretation
Altruistic	0	0	0	-1	Minimize delay for everyone
Collaborative	-0.5	-0.5	0	0	Minimize delay for oneself and one’s own group
Collectivist	-0.1	-0.9	0	0	Primarily minimize delay for one’s own group
Competitive	-2	0	1	0	Minimize self-delay and maximize delay for others
Cooperative	0	-1	0	0	Minimize delay for one’s own group
Malicious	0	0	1	0	Maximize delay for the other group
Militant	0	-2	1	0	Minimize group delay and maximize delay for others
Selfish	-1	0	0	0	Minimize delay for oneself
Social	-0.5	0	0	-0.5	Minimize delay for oneself and everyone