Autonomous racing entails precise high-speed control and trajectory generation under dynamic and uncertain conditions. Traditional planning and control methods, while effective, suffer from scalability and flexibility issues. This paper pioneers the use of deep reinforcement learning in the GTS simulator a platform used for professional driver scouting—to develop an agent capable of not only competing with but outperforming expert human drivers. The authors circumvent common RL challenges in sparse-reward environments by introducing a carefully constructed proxy reward system and applying the SAC algorithm for policy training.

Prior studies in the autonomous racing can be grouped into three categories:

• Trajectory planning and following: Model Predictive Control (MPC), Model Predictive Path Integral control (MPPI)
• Supervised learning: Imitation Learning (IL), Autonomous Land Vehicle in a Neural Network (ALVINN), Convolutional Neural Network (CNN) controller
• Reinforcement learning: Soft-Actor-Critic (SAC)

🎯 Step 1. Designing the Reward Function

Create a proxy reward that approximates lap time by evaluating how much progress the car makes along the track's centerline in small intervals. This gives a dense feedback signal, helping the agent learn more effectively.

But here’s the twist: fast driving often means flirting with the edge of control. To prevent the AI from simply bouncing off track walls (a cheap but effective strategy), the reward includes a penalty for wall contact, scaled by the car’s kinetic energy. The final reward at each timestep \(t\) is:

\[r_t = r^{\text{prog}}_t - \begin{cases} c_w |\mathbf{v}_t|^2 & \text{if wall contact} \\ 0 & \text{otherwise} \end{cases}\]

This encourages the agent to be fast and smooth just like a real racing pro.

🎯 Step 2: Representing the Car’s World

To make smart decisions, the AI needs to “see” its environment. But it doesn’t get pixels or radar scans—it uses structured game state data similar to what a human might infer visually:

• Velocity & Acceleration \((\vec{v}_t, \dot{\vec{v}}_t)\)
• Heading angle relative to the centerline
• Rangefinder distances in 180° front arc (think of LIDAR for edges)
• Previous steering action
• Wall contact flag
• Upcoming track curvature

These features are concatenated into a single observation vector \(s_t\), which serves as input to the neural network.

🎯 Step 3: Defining the Action Space

Instead of separate throttle and brake controls, the authors simplify things with a single combined throttle/brake command \((\omega_t)\), alongside a steering angle \((\delta_t)\). This reflects the observation that top human drivers rarely brake and accelerate simultaneously.

• \(\delta_t \in [-\frac{\pi}{6}, \frac{\pi}{6}]\) radians
• \(\omega_t \in [-1, 1]\), where -1 = full brake, 1 = full throttle

🎯 Step 4: Training the Neural Network (Policy)

The policy is trained using the Soft Actor-Critic (SAC) algorithm, a state-of-the-art model-free RL method known for stable and sample-efficient learning.

The architecture includes:

• Policy network: Outputs steering and throttle/brake
• Two Q-networks: Estimate value of actions
• One value network: Helps stabilize training

🎯 Step 5: Real-Time Simulation and Training

The magic happens on four PlayStation 4 consoles, each running 20 cars in parallel in the GTS environment. The AI collects real driving experiences (observations, actions, rewards), stores them in a replay buffer, and samples from this buffer to update its policy.

Training occurs in real-time, constrained by the PS4 hardware and simulator frame rate (10 Hz during training, 60 Hz during evaluation). This is no synthetic setup—it’s close to how an AI would need to operate in a real-world autonomous racing system.

The learned policy achieved lap times faster than the best human players in all three race settings:

Setting A (Audi TT Cup, Track A): 0.15 seconds faster

Setting B (Mazda Demio, Track A): 0.04 seconds faster

Setting C (Audi TT Cup, Track C): 0.62 seconds faster

The policy mimics professional driving behavior such as out-in-out cornering, anticipatory braking, and maximized exit speeds. Variance in lap time was significantly lower than that of expert humans, suggesting high consistency. Robustness tests under observation noise and inference delay show graceful degradation, maintaining top-tier performance up to 50 ms delay and 9% observation noise.

The paper convincingly demonstrates that end-to-end model-free RL can match and exceed human capabilities in high-speed racing. The approach’s strength lies in its simplicity and adaptability, eliminating the need for handcrafted cost functions or model-based control. Limitations include narrow generalization across unseen cars/tracks and lack of multi-agent interaction. Future work could explore meta-learning and policy distillation for broader applicability and competitive racing environments.

This work sets a new benchmark in autonomous racing by achieving super-human performance through a combination of realistic simulation, thoughtful reward design, and scalable RL training. It validates the potential of model-free deep reinforcement learning for real-world-like vehicle control and opens pathways for its application in broader autonomous driving contexts.