Deep Reinforcement Learning for Autonomous Driving

Project Storyline

This project began with the ambition to develop an autonomous driving system using deep reinforcement learning in a simulated environment. The primary challenge was integrating complex reinforcement learning algorithms with the CARLA simulator, which required handling high-dimensional sensory inputs and ensuring stable training. Early difficulties included preprocessing mismatches between training and testing phases, leading to inconsistent agent behavior, and managing CARLA's actor lifecycle to prevent memory leaks and performance degradation.

To address these issues, we implemented a variational autoencoder (VAE) to compress semantic segmentation images into a latent space, reducing input dimensionality and accelerating learning. We adopted Proximal Policy Optimization (PPO) for its robustness in continuous action spaces, incorporating curriculum learning and action smoothing to stabilize training. Additionally, we developed rigorous cleanup mechanisms for CARLA actors and sensors, resolving warnings and improving simulation performance.

The solution involved a modular architecture with separate components for environment interaction, state encoding, and policy optimization. Through iterative testing and fine-tuning, we achieved reliable autonomous navigation in CARLA's Town02 and Town07 environments. Key learnings included the importance of simulation fidelity for real-world transfer, the value of preprocessing consistency, and the need for proactive resource management in simulation-based RL. This project demonstrates the potential of DRL for autonomous systems while highlighting practical engineering challenges in deploying such technologies.

Showcase Images

Figure 1: Project Methodology Overview

Figure 2: Original vs Reconstructed Images from VAE

Figure 3: VAE + PPO Training Pipeline

Figure 4: Overall System Architecture

File Structure

The project is organized into the following key directories and files:

• continuous_driver.py: Main script for training and testing the PPO agent.
• discrete_driver.py: Script for training the DDQN agent (experimental).
• encoder_init.py: Utility for initializing the VAE encoder.
• parameters.py: Configuration file containing hyperparameters.
• simulation/: Contains environment setup, sensors, and connection logic.
  • environment.py: Gym-style environment wrapper for CARLA.
  • sensors.py: Sensor configuration for the agent.
  • connection.py: Handles connection to CARLA server.
  • settings.py: Environment parameters.
• networks/: RL network implementations.
  • on_policy/ppo/: PPO algorithm and agent code.
  • off_policy/ddqn/: DDQN implementation.
• autoencoder/: VAE training and reconstruction scripts.
• preTrained_models/: Saved model checkpoints.
• checkpoints/: Intermediate training checkpoints.
• logs/: Training logs and metrics.
• runs/: TensorBoard logs.
• tools/: Utility scripts, including video recording tools.
• docs/: Documentation and guides.
• info/: Diagrams, figures, and additional documentation.
• requirements.txt: Python dependencies.

Best Saved Model

The best-performing model is located at preTrained_models/ppo/Town02/best_model_episode_6805_reward_44.88.pth. This model was trained on Town02 and achieved an average reward of 44.88 over 100 episodes, demonstrating stable autonomous driving with minimal collisions and lane deviations.

RL Architecture Diagram

The following diagram illustrates the reinforcement learning architecture used in this project:

graph TD
    A[CARLA Simulator] --> B[Semantic Segmentation Sensor]
    B --> C[VAE Encoder]
    C --> D[Latent State Representation]
    D --> E[PPO Policy Network]
    E --> F[Continuous Actions: Throttle, Brake, Steering]
    F --> G[Vehicle Control]
    G --> H[Environment Feedback: Reward, Next State]
    H --> I[Replay Buffer / Experience Collection]
    I --> E
    J[Training Loop] --> K[Policy Update via PPO]
    K --> E

Loading

This architecture integrates perception (VAE), decision-making (PPO), and action execution in a closed-loop system.

Simulation Gifs

Future demonstrations will include animated gifs showcasing the agent's performance:

• simulation/gif1.gif: Example of successful route completion in Town02.
• simulation/gif2.gif: Demonstration of collision avoidance and lane-keeping.

These gifs will be added to visualize real-time driving behavior.

Team Information

• Om Choksi (23AIML010) - GitHub: Omchoksi108
• Devang Dhandhukiya (23AIML014) - GitHub: DEVang0876

Compatible Python Version and Run Commands

• Python Version: 3.7 (64-bit)
• Operating Systems: Windows, Linux

Key Run Commands

Training a PPO Agent

python continuous_driver.py --exp-name ppo --town Town02 --total-timesteps 4000000

Testing a Trained Agent

python continuous_driver.py --exp-name ppo --train False --town Town02

Training VAE

cd autoencoder && python vae.py

Reconstructing Images with VAE

cd autoencoder && python reconstructor.py

Recording Test Video with Plots

python tools/record_test_video.py --model-path preTrained_models/ppo/Town02/best_model_episode_6805_reward_44.88.pth --town Town02 --output video.mp4 --tk-live --tk-width 800 --tk-height 600

Monitoring Training with TensorBoard

tensorboard --logdir runs/

Key Metrics

• Success Rate (SR100): 58.6% - Percentage of episodes completing a 100m track without crashing.
• Quality Success Rate (QSR100): 15-20% - Success with average lane deviation less than 1.2, indicating smooth driving.
• Best Average Reward: 44.88 (from best model on Town02).
• Training Stability: Achieved through curriculum learning, action std decay (0.4 to 0.1), and gradient clipping.
• Performance Improvements: 40-50% enhancement with optimizations like waypoint augmentation and safety layers.

---

Artificial Intelligence (AI) is advancing rapidly across multiple domains, and autonomous driving is one of its most exciting applications. In this project, we leverage state-of-the-art Deep Reinforcement Learning (DRL) techniques to train an agent capable of driving autonomously. To achieve this, we utilize CARLA, an open-source urban driving simulator, which provides a realistic environment for experimentation and training.

Since deploying untested algorithms directly in the real world is unsafe and raises ethical concerns, we use CARLA as a safe testbed for developing and validating our models.

We specifically adopt Proximal Policy Optimization (PPO), an on-policy DRL algorithm well-suited for continuous action spaces. PPO is trained to navigate through predefined routes in CARLA's towns while avoiding collisions. To accelerate learning, we also integrate a Variational Autoencoder (VAE) to compress high-dimensional sensory input into a lower-dimensional latent space, making the agent's learning process more efficient.

Prerequisites

• CARLA 0.9.8 + Additional Maps (Town02 & Town07).
• Supported OS: Windows or Linux.
• Python 3.7 (64-bit).

Copy additional maps into the CARLA directory to ensure a seamless experience.

How to Run the Project from Zero

1. Install CARLA Simulator: Download and install CARLA 0.9.8 from the official website. Ensure additional maps (Town02, Town07) are added to the CARLA installation directory.

2. Clone the Repository: git clone https://github.com/OMCHOKSI108/self-driving-car-agent-using-deep-reinforcement-learning.git

3. Set Up Python Environment: Create a virtual environment with python -m venv venv and activate it (venv\Scripts\activate on Windows).

4. Install Dependencies: Run pip install -r requirements.txt. Optionally, use Poetry: cd poetry && poetry update.

5. Train the VAE: Navigate to autoencoder/ and run python vae.py to train the Variational Autoencoder on semantic segmentation images. This generates the dataset and models in autoencoder/dataset/ and autoencoder/model/.

6. Train the PPO Agent: Execute python continuous_driver.py --exp-name ppo --town Town02 --total-timesteps 4000000 to train the agent. Monitor progress with TensorBoard: tensorboard --logdir runs/.

7. Test the Trained Agent: Use python continuous_driver.py --exp-name ppo --train False --town Town02 to evaluate the agent's performance.

8. Record Videos: Run python tools/record_test_video.py --model-path preTrained_models/ppo/Town02/best_model_episode_6805_reward_44.88.pth --town Town02 --output video.mp4 for visualization.

9. Fine-Tune if Needed: Adjust parameters in parameters.py and retrain for better results.

10. Monitor and Debug: Check logs in logs/, checkpoints in checkpoints/, and use TensorBoard for metrics.

This process takes several hours to days depending on hardware, starting from data collection to final testing.

Project Setup

1. Clone this repository.
2. Create a virtual environment:

   python -m venv venv

   Activate it:
   • Windows: venv\Scripts\activate
   • Linux/Mac: source venv/bin/activate
3. Install dependencies:

   pip install -r requirements.txt

4. Additionally, install with Poetry:

   cd poetry/ && poetry update

Once setup is complete:

• Launch CARLA server (0.9.8).
• Run the client:

  python continuous_driver.py --exp-name ppo --train False

Built With

• Python
• PyTorch
• CARLA Simulator
• Poetry
• TensorBoard

Methodology

Our methodology combines three essential components:

1. CARLA Simulation Environment.
2. Variational Autoencoder (VAE).
3. PPO Agent.

Running the Project

Run a Trained Agent

We provide pretrained PPO agents for Town02 and Town07 in preTrained_models/PPO/<town>.

Run:

python continuous_driver.py --exp-name ppo --train False

Switch to Town02:

python continuous_driver.py --exp-name ppo --train False --town Town02

Train a New Agent

python continuous_driver.py --exp-name ppo

Checkpoints will be stored in checkpoints/PPO/<town>/. Metrics/logs in logs/PPO/<town>/.

Monitor training with:

tensorboard --logdir runs/

Variational Autoencoder (VAE)

We collected ~12,000 160x80 semantic segmentation images to train the VAE. The VAE encodes these high-dimensional inputs into a compact latent space, which is then fed into the PPO network.

Check reconstruction:

cd autoencoder && python reconstructor.py

Project Architecture

File Overview

File/Folder	Description
continuous_driver.py	Training/testing PPO agent
discrete_driver.py	Experimental Dueling DQN agent
encoder_init.py	Uses trained encoder to convert states into latent space
parameters.py	Hyperparameters of the project
simulation/connection.py	CARLA environment connection class
simulation/environment.py	Main environment setup (Gym-style)
simulation/sensors.py	Agent's sensors setup
simulation/settings.py	Environment parameters
runs/	TensorBoard logs and visualizations
preTrained_models/ppo	Pretrained PPO agents
networks/on_policy/agent.py	PPO agent implementation
networks/on_policy/ppo.py	PPO network code
logs/	Metrics/logs from training
info/	Figures, diagrams, documentation
checkpoints/	Model checkpoints
carla/	CARLA .egg file for Python API
autoencoder/	VAE training and reconstruction code

Authors

• Om Choksi – GitHub
• Devang Dhandhukiya – GitHub