Topics

#adas #advanced-driver-assistance-systems #artificial-intelligence #autonomous-driving #autopilot #autoware #computer-vision #deep-learning #deep-neural-networks #end-to-end-machine-learning #foundation-models #open-source #robotics #self-driving-car

Language breakdown

C++ 82.1%

Python 8.2%

CMake 5.6%

Shell 1.9%

Dockerfile 1.5%

Just 0.8%

▸ Files click to expand

README

Vision Pilot - Open Source L2 ADAS

VisionPilot

GitHub commit activity GitHub Repo stars

⭐ Star us on GitHub — your support motivates us a lot!

Free and fully open-source stack for L2 ADAS

**This codebase contains a productionizable and safety certifiable implementation of an open-source L2 ADAS system called Vision Pilot.**

Vision Pilot is designed to be integrated with automotive OEMs and Tier-1 suppliers in series production passenger cars, and the system can optionally be adopted for transportation and logisitcs use-cases in buses and trucks.

We offer the complete codebase as free and fully open-source, including AI model weights to help democratize access to this vital technology. Vision Pilot is available under the persmissive Apache 2.0 licence and can freely be used for both commercial and research purposes.

Vision Pilot is designed to support basic/entry L2 ADAS features for in-lane autonomous driving including the following features:

ACC - autonomous cruise control
FCW - forward collision warning
AEB - autonomous emergency braking
LKAS - lane keep assist
LDW - lane departure warning
ISA - intelligent speed assist
Autopilot - single-lane hands-free highway autopilot

Sensor specification:

Vision Pilot can be run with a single, front-facing, monocular camera with 52 - 55 degree horizontal field-of-view, and 1MP - 2MP resolution.

Hybrid End-to-End AI Architecture

We utilize a Hybrid End-to-End AI Architecture as the core of Vision Pilot in which data is processed in parallel by perception AI models for safety, and End-to-End AI models for performance.

Specifically, Vision Pilot leverages three open-source AI models developed by the Autoware Foundation, these are:

AutoSpeed - Closest in-path object detection
AutoSteer - Ego path future waypoint detection
AutoDrive - End to End distance/in-path object presence detection

and road curvature estimation

No Reliance on High Definition Maps

Vision Pilot does not require 3D high definition maps and operates in a 'mapless' mode to follow the road in real-time.

Presentation

For more information about Vision Pilot, please view our introductory presentation at this link

Getting Started

There are three ways you can build and run Vision Pilot - we provide example data which you can use for visualization and evaluation, select from the drop down options below:

Option 1 - Download and build from source

To get started with the project, download the source code from:

git clone https://github.com/autowarefoundation/autowarevisionpilot.git

Download ONNX Runtime from the GitHub releases page.

Build the project:

cd VisionPilot

mkdir build && cd build

cmake -DONNXRUNTIMEROOT=<ONNXRUNTIMEROOTPATH> ../

or with ROS2 support

cmake -DONNXRUNTIMEROOT=<ONNXRUNTIMEROOTPATH> -DENABLEROS2INTERFACE=ON ../

or with the optional Occupancy BEV window (heuristic 3D / bird's-eye panel beside the HUD; default is OFF):

cmake -DONNXRUNTIMEROOT=<ONNXRUNTIMEROOTPATH> -DENABLE_OCCUPANCY=ON ../

make

This will build the project and create VisionPilot executable inside the build directory. See VisionPilot/modules/visualization/README.md for Occupancy controls (orbit / pan / zoom).

Run Vision Pilot on test data and visualize outputs

OpenLane Dataset:

To test Vision Pilot using open loop scenario testing, first download the sample data from the Google Drive directory.

This directory contains video composed of image data from sequences in the appropriate dataset and vehicle speed data extracted from the dataset.

Update VisionPilot config files vision_pilot.conf inside config directory and set:

source.mode             = video

and visionpilottest.conf set:

source.inputvideo         = <INPUTVIDEOFILEPATH>
source.inputvehiclespeed = <INPUTVEHICLESPEEDFILEPATH>

to point to the appropriate video file path and vehicle speed file path.

Note: When VisionPilot built from source, update config files before the build.

Run VisionPilot from inside build directory

./VisionPilot

Build VisionPilot DEB package

To build Debian DEB package for VisionPilot run

cpack -G DEB

To build Debian DEB package for VisionPilot with CPU support build VisionPilot with

cmake -DONNXRUNTIMEROOT=<ONNXRUNTIMEROOTPATH> -DGPU=OFF ../

make

cpack -G DEB

Option 2 - Use the pre-built Debian pacakge

Install from the pre-built Debian package

This method is recommended in case new system installation and cuda dependencies are not installed yet.

Download VisionPilot prebuilt binary.

Install the .deb package

sudo apt install ./VisionPilot-1.0-x86_64.deb

Reboot the system, cuda dependencies for VisionPilot installed.

Run Vision Pilot on test data and visualize outputs

OpenLane Dataset:

To test Vision Pilot using open loop scenario testing, first download the sample data from the Google Drive directory.

This directory contains video composed of image data from sequences in the appropriate dataset and vehicle speed data extracted from the dataset.

Update VisionPilot config files vision_pilot.conf set:

source.mode             = video

and visionpilottest.conf set:

source.inputvideo         = <INPUTVIDEOFILEPATH>
source.inputvehiclespeed = <INPUTVEHICLESPEEDFILEPATH>

to point to the appropriate video file path and vehicle speed file path.

Note: When VisionPilot installed from prebuilt binary update the config files inside:

/usr/share/visionpilot/config

directory.

Run VisionPilot from the command line

VisionPilot

Option 3 - Use the Docker container

Run inside Docker container

To run Vision Pilot inside a Docker container, build the container using the Dockerfiles provided in the docker directory of the repository. Docker containers can be built with GPU/CPU support, NO_ROS2/ROS2 support, and optional Occupancy BEV support.

To build the container, go to the docker subdirectory and run the following commands:

Default is GPU support NO ROS2 support

./build.sh --gpu --ros2

to build with CPU support

./build.sh --cpu

to enable the optional Occupancy BEV window (pass alongside --gpu or --cpu):

./build.sh --gpu --occupancy

To run the container use the run.sh script. For example to run the container with CPU support

./run.sh --cpu

or with GPU support and ROS2 support

./run.sh --gpu --ros2

Note: Before building the container update the config files inside: config directory.

If building with CPU support update config/vision_pilot.conf

engine.provider     = cpu

and when building with ros2 support update

source.mode         = ros2

When using video input source update config/visionpilottest.conf

source.inputvideo         = <INPUTVIDEOFILEPATH>
source.inputvehiclespeed = <INPUTVEHICLESPEEDFILEPATH>

Input directory can be modified in the source by using run.sh script switch

./run.sh --gpu --data <HOSTDIR>:<CONTAINERDIR>

Note: need to be the same as during the container build.

Running Vision Pilot in a Simulator

We support the CARLA Simulator version 0.9.16 for closed-loop testing of Vision Pilot in a virtual environment

CARLA 0.9.16 with Unreal Engine 4: https://carla.readthedocs.io/en/latest/

How to Install CARLA Simulator

Download binaries and dependencies following official

documentation: https://carla-ue5.readthedocs.io/en/latest/start_quickstart/#

If your GPU has 6GB or less of VRAM then you should please follow the modifications specified

in https://gist.github.com/xmfcx/a5e32fdecfcd85c6cc9d472ce7a3a98d to run CARLA with lower VRAM requirements using docker (tested with RTX3060 laptop version)

How to run CARLA Simulator with Vision Pilot

Change the file path below to where CARLA is downloaded and run. Make sure to change the --volume path to your local specific directory path.

docker run -it --rm \
  --runtime=nvidia \                        # Use NVIDIA runtime for GPU access
  --net=host \                              # Use the host's network stack (helps with networking/performance)
  --env=DISPLAY=$DISPLAY \                  # Pass the host's DISPLAY environment variable (for GUI forwarding)
  --env=NVIDIAVISIBLEDEVICES=all \        # Expose all GPUs to the container
  --env=NVIDIADRIVERCAPABILITIES=all \    # Enable all driver capabilities (graphics, compute, etc.)
  --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \ # Mount X11 UNIX socket to enable GUI apps to display
  --volume="$HOME/Downloads/carla/CARLA_0.9.16/:/home/carla/host-carla" \ 
                                            # CHANGE AS NEEDED: Mount your local CARLA folder into the container
  --workdir="/home/carla/host-carla" \      # Set the working directory to the mounted CARLA folder
  carlasim/carla:0.9.16 \                   # Use the official CARLA Docker image, version 0.9.16
  bash CarlaUE4.sh -nosound                 # Run the CARLA startup script with -nosound flag

To run with ROS2 native interface, add --ros2 at the end

Update VisionPilot config

Update vision_pilot.conf set source.mode

source.mode = ros2

and also visionpilotros2.conf set ource.inputcameratopic

source.inputcameratopic = /carla/hero/main_cam/image

Build VisionPilot

Build VisionPilot with ROS2 support

cmake -DONNXRUNTIMEROOT=<ONNXRUNTIMEROOTPATH> -DENABLEROS2INTERFACE=ON ../

Build CARLA bridge

Go to Simulation/CARLA/ROS2 directory and build the CARLA ROS2 bridge

colcon build

Run CARLA Bridge

Source the installation directory

source ./install/setup.bash

ros2 launch  carlabridgebringup carla_bridge.launch.py host:=<HOST> port:=<PORT>

Note: If you are running CARLA on a different machine to the one which is running Vision Pilot you need to specify the host and port parameters. Set the IP address of the machine running CARLA as a host and set the port on which CARLA is publishing as a port.

Run VisionPilot

Go to VisionPilot build directory and run the VisionPilot

./VisionPilot

Running Vision Pilot with your own camera

In order to run Vision Pilot with your own camera, you will need to calibrate your camera and provide the calibraiton information to the Vision Pilot app - this is important for Vision Pilot to accurately measure the shape of the road and the distance to objects.

Calibrate your camera

Please follow the steps in the calibration guide and save the calibration yaml file in the Vision Pilot config folder and overwrite the data of the H.yaml file with your calibraiton file parameters - we suggest you retain a copy of the original H.yaml parameters so that you can run Vision Pilot on our example data.

Roadmap

Support for fusion between front-facing camera and automotive RADAR
Support for 8MP camera resolution with 120 degree horizontal field-of-view
Safety Verification and Automotive Standards Compliance (ISO26262, ISO8800)

Contributing

To learn more about how to participate in this project, please read the onboarding guide

🔗 More in this category