Sensor Calibration Tools

Calibration tools for sensors used in autonomous driving and robotics (camera, lidar, and radar).

Table of contents

• Sensor Calibration Tools

- Table of contents - Installation - Requirements - Installation alongside autoware - Standalone installation (for non-autoware users) - Standalone installation using Docker (for non-autoware users) - Available tools - Extrinsic calibration tools - Intrinsic calibration tools - Design - Calibrator node - Sensor calibration manager - Projects and calibrators - Calibrator interface - Launch files - Launching the sensor calibration manager - Preferences - Integration - Using your vehicle/robot - Create a new project - Integrate a new calibrator

Installation

Requirements

• Ubuntu 22.04 / 24.04
• ROS 2 Humble / Jazzy

Installation alongside autoware

After installing autoware (please see source-installation page), execute the following commands:

cd autoware
wget https://raw.githubusercontent.com/tier4/CalibrationTools/tier4/universe/calibrationtoolsautoware.repos
vcs import src < calibrationtoolsautoware.repos
rosdep install -y --from-paths src --ignore-src --rosdistro $ROS_DISTRO
colcon build --symlink-install --cmake-args -DCMAKEBUILDTYPE=Release

Standalone installation (for non-autoware users)

The sensor calibration tools are usually used as part of the autoware ecosystem. However, they can also be used for projects outside autoware, or even outside autonomous driving. Note: due to its use in autoware, even if it is possible to use the sensor calibration tools independently, due to some light dependencies, parts of autoware still need to be downloaded, even if they are not all compiled.

The following commands present an example of how to install the sensor calibration tools and their dependencies assuming you have a ROS2 workspace called workspace (if the workspace is new, the user must also create the src directory inside the workspace):

# Install vcs (if needed, follow the instructions from https://github.com/dirk-thomas/vcstool)
sudo apt-get install python3-vcstool
Download the calibration tools and its dependencies
cd workspace
wget https://raw.githubusercontent.com/tier4/CalibrationTools/tier4/universe/calibrationtoolsstandalone.repos
vcs import src < calibrationtoolsstandalone.repos
Install all the dependencies from rosdep
rosdep install -y --from-paths colcon list --packages-up-to sensorcalibrationtools -p --ignore-src
Build the sensor calibration tools. sensorcalibrationtools is a meta package that guarantees that only the related packages are compiled
colcon build --symlink-install --cmake-args -DCMAKEBUILDTYPE=Release --packages-up-to sensorcalibrationtools

Standalone installation using Docker (for non-autoware users)

With a similar motivation to that of the previous Section, in some cases, a native build is not possible nor convenient. To accommodate those situations, we also offer the sensor calibration tools as a docker image:

# Build
export ROS_DISTRO=humble  # for ROS 2 Humble
export ROS_DISTRO=jazzy  # for ROS 2 Jazzy
DOCKERBUILDKIT=1 docker build --ssh default -t ghcr.io/tier4/sensor-calibration-tools:$ROSDISTRO -f docker/Dockerfile --build-arg ROSDISTRO=$ROSDISTRO ..
Run - Modify if needed
docker run --gpus all --net=host -e ROSDOMAINID=$ROSDOMAINID -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --device=/dev/dri:/dev/dri -it ghcr.io/tier4/sensor-calibration-tools:${ROS_DISTRO} /bin/bash
If user encounters issues like "Authorization required", use one of the alternatives below.
Solution 1 (Not recommended):
xhost +local:docker
Solution 2:
touch /tmp/.docker.xauth
chmod a+r /tmp/.docker.xauth
xauth nlist $DISPLAY | sed -e 's/^..../ffff/' | xauth -f /tmp/.docker.xauth nmerge -
docker run --gpus all --net=host \   -e ROSDOMAINID=$ROSDOMAINID \   -e DISPLAY=$DISPLAY \   -e XAUTHORITY=/tmp/.docker.xauth \   -v /tmp/.X11-unix:/tmp/.X11-unix \   -v /tmp/.docker.xauth:/tmp/.docker.xauth \   --device=/dev/dri:/dev/dri \   -it ghcr.io/tier4/sensor-calibration-tools:${ROS_DISTRO} /bin/bash

Available tools

Extrinsic calibration tools

| Name | Sensors calibrated | Feature type | Calibration type | Documentation | Tutorial | | ----------------------------------- | ------------------ | --------------------------- | ------------------ | ----------------------------------------------------------- | ------------------------------------------------------- | | ground plane calibrator | base-lidar | ground | roll, pitch, z | N/A | N/A | | interactive camera-lidar calibrator | camera-lidar | manual correspondences | full pose | N/A | N/A | | lidar-lidar 2d calibrator | lidar-lidar | natural features | x, y, yaw | N/A | N/A | | mapping calibrator (lidar-lidar) | lidar-lidar | natural features | full pose | Link | Link | | mapping calibrator (base-lidar) | base-lidar | natural features and ground | roll, pitch, and z | N/A | N/A | | marker radar-lidar calibrator | radar-lidar | marker | x, y, yaw | Link | Link | | tag-based PnP calibrator | camera-lidar | marker | full pose | Link | Link | | tag-based SfM calibrator | camera-lidar-base | marker | full pose | Link | Link |

Intrinsic calibration tools

| Name | Sensors calibrated | Feature type | Calibration type | Documentation | Tutorial | | ---------------------------- | ------------------ | ------------------ | ------------------- | ------------- | -------- | | camera intrinsics calibrator | camera intrinsics | calibration boards | OpenCV camera model | N/A | N/A |

Design

The sensor calibration tools repository provides multiple alternatives for both intrinsic and extrinsic calibration. That being said, the rest of this document focuses only on extrinsic calibration since camera intrinsic calibration is a direct and easy-to-understand process.

The architecture of the extrinsic calibration process consists of two entities: the calibrator node itself and the sensor calibration manager (additional nodes may be used, but they do not participate directly in the calibration process). In what follows, we proceed to detail the roles of each one of these elements.

Calibrator node

The calibrator node is a regular node that implements the ExtrinsicCalibrator service:

---
tier4sensorcalibration_msgs/CalibrationResult[] results

where CalibrationResult contains a transformation between frames, a status flag, and optional scores and text messages for evaluation and debug purposes.

geometrymsgs/TransformStamped transformstamped
bool success
float32 score
std_msgs/String message

The design is intended to decouple the calibrator node and its internal logic as much as possible from the details of a particular calibration use-case and the tf structure used. The code of the calibrator itself is agnostic to everything other than its particular task, with the service request not even containing the frames to calibrate. This way, all use-case-specific concerns are specified during node and launcher configuration implemented in the sensorcalibrationmanager package, achieving a high level of separation of concerns and code reusability.

Sensor calibration manager

Although the calibrator process can be performed directly via launching the calibrator node (which involves non-trivial parameterization) and using the service interface using the CLI (ros2 service call ...), it is highly recommended to automate the process using the sensorcalibrationmanager package. The sensorcalibrationmanager implements a UI that allows the user to select a particular combination of project and calibrator, makes sure that the required tf and services are available, and processes/saves the calibration results.

Projects and calibrators

At TIER IV, we currently run several projects that use various types of sensors. However, we do not create calibrators nodes for each particular project, and instead reuse the same code, only modifying the parameters and helper nodes. To achieve this, in the sensorcalibrationmanager package, we introduce the concepts of projects and calibrators. In this context, a project consists of a list of calibrators (note that in this context calibrators are different from calibrator nodes), with the same calibrator (semantically) being able to belong to multiple projects.

An example of the files involved in this scheme is:

cd sensorcalibrationmanager/sensorcalibrationmanager/calibrators && find .
./projectA/
./projectA/calibratorA.py
./projectA/calibratorB.py
./projectA/init.py
./projectB/
./projectB/calibratorA.py
./projectB/calibratorB.py
./projectB/init.py
./projectC/
./projectC/calibratorC.py
./projectC/init.py
./init.py

In this example, the calibrators folder is placed inside the sensorcalibrationmanager package, each project is organized in its own folder (e.g., projectA, projectB, and projectC), and inside each project folder, one or more calibrators are represented via python files (e.g., calibratorA.py, calibratorB.py, and calibratorC.py).

• Note how some calibrators are present in more than one project. This essentially means that said calibrator can be used in multiple projects, albeit with its own set of configuration files (more on this later). For example, we would like to calibrate buses and robo-taxis with the tag-based PnP calibrator method.
• init.py files are used to register the projects and calibrators within the sensorcalibrationmanager package. How to write these files is explained in the Integration Section.

Calibrator interface

A calibrator interface is the representation of the calibration process inside the sensorcalibrationmanager package. It specifies its project, the calibrator name, the tfs that are required during the calibration process, and the expected frames that the calibrator node should return.

Following the previous example, the calibratorA.py file could be implemented as follows:

@CalibratorRegistry.register_calibrator(
    projectname="projectA", calibratorname="calibratorA"
)
class CalibratorA(CalibratorBase):
    requiredframes = ["calibrationparentframe", "calibrationchildframe", "auxiliarframe"]
def init(self, ros_interface: RosInterface, **kwargs):         super().init(ros_interface)
self.add_calibrator(             servicename="thenameofthecalibrationservice",             expectedcalibrationframes=[                 FramePair(parent="calibrationparentframe", child="calibrationchildframe"),             ],         )

In addition to specifying requiredframes and servicesname, the calibrator interfaces are also used to post-process the calibration results if needed to conform to robotics frame conventions and other project-specific requirements.

For example, camera-lidar calibration returns the tf from the opticallink to the lidar frame itself. However, in most scenarios, instead of the opticallink, the cameralink is preferred in configuration files (the cameralink has different axes), and for some lidars, integrators would prefer to use their baselink or footprint (not to be confused with the vehicle's baselink) since it allows them to work better with CAD files.

At TIER IV, most sensors are mounted in a structure called sensorkit and most tfs that correspond to sensor calibration either start or end at this frame (e.g., baselink to sensorkit or sensorkit to lidarbaselink).

In particular, for the case of camera-lidar, the tf that represents the camera-lidar calibration in most of our projects is sensorkitbaselink to cameraX/cameralink. To transform the tf that the calibrator returns (cameraX/cameraopticallink to lidar) to the one we need to save, the post-process step can be implemented as follows:

# Taken from sensorcalibrationmanager/sensorcalibrationmanager/calibrators/xx1/tagbasedpnp_calibrator.py
def postprocess(self, calibrationtransforms: Dict[str, Dict[str, np.array]]):
    opticallinktolidartransform = calibration_transforms[
        f"{self.cameraname}/cameraoptical_link"
    ]["velodyne_top"]
    sensorkittolidartransform = self.gettransformmatrix(
        "sensorkitbaselink", "velodynetop"
    )
    cameratoopticallinktransform = self.gettransformmatrix(
        f"{self.cameraname}/cameralink", f"{self.cameraname}/cameraoptical_link"
    )
    sensorkitcameralinktransform = np.linalg.inv(
        cameratoopticallinktransform
        @ opticallinktolidartransform
        @ np.linalg.inv(sensorkittolidartransform)
    )
result = {         "sensorkitbase_link": {             f"{self.cameraname}/cameralink": sensorkitcameralinktransform         }     }     return result

\*Note: in this example, sensorkittolidartransform is assumed as known and is fixed, since it corresponds to a previous lidar-lidar calibration result or it is a hardcoded value.

Launch files

The calibrator interface does not implement any of the ROS logic in terms of the involved nodes. This part of the process is implemented by regular launch files that are called by the sensorcalibrationmanager package.

Following the previous example, the launcher structure would be as follows:

launch/
launch/projectA/
launch/projectA/calibratorA.launch.xml
launch/projectA/calibratorB.launch.xml
launch/projectB/
launch/projectB/calibratorA.launch.xml
launch/projectB/calibratorB.launch.xml
launch/projectC/
launch/projectC/calibratorC.launch.xml

\*Note: a calibrator interface with the values projectname="projectA" and calibratorname="calibratorA" will launch launch/projectA/calibratorA.launch.xml

The launch file can have arguments with and without default arguments that will be automatically transformed into a configurable UI so the user can set them during start-up. One point of note is that the service specified in the calibrator interface must be offered by a node in the launch file.

\*Note: the values of the arguments defined in the launcher file are accessible to the calibration interface via kwargs.

Launching the sensor calibration manager

To use the calibration manager, execute the following command (after sourcing the ROS workspace):

ros2 run sensorcalibrationmanager sensorcalibrationmanager

The following window will be displayed:

Then, the user must select a combination of project and calibrator and press Continue, which will display the following menu:

Here, the user must configure the launcher arguments as required. Since we are using the combination of defaultproject and tagbasedpnpcalibrator, the launcher file being parameterized under the hood is sensorcalibrationmanager/launch/defaultproject/tagbasedpnpcalibrator.launch.xml. Once the user finishes setting the parameter, he must click the Launch button.

After this, the sensor calibrator manager will execute the previous launch file with the corresponding parameters and internally use sensorcalibrationmanager/sensorcalibrationmanager/calibrators/defaultproject/tagbasedpnpcalibrator.py as its calibrator interface. The following window will be displayed:

If the calibrator node launches successfully, its service becomes available, and the required tf are all present, the calibrate button should become enabled. Clicking this calls the calibration service, and starts the process.

For visualization purposes, the sensor calibration manager displays the required tf specified in the calibrator interface as shown in the following image:

Once the calibration finishes, the result from the ExtrinsicCalibrator is displayed in the Calibration tree widget as shown in the following image:

If the calibrator interface had post-processing steps, the Final TF tree widget would show the processed results. In this case, since there is not a post-process step, both widgets are the same.

Finally, to save the results, press the save calibration button.

Preferences

The following calibrators save some of the last user configuration and load on relaunch, which is helpful in case of repeating the same calibration over mostly identical configurations.

• Sensor calibration manager: ~/.config/tier4/sensorcalibrationmanager.conf
• Intrinsic camera calibrator : ~/.config/tier4/intrinsiccameracalibrator.conf

Integration

Using your vehicle/robot

Although we provide several projects and examples, in most cases the user would need to modify several parts of this repository to create their calibration projects. To ease this process, we also created a default_project that exposes most of each calibrator's parameters so that users can use this repository without creating any new files. In turn, however, they must properly configure most of the calibrator's options in the Launcher configuration widget, which is mostly done automatically in our internal projects.

Create a new project

If the default project is not enough or does not meet the needs of the user (for example, if they are going to start projects that require running the tools frequently), it is relatively easy to create new projects and calibrator interfaces. In what follows, we will create a new project titled mynewproject with a calibrator called mynewcalibrator.

First, to make mynewproject known within the sensorcalibrationmanager package, add the following line to sensorcalibrationmanager/sensorcalibrationmanager/calibrators/init.py:

from .mynewproject import *  # noqa: F401, F403

This will make the UI attempt to load all the calibration interfaces of mynewproject. Then, create a file called sensorcalibrationmanager/sensorcalibrationmanager/calibrators/mynewproject/init.py.

from .mynewcalibrator import MyNewCalibrator
all = [     "MyNewCalibrator", ]

This will make MyNewCalibrator to be imported when importing mynewproject. After this, create a file called sensorcalibrationmanager/sensorcalibrationmanager/calibrators/mynewproject/mynewcalibrator.py with the following contents:

from sensorcalibrationmanager.calibrator_base import CalibratorBase
from sensorcalibrationmanager.calibrator_registry import CalibratorRegistry
from sensorcalibrationmanager.ros_interface import RosInterface
from sensorcalibrationmanager.types import FramePair
@CalibratorRegistry.register_calibrator(     projectname="mynewproject", calibratorname="mynewcalibrator" ) class MyNewCalibrator(CalibratorBase):     required_frames = []
def init(self, ros_interface: RosInterface, **kwargs):         super().init(ros_interface)
self.sourceframe: str = kwargs["sourceframe"]         self.targetframe: str = kwargs["targetframe"]
self.requiredframes.extend([self.baseframe, self.sourceframe, self.targetframe])
self.add_calibrator(             servicename="calibrateservice_name",             expectedcalibrationframes=[                 FramePair(parent=self.targetframe, child=self.sourceframe),             ],         )

This calibrator will expect a calibrator between sourceframe and targetframe, which are parameters provided by the user through the launcher file.

Finally, the user must create the correspondent launch file sensorcalibrationmanager/launch/mynewproject/mynewcalibrator.launch.xml. The launcher itself must contain the following arguments to match the calibrator interface:

<arg name="source_frame"/>
<arg name="target_frame"/>

Default values are optional, and somewhere within the launch file, the user needs to add a node that provides the calibrateservicename to match the calibrator interface. Complete examples of the contents of the launcher and calibration interfaces can be found in our current projects and calibrators.

Integrate a new calibrator

In the previous Section, we created a new calibrator interface and added it to the sensor calibration manager. However, that assumes that the user will use one of the calibrator packages offered by the sensor calibration tools.

In the case the user wants to integrate his own algorithms, he must comply with the following the instructions, which assume the reader knows and is used to creating ROS2 packages.

• Create a ROS2 package called mynewcalibrator_package. The node itself needs to be part of a multi-thread executor with at least two threads. This is due to the calibration service call only returning once the calibration process finishes.
• Add a dependency to the tier4sensorcalibration_msgs package in package.xml to use the calibration services.
• In the node's header file add a calibration service.
• In most cases, creating a group exclusive to the services is also required.

In the header file:

++
...
rclcpp::Service<tier4sensorcalibrationmsgs::srv::ExtrinsicCalibrator>::SharedPtr serviceserver_;
rclcpp::CallbackGroup::SharedPtr srvcallbackgroup_
...

In the source file:

++
...
// The service server runs in a dedicated thread since it is a blocking call
srvcallbackgroup = createcallback_group(rclcpp::CallbackGroupType::MutuallyExclusive);
serviceserver = this->createservice<tier4sensorcalibrationmsgs::srv::ExtrinsicCalibrator>(   "extrinsiccalibrationservice_name",   std::bind(     &MyNewCalibratorPackage::requestReceivedCallback, this, std::placeholders::_1,     std::placeholders::_2),   rmwqosprofileservicesdefault, srvcallbackgroup_);   ...

In addition to this, the user must implement requestReceivedCallback to comply with the interface.