3.1.2. libdraw3d_joints.py

python libdraw3d_joints.py
roslaunch rosopenposergbd run_rviz.launch

3.2. Test Data from Disk

DATA="data/images_n40/"
rosrun rosopenposergbd detectanddraw_joints.py \
    --data_source disk \
    --basefolder $(rospack find rosopenpose_rgbd) \
    --folder_color $DATA/color \
    --folder_depth $DATA/depth \
    --camerainfofile $DATA/camparamsrealsense.json \
    --detect_hand true # true or false
roslaunch rosopenposergbd run_rviz.launch

3.3. Test Data from ROS Topic

Please download the color/depth images pulisher:

ROOT=$(rospack find rosopenposergbd) cd ${ROOT}/.. # cd to ~/catkin_ws/src/ folder. if [ ! -d "rospubandsubrgbdandcloud" ]; then     repo="https://github.com/felixchenfy/rospubandsubrgbdandcloud"     git clone ${repo}     cd rospubandsubrgbdandcloud     chmod a+x pubrgbdand_cloud.py fi

OK. Now publish data, and start detection and drawing:

roslaunch rosopenposergbd publishtestrgbd_data.launch
rosrun rosopenposergbd detectanddraw_joints.py \
    --data_source rostopic \
    --rostopiccolor camera/color/image_raw \
    --rostopicdepth camera/aligneddepthtocolor/imageraw \
    --rostopiccamerainfo camera/color/camerainfo \
    --detect_hand true # true or false

The publish rate is set as 1 image/second. Please read launch/publishtestrgbddata.launch, and change publish settings in config/publishtestrgbddataconfig.yaml.

3.4. Test Data from Realsense

In 3.3, the data is read from disk and published to ROS topics. Here we read it from Realsense D435.

roslaunch rosopenposergbd run_realsense.launch
rosrun rosopenposergbd detectanddraw_joints.py \
    --data_source rostopic \
    --isusingrealsense true \
    --detect_hand false # My laptop is not fast enough to detect hand.

4.Results

4.1. Speed

Running speed of the main program detectanddraw_joints.py is:

• Settings: RTX 2070; Image resize to 320x240.
• Results:

* body + hand: 3 fps. * Only body: 12 fps.

4.2. Accuracy

See the two gif at the top of this README.

• The 3D body joints are not too bad, but still has some "trembling" even if I'm standing still.
• The 3D hand joints are awful.

(1) One thing is that due to self-occlusion and motion blur, sometimes 2D joints are not detected or are not accurate. (2) Another thing is that finger's depth is very inaccurate. We know that the depth near an object's edge is likely to be noisy, and this is especially severe for small and thing objects like fingers.

Besides the above problem, the Realsense also doesn't work well as expected. The measured depth value becomes inaccurate when the object is 1 meter away. I didn't do the experiment, but I feel it's about 1cm error at 1 meter, and up to 10cm when the object is several meters away (Let's say 4 meters). Also, the depth is fluctuating noticeably for distant objects, which can even be noticed from depth image.

5. Others

5.1. Helper Tools

• Draw point cloud of color/depth image.

python scripts/drawpointcloud.py -h

You can directly run it and see the point cloud of data/image_i1.

• Record color/depth images to disk.

See this github: https://github.com/felixchenfy/rosrecordrgbd_images

• Publish color/depth images.

See this github: https://github.com/felixchenfy/rospubandsubrgbdandcloud

5.2. Noisy Realsense

Below are two screenshots of the point cloud of data/image_i1 taken from different angles. You can see that it's very noisy. When I recorded this data, I forgot to turn on Realsense's spacial and temporal filtering of the point cloud. I tried it later, but the result was still bad.

5.3. Bugs

• The ROS marker delete function sometimes doesn't work. I can't solve it right now, maybe you need to close rviz and open it again in order to clear the ROS markers. The corresponding delete function is defined in utils/librvizmarker.py and def deletemarker.