Topics

#autonomous-driving #classification #convolutional-neural-networks #deep-neural-networks #machine-learning #point-cloud #pointcloud #robotics #scannet #segmentation #shapenet

Language breakdown

No language data available.

▸ Files click to expand

README

PointCNN: Convolution On X-Transformed Points

Created by Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.

🚀 Exciting Update: Introducing PointCNN++

>

We are thrilled to share our most recent work, PointCNN++! Building upon the foundations laid here, PointCNN++ features our latest architectural advancements, modernized codebase, and improved performance for point cloud processing.

>

We highly encourage researchers and developers to check out the new repository for state-of-the-art results and ongoing development.

Introduction

PointCNN is a simple and general framework for feature learning from point cloud, which refreshed five benchmark records in point cloud processing (as of Jan. 23, 2018), including:

classification accuracy on ModelNet40 (91.7%, with 1024 input points only)
classification accuracy on ScanNet (77.9%)
segmentation part averaged IoU on ShapeNet Parts (86.13%)
segmentation mean IoU on S3DIS (65.39%)
per voxel labelling accuracy on ScanNet (85.1%)

See our preprint on arXiv (accepted to NeurIPS 2018) for more details.

Pretrained models can be downloaded from here.

Performance on Recent Benchmarks

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

ABC: A Big CAD Model Dataset For Geometric Deep Learning

Practical Applications

3D cities: Deep Learning in three-dimensional space (from Esri)

PointCNN: replacing 50,000 man hours with AI (from Esri)

Point Cloud Segmentation using PointCNN in ArcGIS API for Python (from Esri)

More Implementations

We highly welcome issues, rather than emails, for PointCNN related questions.

License

Our code is released under MIT License (see LICENSE file for details).

Code Organization

The core X-Conv and PointCNN architecture are defined in pointcnn.py.

The network/training/data augmentation hyper parameters for classification tasks are defined in pointcnncls, for segmentation tasks are defined in pointcnnseg.

Explanation of X-Conv and X-DeConv Parameters

Take the xconvparams and xdconvparams from shapenetx82048fps.py for example:

xconvparamname = ('K', 'D', 'P', 'C', 'links')
xconvparams = [dict(zip(xconvparamname, xconvparam)) for xconv_param in
                [(8, 1, -1, 32 * x, []),
                 (12, 2, 768, 32 * x, []),
                 (16, 2, 384, 64 * x, []),
                 (16, 6, 128, 128 * x, [])]]
xdconvparamname = ('K', 'D', 'ptslayeridx', 'qrslayeridx') xdconvparams = [dict(zip(xdconvparamname, xdconvparam)) for xdconv_param in                  [(16, 6, 3, 2),                   (12, 6, 2, 1),                   (8, 6, 1, 0),                   (8, 4, 0, 0)]]

Each element in xconv_params is a tuple of (K, D, P, C, links), where K is the neighborhood size, D is the dilation rate, P is the representative point number in the output (-1 means all input points are output representative points), and C is the output channel number. The links are used for adding DenseNet style links, e.g., [-1, -2] will tell the current layer to receive inputs from the previous two layers. Each element specifies the parameters of one X-Conv layer, and they are stacked to create a deep network.

Each element in xdconvparams is a tuple of (K, D, ptslayeridx, qrslayeridx), where K and D have the same meaning as that in xconvparams, ptslayeridx specifies the output of which X-Conv layer (from the xconvparams) will be the input of this X-DeConv layer, and qrslayeridx specifies the output of which X-Conv layer (from the xconvparams) will be forwarded and fused with the output of this X-DeConv layer. The P and C parameters of this X-DeConv layer is also determined by qrslayeridx. Similarly, each element specifies the parameters of one X-DeConv layer, and they are stacked to create a deep network.

PointCNN Usage

PointCNN is implemented and tested with Tensorflow 1.6 in python3 scripts. Tensorflow before 1.5 version is not recommended, because of API. It has dependencies on some python packages such as transforms3d, h5py, plyfile, and maybe more if it complains. Install these packages before the use of PointCNN.

If you can only use Tensorflow 1.5 because of OS factor(UBUNTU 14.04),please modify "isnan()" to "std::nan()" in "/usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/numeric_types.h" line 49

Here we list the commands for training/evaluating PointCNN on classification and segmentation tasks on multiple datasets.

### Classification

* #### ModelNet40

cd data_conversions
  python3 ./download_datasets.py -d modelnet
  cd ../pointcnn_cls
  ./trainvalmodelnet.sh -g 0 -x modelnetx3l4

* #### ScanNet Please refer to for downloading ScanNet task data and scannet_labelmap, and refer to https://github.com/ScanNet/ScanNet/tree/master/Tasks/Benchmark for downloading ScanNet benchmark files:

scannetdatasetdownload

|_ data

| scannetlabelmap

|_ benchmark

cd ../data/scannet/scannetdatasetdownload/ mv ./scannet_labelmap/scannet-labels.combined.tsv ../benchmark/

#./pointcnn_root cd ../../../pointcnn/data_conversions python extractscannetobjs.py -f ../../data/scannet/scannetdatasetdownload/data/ -b ../../data/scannet/scannetdatasetdownload/benchmark/ -o ../../data/scannet/cls/ python preparescannetcls_data.py -f ../../data/scannet/cls/ cd ../pointcnn_cls/ ./trainvalscannet.sh -g 0 -x scannetx3l4

* #### tu_berlin

cd data_conversions   python3 ./downloaddatasets.py -d tuberlin   python3 ./preparetuberlindata.py -f ../../data/tuberlin/ -a --create-train-test   cd ../pointcnn_cls   ./trainvaltuberlin.sh -g 0 -x tuberlinx3l4

* #### quick_draw Note that the training/evaluation of quick_draw requires LARGE RAM, as we load all stokes into RAM and converting them into point cloud on-the-fly.

cd data_conversions   python3 ./downloaddatasets.py -d quickdraw   cd ../pointcnn_cls   ./trainvalquickdraw.sh -g 0 -x quickdrawfullx2_l6

* #### MNIST

cd data_conversions   python3 ./download_datasets.py -d mnist   python3 ./preparemnistdata.py -f ../../data/mnist   cd ../pointcnn_cls   ./trainvalmnist.sh -g 0 -x mnistx2l4

* #### CIFAR-10

cd data_conversions   python3 ./download_datasets.py -d cifar10   python3 ./preparecifar10data.py   cd ../pointcnn_cls   ./trainvalcifar10.sh -g 0 -x cifar10x3l4

### Segmentation

We use farthest point sampling (the implementation from PointNet++) in segmentation tasks. Compile FPS before the training/evaluation:

cd sampling
  bash tfsamplingcompile.sh

* #### ShapeNet

cd data_conversions   python3 ./downloaddatasets.py -d shapenetpartseg   python3 ./preparepartsegdata.py -f ../../data/shapenet_partseg   cd ../pointcnn_seg   ./trainvalshapenet.sh -g 0 -x shapenetx82048_fps   ./testshapenet.sh -g 0 -x shapenetx82048fps -l ../../models/seg/pointcnnsegshapenetx82048fpsxxxx/ckpts/iter-xxxxx -r 10   cd ../evaluation   python3 evalshapenetseg.py -g ../../data/shapenetpartseg/testlabel -p ../../data/shapenetpartseg/testdatapred10 -a

* #### S3DIS Please refer to data_conversions for downloading S3DIS, then:

cd data_conversions   python3 prepares3dislabel.py   python3 prepares3disdata.py   python3 prepares3disfilelists.py   mv S3DISfiles/* ../../data/S3DIS/outpart_rgb/   ./trainvals3dis.sh -g 0 -x s3disx82048_fps -a 1   ./tests3dis.sh -g 0 -x s3disx82048fps -a 1 -l ../../models/seg/s3disx82048fpsxxxx/ckpts/iter-xxxxx -r 4   cd ../evaluation   python3 s3dismerge.py -d <path to *pred.h5>   python3 eval_s3dis.py

We use a hidden marker file to note when prepare is finished to avoid re-processing. This cache can be invalidated by deleting the markers. Please notice that these command just for Area 1 (specified by -a 1 option) validation. Results on other Areas can be computed by iterating -a option.

* #### ScanNet Please refer to data_conversions for downloading ScanNet, then:

cd data_conversions   python3 preparescannetseg_data.py   python3 preparescannetseg_filelists.py   cd ../pointcnn_seg   ./trainvalscannet.sh -g 0 -x scannetx82048k8fps   ./testscannet.sh -g 0 -x scannetx82048k8fps -l ../../models/seg/pointcnnsegscannetx82048k8fpsxxxx/ckpts/iter-xxxxx -r 4   cd ../evaluation   python3 evalscannet.py -d <path to *pred.h5> -p <path to scannet_test.pickle>

* #### Semantic3D

Please check the free disk space before start, about 900 GB will be required.

cd data_conversions
  bash download_semantic3d.sh
  bash un7z_semantic3d.sh
  python3 preparesemantic3ddata.py
  mkdir ../../data/semantic3d/filelists
  python3 preparesemantic3dfilelists.py
  cd ../pointcnn_seg
  ./trainvalsemantic3d.sh -g 0 -x semantic3dx42048_fps
  ./testsemantic3d.sh -g 0 -x semantic3dx42048fps -l <path to ckpt>
  cd ../evaluation
  python3 semantic3dmerge.py -d <path to *pred.h5> -v <reduced or full>

### Tensorboard

If you want to monitor your train step, we recommend you use the following command

cd <your path>/PointCNN
  tensorboard --logdir=../models/<seg/cls> <--port=6006>

🔗 More in this category