diff --git a/.github/2d3ds-results-large-1.jpg b/.github/2d3ds-results-large-1.jpg new file mode 100644 index 0000000..c903cf6 Binary files /dev/null and b/.github/2d3ds-results-large-1.jpg differ diff --git a/.github/inference_time.png b/.github/inference_time.png new file mode 100644 index 0000000..bd56ca2 Binary files /dev/null and b/.github/inference_time.png differ diff --git a/.github/shapenet.png b/.github/shapenet.png new file mode 100644 index 0000000..fd46e8d Binary files /dev/null and b/.github/shapenet.png differ diff --git a/.github/youtube.jpg b/.github/youtube.jpg new file mode 100644 index 0000000..db5f9df Binary files /dev/null and b/.github/youtube.jpg differ diff --git a/README.md b/README.md index 06e5dd5..09fe613 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Flex-Convolution (Million-Scale Pointcloud Learning Beyond Grid-Worlds) +# Flex-Convolution (Million-Scale Point-Cloud Learning Beyond Grid-Worlds) Fabian Groh, Patrick Wieschollek, Hendrik P.A. Lensch ![Build Status TensorFlow v1.10 ](https://ci.patwie.com/api/badges/cgtuebingen/Flex-Convolution/status.svg) @@ -14,7 +14,10 @@ The following figure shows the *raw* network semantic segmentation prediction on

-This repository contains contains the source code of our FlexConv Layer from our 2018 ACCV paper "Flex-Convolution (Million-Scale Pointcloud Learning Beyond Grid-Worlds)". +This repository contains contains the source code of our FlexConv Layer from our 2018 ACCV paper "Flex-Convolution (Million-Scale Point-Cloud Learning Beyond Grid-Worlds)". + +

+ Example - Usage ------------------- @@ -25,7 +28,7 @@ We provide GPU-tailored cuda implementation of our novel FlexConv in TensorFlow. user@host $ cd user_ops user@host $ cmake . -DPYTHON_EXECUTABLE=python2 && make -j user@host $ cd .. -user@host $ python example +user@host $ python example.py ``` @@ -34,13 +37,53 @@ Experiments Deep learning on point-clouds is a complex matter and our codebase reflect that complexity. We are currently working on refactoring our research implementation to ease the usage. Therefore, -`layers.py` contains a Keras/tf.layers compatible implementation. +`layers.py` contains a Keras/tf.layers compatible implementation. We will add the models later. + +### Benchmark + +We benchmarked the inference time of entire network on the 2D-3D-S dataset and with recent tests on V100, we were able to process ~18 Million Points. + +

+ + +### ShapeNet Segmentation + +ShapeNet part segmentation results per category and mIoU (%) for different methods and inference speed (on a NVIDIA GeForce GTX 1080 Ti). + +|| Airplane | Bag | Cap | Car | Chair | Earphones | Guitar | Knife | Lamp | Laptop | Motorbike | Mug | Pistol | Rocket | Skateboard | Table | mIoU | shapes/sec | +| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | ---- | ---- | --- | +|Kd-Network [4] | 80.1 | 74.6 | 74.3 | 70.3 | 88.6 | 73.5 | 90.2 | **87.2** | 81.0 | 94.9 | 57.4 | 86.7 | 78.1 | 51.8 | 69.9 | 80.3 | 77.4 | n.a.| +|PointNet [1] | 83.4 | 78.7 | 82.5 | 74.9 | 89.6 | 73.0 | 91.5 | 85.9 | 80.8 | 95.3 | 65.2 | 93.0 | 81.2 | 57.9 | 72.8 | 80.6 | 80.4 | n.a. | +|PointNet++ [2] | 82.4 | 79.0 | 87.7 | 77.3 | **90.8** | 71.8 | 91.0 | 85.9 | 83.7 | 95.3 | 71.6 | 94.1 | 81.3 | 58.7 | 76.4 | 82.6 | 81.9 | 2.7 | +|SPLATNet3D [3] | 81.9 | 83.9 | 88.6 | **79.5** | 90.1 | 73.5 | 91.3 | 84.7 | **84.5** | 96.3 | 69.7 | 95.0 | 81.7 | 59.2 | 70.4 | 81.3 | 82.0 | 9.4 | +|SGPN [5] | 80.4 | 78.6 | 78.8 | 71.5 | 88.6 | **78.0** | 90.9 | 83.0 | 78.8 | 95.8 | **77.8** | 93.8 | **87.4** | 60.1 | **92.3** | **89.4** | 82.8 | n.a. | +Ours | **83.6** | **91.2** | **96.7** | **79.5** | 84.7 | 71.7 | **92.0** | 86.5 | 83.2 | **96.6** | 71.7 | **95.7** | 86.1 | **74.8** | 81.4 | 84.5 | **85.0** | **489.3** | + +![example segmentation](./.github/shapenet.png) + +### 2D-3D-S dataset + +Class specific average precision (AP) on the 2D-3D-S dataset. + +| | Table | Chair | Sofa | Bookcase | Board | Ceiling | Floor | Wall | Beam | Col. | Wind. | Door | mAP | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +Armeni [6] | 46.02 | 16.15 | 6.78 | 54.71 | 3.91 | 71.61 | 88.70 | 72.86 | 66.67 | **91.77** | 25.92 | 54.11 | 49.93| +Armeni [6] | 39.87 | 11.43 | 4.91 | **57.76** | 3.73 | 50.74 | 80.48 | 65.59 | 68.53 | 85.08 | 21.17 | 45.39 | 44.19| +PointNet [1] | 46.67 | 33.80 | 4.76 | n.a. | 11.72 | n.a. | n.a. | n.a. | n.a. | n.a. | n.a. | n.a. | n.a. | +SGPN [5] | 46.90 | 40.77 | 6.38 | 47.61 | 11.05 | 79.44 | 66.29 | **88.77** | **77.98** | 60.71 | **66.62** | **56.75** | 54.35 | +Ours | 66.03 | 51.75 | 15.59 | 39.03 | 43.50 | 87.20 | 96.00 | 65.53 | 54.76 | 52.74 | 55.34 | 35.81 | 55.27| +Ours** | **67.02** | **52.75** | **16.61** | 39.26 | **47.68** | **87.33** | **96.10** | 65.52 | 56.83 | 55.10 | 57.66 | 36.76 | **56.55**| + + +![example segmentation](./.github/2d3ds-results-large-1.jpg) More Resources ------------------- - [Arxiv Pre-Print](https://arxiv.org/abs/1803.07289) - [Video](https://www.youtube.com/watch?v=5ftWmuQXU_s) +- [Project-Page](https://uni-tuebingen.de/fakultaeten/mathematisch-naturwissenschaftliche-fakultaet/fachbereiche/informatik/lehrstuehle/computergrafik/lehrstuhl/publications/publications-since-2012/flex-convolution-million-scale-point-cloud-learning-beyond-grid-worlds/) + Citation @@ -52,9 +95,19 @@ If you use the code in this repository, please cite our paper: Patrick Wieschollek and Hendrik P. A. Lensch }, - title = {Flex-Convolution (Million-Scale Pointcloud Learning Beyond Grid-Worlds)}, + title = {Flex-Convolution (Million-Scale Point-Cloud Learning Beyond Grid-Worlds)}, booktitle = {Asian Conference on Computer Vision (ACCV)}, month = {Dezember}, year = {2018} } ``` + +References +------------------- + +[1] C. Qi, H. Su, K. Mo, L. Guibas, "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017.
+[2] C. Qi and L. Yi, H. Su, L. Guibas, "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space", Advances in Neural Information Processing Systems (NIPS) 2017.
+[3] H. Su, V. Jampani, D.Sun, S. Maji, E. Kalogerakis, M.-H. Yang, J. Kautz, "SPLATNet: Sparse Lattice Networks for Point Cloud Processing", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018.
+[4] R. Klokov, V. Lempitsky, "Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models", Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2017.
+[5] W. Wang, R. Yu, Q. Huang, U. Neumann, "Sgpn: Similarity group proposal network for 3d point cloud instance segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018.
+[6] I. Armeni, A. Sax, A.-R. Zamir, S. Savarese, "Joint 2D-3D-Semantic Data for Indoor Scene Understanding", ArXiv e-prints 2017.