Open3D: A Modern Library for 3D Data Processing

1 open3d : A Modern Library for 3D data ProcessingQian-Yi ZhouJaesik ParkVladlen KoltunIntel LabsAbstractOpen3D is an open-source Library that supports rapiddevelopment of software that deals with 3D data . TheOpen3D frontend exposes a set of carefully selected datastructures and algorithms in both C++ and Python. Thebackend is highly optimized and is set up for was developed from a clean slate with a small andcarefully considered set of dependencies. It can be set upon different platforms and compiled from source with mini-mal effort. The code is clean, consistently styled, and main-tained via a clear code review mechanism. open3d hasbeen used in a number of published research projects andis actively deployed in the cloud. We welcome contributionsfrom the open-source IntroductionThe world is three-dimensional. Systems that operate inthe physical world or deal with its simulation must oftenprocess three-dimensional data .

2 Such data takes the formof point clouds, meshes, and other representations. data inthis form is produced by sensors such as LiDAR and depthcameras, and by software systems that support 3D recon-struction and the central role of 3D data in fields such asrobotics and computer graphics, writing software that pro-cesses such data is quite laborious in comparison to otherdata types. For example, an image can be efficiently loadedand visualized with a few lines of OpenCV code [3]. A sim-ilarly easy to use software framework for 3D data has notemerged. A notable prior effort is the Point Cloud Library (PCL) [18]. Unfortunately, after an initial influx of open-source contributions, PCL became encumbered by bloat andis now largely dormant. Other open-source efforts includeMeshLab [6], which provides a graphical user interface forprocessing meshes; libigl [11], which supports discrete dif-ferential geometry and related research; and a variety of so-lutions for image-based reconstruction [13].

3 Nevertheless,there is currently no open-source Library that is fast, easy touse, supports common 3D data Processing workflows, andis developed in accordance with Modern software engineer-ing was created to address this need. It is an open-source Library that supports rapid development of softwarethat deals with 3D data . The open3d frontend exposes a setof carefully selected data structures and algorithms in bothC++ and Python. The open3d backend is implemented inC++11, is highly optimized, and is set up for OpenMP par-allelization. open3d was developed from a clean slate witha small and carefully considered set of dependencies. It canbe set up on different platforms and compiled from sourcewith minimal effort. The code is clean, consistently styled,and maintained via a clear code review has been in development since 2015 and hasbeen used in a number of published research projects [22,13, 15, 12].

4 It has been deployed and is currently runningin the Tanks and Temples evaluation server [13]. open3d is released open-source under the permissiveMIT license and is available welcome contributions from the DesignTwo primary design principles of open3d are usefulnessand ease of use [8]. Our key decisions can be traced tothese two principles. Usefulness motivated support for pop-ular representations, algorithms, and platforms. Ease of useserved as a countervailing force that guarded against heavy-weight dependencies and feature provides data structures for three kinds of repre-sentations: point clouds, meshes, and RGB-D images. Foreach representation, we have implemented a complete setof basic Processing algorithms such as I/O, sampling, visu-alization, and data conversion. In addition, we have imple-mented a collection of widely used algorithms, such as nor-mal estimation [16], ICP registration [2, 4], and volumet-ric integration [7].

5 We have verified that the functionalityof open3d is sufficient by using it to implement completeworkflows such as large-scale scene reconstruction [5, 15].We carefully chose a small set of lightweight depen-dencies, including Eigen for linear algebra [9], GLFWfor OpenGL window support, and FLANN for fast near-est neighbor search [14]. For easy compilation, powerful1but heavyweight libraries such as Boost and Ceres are ex-cluded. Instead we use either lightweight alternatives ( ,pybind11 instead of ) or in-house implemen-tations ( , for Gauss-Newton and Levenberg-Marquardtgraph optimization). The source code of all dependencies isdistributed as part of open3d . The dependencies can thusbe compiled from source if not automatically detected bythe configuration script. This is particularly useful for com-pilation on operating systems that lack package manage-ment software, such as Microsoft development of open3d started from a clean slateand the Library is kept as simple as possible.

6 Only algo-rithms that solve a problem of broad interest are a problem has multiple solutions, we choose one thatthe community considers standard. A new algorithm isadded only if its implementation demonstrates significantlystronger results on a well-known is written in standard C++11 and uses CMaketo support common C++ toolchains including GCC and later on Linux XCode and later on OS X Visual Studio 2015 and later on WindowsA key feature of open3d is ubiquitous Python follow the practice of successful computer vision anddeep learning libraries [1, 3]: the backend is implemented inC++ and is exposed through frontend interfaces in use Python as a glue language to assemble com-ponents implemented in the 1 shows code snippets for a simple 3D data pro-cessing workflow. An implementation using the Open3 DPython interface is compared to an implementation us-ing the open3d C++ interface and to an implementationbased on PCL [18].

7 The implementation using the Open3 DPython interface is approximately half the length of the im-plementation using the open3d C++ interface, and aboutfive times shorter than the implementation based on an added benefit, the Python code can be edited and de-bugged interactively in a Jupyter FunctionalityOpen3D has nine modules, listed in Table DataTheGeometrymodule implements three geometric rep-resentations:PointCloud,TriangleMesh , structure has three data , , are used to store coordinates, normals, and colors. Themaster data field The other two fieldsare considered valid only when they have the same numberof records open3d provides directmemory access to these data fields via a numpy array. Thefollowing code sample demonstrates reading and accessingthe point coordinates of a point py3d import*import numpy as nppointcloud = read_point_cloud( )print( ( ))Similarly,TriangleMeshhas two master data fields , , structure is implemented as a 2D or 3 Darray and can be directly converted to a numpy array.

8 Apair of depth and color images of the same resolution can becombined into a data structure namedRGBDI mage. Sincethere is no standard depth image format, we have imple-mented depth image support for multiple datasets includingNYU [19], TUM [20], SUN3D [21], and Redwood [5]. Thefollowing code sample reads a pair of RGB-D images fromthe TUM dataset and converts them to a point py3d import*import numpy as npdepth = read_image( )color = read_image( )rgbd = create_rgbd_image_from_tum_format(color, depth)pointcloud = create_point_cloud_from_rgbd_image(rgbd, ) VisualizationOpen3D provides a functiondrawgeometries()for visu-alization. It takes a list of geometries as input, creates a win-dow, and renders them simultaneously using OpenGL. Wehave implemented many functions in the visualizer, suchas rotation, translation, and scaling via mouse operations,changing rendering style, and screen capture.

9 A code sam-ple usingdrawgeometries()and its result are shown in Fig-ure addition todrawgeometries(), open3d has anumber of sibling functions with more advanced ()allows the programmer to define a custom ()anddrawgeometrieswithkeycallback()acce ptPythoncallback functions as input. The callback function is calledin an automatic animation loop, or upon a key press following code sample shows a rotating point cloudusingdrawgeometrieswithanimationcal lback().2 DownsampleEstimatenormal(a) A simple 3D data Processing task: load a point cloud, downsample it, and estimate py3d import*pointcloud = read_point_cloud( )downsampled = voxel_down_sample(pointcloud, voxel_size = )estimate_normals(downsampled, KDTreeSearchParamHybrid(radius = , max_nn = 30))draw_geometries([downsampled])(b) An implementation using the open3d Python interface.

10 #include < >#include < >#include < >void main(int argc, char*argv[]){using namespace three;auto pointcloud = CreatePointCloudFromFile(argv[1]);auto downsampled = VoxelDownSample(pointcloud, )EstimateNormals(*downsampled, KDTreeSearchParamHybrid( , 30));DrawGeometry(downsampled);}(c) An implementation using the open3d C++ interface.#include < >#include < >#include <pcl/ >#include <pcl/ >#include <pcl/ >#include <pcl/ >void main(int argc, char*argv[]){using namespace pcl;PointCloud<PointNormal>::Ptr pcd(new PointCloud<PointNormal>());io::loadPCDFile<PointNormal>(argv[1],*pcd);VoxelGrid<PointNormal> grid; ( , , ); (pcd); (*pcd);PointCloud<PointNormal>::Ptr result(new PointCloud<PointNormal>());NormalEstimation<PointNormal, PointNormal> est; (pcd); ( ); (*result);pcl::visualization::PCLV isualizer viewer("PCLV iewer"); <PointNormal, PointNormal>(pcd, result, 1, );while (! ()){ ();}}(d) An implementation based on 1: Code snippets for a simple 3D data Processing workflow.