Convert and visualise in realtime Realsense poincloud to Open3D pointcloud

[marihstr]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for main branch).

My Question

I wish to convert and visualise in realtime Realsense poincloud to Open3D pointcloud. This I can easily do if I were to first save the realsense pointcloud as a .ply file and import this as a open3d pointcloud, but I don't know how to perform the direct conversion.

I've been looking at many different issues that have looked at this case (such as #473) and feel that the following code block should be the solution:

import pyrealsense2 as rs
import numpy as np
import open3d as o3d

# Pipeline & config
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

profile = pipeline.start(config)

profile = pipeline.get_active_profile()
depth_profile = rs.video_stream_profile(profile.get_stream(rs.stream.depth))
depth_intrinsics = depth_profile.get_intrinsics()
w, h = depth_intrinsics.width, depth_intrinsics.height

pc = rs.pointcloud()

try:
    vis = o3d.visualization.Visualizer()
    vis.create_window("RealSense + Open3D")
    pcd = o3d.geometry.PointCloud()

    while True:
        vis.add_geometry(pcd)
        pcd.clear()

        frames = pipeline.wait_for_frames()
        color = frames.get_color_frame()
        depth = frames.get_depth_frame()
        if not color or not depth:
            continue

        pc.map_to(color)
        points = pc.calculate(depth)
        vtx = np.asarray(points.get_vertices(2)).reshape(h, w, 3) 
        pcd.points = o3d.utility.Vector3dVector(vtx)
        vis.update_geometry()
        vis.poll_events()
        vis.update_renderer()

finally:
    pipeline.stop()

However, I received the following error from the line pcd.points = o3d.utility.Vector3dVector(vtx):

RuntimeError: Unable to cast Python instance of type <class 'numpy.ndarray'> to C++ type