Skip to content

feat: add CameraLabelsComponent for typed per-camera image labels#84

Open
janickm wants to merge 3 commits intoNVIDIA:mainfrom
janickm:janickm/camera-labels-component
Open

feat: add CameraLabelsComponent for typed per-camera image labels#84
janickm wants to merge 3 commits intoNVIDIA:mainfrom
janickm:janickm/camera-labels-component

Conversation

@janickm
Copy link
Copy Markdown
Collaborator

@janickm janickm commented Apr 29, 2026

Summary

Add a new V4 CameraLabelsComponent for storing per-camera image-aligned labels (depth maps, optical flow, segmentation masks, surface normals, etc.) with typed, schema-validated storage.

  • Tagged-union type system: LabelCategory enum (DEPTH, FLOW, SEGMENTATION, MASK, GEOMETRY, MATERIAL, FEATURE, OTHER) + free-form qualifier string. New label variants require zero code changes.
  • Dual encoding: RAW array storage (Blosc lz4) and IMAGE_ENCODED (PNG, JPEG) with handle-based deferred decoding for GPU pipelines.
  • Independent timestamps: Each label instance has its own timestamps, decoupled from camera frame rate.
  • Ergonomic compat API: CameraLabelsProtocol, CameraLabel sub-protocol, query_camera_labels(camera_id, label_type?, label_category?).

Commits

  1. CameraLabelsComponent with tagged-union type system, compat layer, and tests — Core implementation: LabelCategory, LabelType(category, qualifier, unit), CameraLabelDescriptor, LabelSchema, CameraLabelsComponent (Writer/Reader), CameraLabelsProtocol, CameraLabel protocol, SequenceLoaderV4 integration, ComponentGroupAssignments extension, 12 round-trip tests.

  2. Migrate Waymo panoptic segmentation from generic_data to CameraLabelsComponent — First real-world usage: Waymo converter now stores panoptic_label_png as SEGMENTATION/"panoptic" with IMAGE_ENCODED/png encoding. Per-label metadata carries the panoptic_label_divisor. Class-ID map moved from camera sensor metadata to label component metadata.

  3. Add camera label overlay support to ncore_vis — CameraComponent gains a "Camera Labels" overlay option with label source dropdown, opacity slider, and replace-image toggle. Category-aware rendering: depth (TURBO colormap), segmentation (20-color palette), masks (green tint), geometry/normals (RGB mapping), generic (grayscale).

Design

See Camera Labels Component Design wiki page for full design spec including dataverse, sauron, and NRE compatibility tables.

Files Changed

Area Files
Type system ncore/impl/data/types.py
Component ncore/impl/data/v4/components.py
Compat layer ncore/impl/data/compat.py, ncore/impl/data/v4/compat.py
Group assignments ncore/impl/data/v4/types.py
Public API ncore/data/__init__.py, ncore/data/v4/__init__.py
Tests ncore/impl/data/v4/components_test.py
Waymo converter tools/data_converter/waymo/converter.py
Visualization tools/ncore_vis/components/camera.py, tools/ncore_vis/data_loader.py
Other converters tools/data_converter/{colmap,pai}/converter.py (added camera_labels_ids=[])

@janickm janickm force-pushed the janickm/camera-labels-component branch 2 times, most recently from d764527 to 545a6d5 Compare April 30, 2026 13:50
@janickm janickm force-pushed the janickm/camera-labels-component branch from ebb0949 to 91a6e50 Compare April 30, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant