Code Samples

For code samples of LUT-based image manipulation, please check out our lut-examples repository.

Stereo Lookup Table

Usage

Stereo Lookup Table - CLI Example

metrical shape stereo-lut [OPTIONS] \
    --dominant <DOMINANT> \
    --secondary <SECONDARY> \
    <INPUT_PLEX> <ARTIFACT>

Stereo Lookup Table - Manifest Example

command = "shape-stereo-lut"
input-plex = "{{calibrate-stage.results}}"
format = "json|msgpack"
dominant = "camera_to_use_as_dominant_eye"
secondary = "camera_to_use_as_secondary_eye"
artifact = "{{auto}}"

Purpose

This command creates two pixel-wise lookup tables to produce a stereo rectified pair based on an existing calibration. Rectification occurs in a hallucinated frame with a translation at the origin of the dominant eye, but with a rotation halfway between the dominant frame and the secondary frame.

Rectification is the process of transforming a stereo pair of images into a common plane. This is useful for a ton of different applications, including feature matching and disparity estimation. Rectification is often a prerequisite for other computer vision tasks.

The Stereo LUT subcommand will create two lookup tables, one for each camera in the stereo pair. These LUTs are the same format as the ones created by the lut command, but they map the pixel-wise shift needed to move each image into that common plane. The result is a pair of rectified images!

The Stereo LUT subcommand will also output the values necessary to calculate depth from disparity values.

Pixel Convention

The pixel-coordinate fields in the stereo-lut artifact use OpenCV's center pixel convention: the integer coordinate (0, 0) refers to the center of the top-left pixel, and every pixel center falls on an integer coordinate. The fields written in this convention are:

• rectifier_luts[Primary].remap_rows and rectifier_luts[Primary].remap_columns
• rectifier_luts[Secondary].remap_rows and rectifier_luts[Secondary].remap_columns
• virtual_cx and virtual_cy

virtual_focal_length_pixels, baseline, and alignment describe scales, distances, and orientation rather than pixel offsets, so the convention does not apply to them.

In practice this means the artifact is ready to consume with OpenCV directly. Each per-eye LUT can be handed to cv::remap without any sub-pixel correction, and virtual_focal_length_pixels, virtual_cx, and virtual_cy can be assembled into an OpenCV camera matrix and fed to cv::triangulatePoints, cv::reprojectImageTo3D, or any equivalent routine. The rectified imagery produced by cv::remap is self-consistent with that camera matrix.

Differs from MetriCal's general convention

This is specific to the stereo-lut output. MetriCal's other artifacts — including plexes and the intrinsics they carry, calibration results, and reported pixel-space residuals — use the corner pixel convention, where (0, 0) is the top-left corner of the top-left pixel and pixel centers fall at integer + 0.5. The stereo-lut artifact is shifted by -0.5 in each axis relative to that convention so it lands in OpenCV's center convention at the file boundary.

If you compare a virtual_cx / virtual_cy from this output to a cx / cy taken directly from a plex (or feed pixel coordinates between the two), account for the half-pixel offset between the two conventions.

Examples

Create a rectification lookup table between a stereo pair

metrical shape stereo-lut --dominant ir_one --secondary ir_two \
   <INPUT_PLEX> <ARTIFACT>

Arguments

[INPUT_PLEX]

The path to the input plex. This can be a MetriCal Results MCAP or a plex JSON.

[ARTIFACT]

The artifacts output location.

Options

Global Arguments

As with every command, all global arguments are supported (though not all may be used).

-a, --dominant [DOMINANT]

The dominant eye in this stereo pair. Must be a camera component specifier (UUID or component name)

-b, --secondary [SECONDARY]

The secondary eye in this stereo pair. Must be a camera component specifier (UUID or component name)

-f, --format [FORMAT]

Default: json

What serializer to use to format the output. Possible values:

• json: Output the shape data as JSON
• msgpack: Output the shape data as MsgPack

--focal [F]

Focal length of the virtual output camera. If not specified, uses the average of the two cameras' focal lengths.

--cx [CX]

Principal point X offset of the virtual output camera. If not specified, uses the average of the two cameras' cx values.

--cy [CY]

Principal point Y offset of the virtual output camera. If not specified, uses the average of the two cameras' cy values.

--width [WIDTH]

Width of the output image. If not specified, uses the dominant camera's width.

--height [HEIGHT]

Height of the output image. If not specified, uses the dominant camera's height.

Output Schema

Loading ....

Raw schema file