Narrow Field-of-View Camera Calibration

Narrow field-of-view (NFOV) cameras are naturally challenging to calibrate, due to an inability to observe the focal length parameter of the camera when using typical planar targets.

Before getting started, you should familiarize yourself with the camera calibration process by reviewing the Single Camera Calibration guide. Many of the same guidelines for data capture, best practices, and procedures apply here as well.

Tangram Vision Blog: NFOV Calibration Made Easy(er)

If you're looking for the "why", and not the "how" (this guide), we've got a detailed blog post which dives into the background and theory of narrow field-of-view camera calibration: Narrow FOV Calibration Made Easy(er)

Physical Requirements

NFOV camera calibration works by combining multiple calibration targets into a single object space. This allows the calibration process to observe more depth variation than any single target could provide, which is necessary to properly constrain the focal length parameter of the camera model.

The target consolidation process requires a surveying step to determine the relative positions of the targets in object space. Accordingly this process has the following additional requirements:

• Multiple calibration targets (at least two). These need to be compatible with simultaneous use, i.e. they should not have overlapping fiducial IDs. See Multiple Targets for more details.
• An additional camera that can be more easily calibrated (e.g., a wide field-of-view camera)

Don't worry about the mathematics; MetriCal actually already surveys your object space for every calibration! You just need to tell MetriCal to use those object spaces together instead of separately.

Procedure Overview

The general procedure is broken into two phases: a survey phase and an NFOV calibration phase. Note that these phases can be done simultaneously if using a wide FOV camera + narrow FOV camera stereo pair.

Phase 1: Survey

• A target field with targets at varying depths is constructed.
• Data is captured with a more easily calibrated camera (i.e. a wider FOV one).
• The camera is calibrated and the target field is surveyed with MetriCal either in one pass or separately. This is done with the Calibrate command.
• The target field survey and calibration are used to create a consolidated object space configuration for the target field using the Consolidate Object Spaces command.

Phase 2: NFOV Calibration

• Data is captured with the narrow field-of-view camera. The camera should observe the target field as it was surveyed. Generally the same sorts of data capture requirements apply as always. The image space should be well-covered.
• The narrow field-of-view dataset and consolidated object space configuration are used to calibrate the narrow field-of-view camera.

Target Visibility Requirements

Target Visibility On Survey: The survey needs to observe multiple targets within a single image to infer the relationship between them. This doesn't necessarily mean that all targets need to be visible at once; however, there must be some overlap between the targets in the surveying camera's field of view such that the relationship between all targets can be inferred.

Target Visibility on Calibration: Similarly, the narrow field-of-view camera data capture process should follow typical calibration best practices while also observing multiple targets in each frames. Observing a single target per frame is not sufficient to constrain the focal length parameter. Also it is very important that the target field is observed in the same physical configuration in which it was surveyed. If the targets move relative to each other between the survey and narrow field-of-view data capture, the calibration will be incorrect.

Procedure Breakdown

We'll do an example walkthrough of each step. For this example, we'll use a synthetically generated dataset with two AprilGrid targets and two cameras: an easily calibrated wide field-of-view camera (topic wfov) and a narrow field-of-view camera (topic nfov). The data has a a small amount distortion added to keep things interesting.

Setup

First, let's set up some paths and variables we'll use throughout the example. The process will vary slightly depending on the desired calibration workflow. This example script is somewhat verbose because it attempts to encompass a few different workflow options. In practice, this can be simplified based on your needs.

# Path to the directory containing the data
DATA="PATH-TO-DATA"

# There's only a single dataset containing both cameras.
# Some folks may wish to split or capture separate datasets.
NFOV_DATASET="$DATA"
WFOV_CAL_DATASET="$DATA"
WFOV_SURVEY_DATASET="$DATA"
WFOV_CAL_AND_SURVEY_DATASET="$DATA"

# Plexes initialized by `metrical init`
PLEX_WFOV_IN="$DATA/plex_init_wfov.json"
PLEX_NFOV_IN="$DATA/plex_init_nfov.json"

# Input plexes possibly modified by hand
PLEX_WFOV="$DATA/plex_wfov.json"
PLEX_NFOV="$DATA/plex_nfov.json"

# Calibrated plex for wfov
PLEX_WFOV_CAL="$DATA/plex_wfov_cal.json"

# Object space configurations
OBJECT="$DATA/object.json"
CONSOLIDATED_OBJECTS="$DATA/object_consolidated.json"

# Results
WFOV_CAL_RESULTS="$DATA/wfov_cal_results.mcap"
WFOV_SURVEY_RESULTS="$DATA/wfov_survey_results.mcap"
WFOV_CAL_AND_SURVEY_RESULTS="$DATA/wfov_survey_cal_results.mcap"
NFOV_RESULTS="$DATA/nfov_cal_results.mcap"

Init

We'll use metrical init to create plexes for each of the cameras. Normally, we suggest creating a plex for all your sensors at once, but in this case we'll deliberately create separate plexes for each camera. Since the NFOV camera isn't yet calibrated, it will have a deleterious effect on the surveying process if it's included. Generally, you should exclude anything but the surveying camera(s) from the plex used for surveying. Feel free to include other sensors in your NFOV plex if you want to.

# Create a plex for the wide field-of-view camera
metrical init -m"wfov":"opencv_radtan" "$DATA" "$PLEX_WFOV_IN"

# Create a plex for the narrow field-of-view camera
metrical init -m"nfov":"opencv_radtan" "$DATA" "$PLEX_NFOV_IN"

At this point you can adjust your plexes as needed. Feel free to add intrinsics values if you have a rough idea. MetriCal uses a very basic heuristic to initialize focal length. From the provided resolution it uses a typical pixel pitch value and assumes the lens has some focal length that is neither extremely wide nor extremely narrow. This guess is often good enough but in extreme cases it can be helpful to provide a better initial guess. We know from the synthesis process that the NFOV camera has a focal length of around 3248 pixels, so we'll put something in that ballpark in the NFOV plex.

# WFOV is fine as-is
cp $PLEX_WFOV_IN "$PLEX_WFOV"

# Set NFOV focal length to 2800 pixels with jq (or you can edit by hand)
jq '.components[0].camera.intrinsics.opencv_radtan.f=2800' "$PLEX_NFOV_IN" > "$PLEX_NFOV"

Calibrate and Survey

Next, we'll use metrical calibrate to calibrate the wide field-of-view camera and survey the target field. If your target field and camera are amenable to it, you can do this in one pass. Since this dataset is synthetic, it has no motion blur so we'll disable the camera motion filter.

metrical calibrate \
    "$WFOV_CAL_AND_SURVEY_DATASET" \
    "$PLEX_WFOV_IN" \
    "$OBJECT" \
    --camera-motion-filter=disabled \
    --results-path "$WFOV_CAL_AND_SURVEY_RESULTS"

or if you need to do it in two passes:

# Calibrate the wide field-of-view camera
metrical calibrate \
    "$WFOV_CAL_DATASET" \
    "$PLEX_WFOV_IN" \
    "$OBJECT" \
    --camera-motion-filter=disabled \
    --results-path \
    "$WFOV_CAL_RESULTS"

# Get the plex
mcap get attachment --name optimized-plex "$WFOV_CAL_RESULTS" > "$PLEX_WFOV_CAL"

# Survey the target field
metrical calibrate \
    "$WFOV_SURVEY_DATASET" \
    "$PLEX_WFOV_CAL" \
    "$OBJECT" \
    --camera-motion-filter=disabled \
    --results-path \
    "$WFOV_SURVEY_RESULTS"

Consolidate Object Spaces

Now that we have surveyed the target field, we can consolidate the object spaces into a single configuration using metrical consolidate-object-spaces. Passing the -r flag tells MetriCal to open Rerun and visualize the consolidated object space.

metrical consolidate-object-spaces \
    "$WFOV_CAL_AND_SURVEY_RESULTS" \
    -o "$CONSOLIDATED_OBJECTS" \
    -r

If you requested visualization, Rerun will pop up and you should see something like this:

Targets will be grouped together based on the mutual target observations in the data. As previously noted, we can only consolidate targets that were observed together in the same images. MetriCal will stitch together target groups even if they were not directly observed together, as long as there is a chain of mutual observations connecting them. So if we see target A and B together in some images, and targets B and C together in other images, MetriCal can infer the relationship between A and C, even if they were never observed together. If there are no mutual observations of a target, it will be left unconsolidated. Also it's possible to have multiple disjoint groups of consolidated targets if there are no mutual observations connecting them.

It's useful to inspect Rerun or the consolidated object space JSON to ensure that the intended consolidation takes place. If something looks wrong, you may need to recapture survey data with better target overlap.

Calibrate the Narrow Field-of-View Camera

Finally, we can use metrical calibrate again to calibrate the narrow field-of-view camera using the consolidated object space configuration.

metrical calibrate \
    "$NFOV_DATASET" \
    "$PLEX_NFOV" \
    "$CONSOLIDATED_OBJECTS" \
    --override-diagnostics \
    --camera-motion-filter=disabled \
    --results-path "$NFOV_RESULTS"

At this point you can inspect the results in Rerun or use the calibrated plex for your application.