Narrow Field-of-View Camera Calibration

Narrow field-of-view (NFOV) cameras are naturally challenging to calibrate, due to an inability to observe the focal length parameter of the camera when using typical planar targets.

Prerequisites

Before getting started, you should familiarize yourself with the camera calibration process by reviewing the Single Camera Calibration guide. Many of the same guidelines for data capture, best practices, and procedures apply here as well.

Tangram Vision Blog: NFOV Calibration Made Easy(er)

If you're looking for the "why", and not the "how" (this guide), we've got a detailed blog post which dives into the background and theory of narrow field-of-view camera calibration: Narrow FOV Calibration Made Easy(er)

Physical Constraints

Narrow field-of-view means a longer focal length, and longer focal lengths require more depth variation in each target to calibrate correctly. However, calibrating very long focal lengths, e.g. with NFOV cameras with 25m depths of field, would require a massive board setup to achieve sufficient depth variation. Think 8m$^2$ boards placed 25m away from the camera!

MetriCal gets around this by allowing you to combine multiple targets at different depths into a single object space; we call this consolidation. By effectively surveying multiple targets at different depths, we can achieve sufficient depth variation to constrain the focal length parameter without requiring an impractically large target. There's a whole command dedicated to merging calibrated object spaces like this, and we'll be leveraging it here.

Given the nature of NFOV camera calibration, the process has the following additional physical requirements:

• Multiple calibration targets (at least two). These need to be compatible with simultaneous use, i.e. they should not have overlapping fiducial IDs. See Multiple Targets for more details.
• An additional camera that can be more easily calibrated, e.g., a wide field-of-view camera

Here's a snapshot from our synthetic example dataset to give you an idea of a typical target setup for NFOV calibration:

Procedure Overview

The general procedure is broken into two phases: a survey phase and an NFOV calibration phase. Note that this data can be gathered from the same run if using a wide FOV camera + narrow FOV camera stereo pair. This is what we do in our example dataset.

Phase 1: Survey

• A target field with targets at varying depths is constructed.
• Data is captured with a more easily calibrated camera (i.e. a wider FOV one).
• The camera is calibrated and the target field is surveyed with MetriCal either in one pass or separately. This is done with the Calibrate command.
• The target field survey and calibration are used to create a consolidated object space configuration for the target field using the Consolidate Object Spaces command.

Phase 2: NFOV Calibration

• Data is captured with the narrow field-of-view camera. The camera should observe the target field as it was surveyed. Generally the same sorts of data capture requirements apply as always. The image space should be well-covered.
• The narrow field-of-view dataset and consolidated object space configuration are used to calibrate the narrow field-of-view camera.

Target Visibility Requirements

Target Visibility On Survey: The survey needs to observe multiple targets within a single image to infer the relationship between them. This doesn't necessarily mean that all targets need to be visible at once; however, there must be some overlap between the targets in the surveying camera's field of view such that the relationship between all targets can be inferred.

Target Visibility on Calibration: Similarly, the narrow field-of-view camera data capture process should follow typical calibration best practices while also observing multiple targets in each frames. Observing a single target per frame is not sufficient to constrain the focal length parameter. Also it is very important that the target field is observed in the same physical configuration in which it was surveyed. If the targets move relative to each other between the survey and narrow field-of-view data capture, the calibration will be incorrect.

Example Dataset and Manifest

This dataset features:

• Observations as a folder of folders
• One wide field-of-view camera, which is designated as the surveying camera
• One narrow field-of-view camera with a focal length of approximately 3248px (very narrow!)
• Two aprilgrid targets positioned 2m away from one another in depth

We've also added a small amount of radial distortion to the cameras to keep things interesting.

📁Download: Narrow FOV Camera Example Dataset

The Manifest

This manifest is larger than most, since it covers both the surveying and narrow field-of-view calibration phases. We'll break it down step-by-step in the sections that follow.

narrow_fov_manifest.toml

[project]
name = "MetriCal Demo: Narrow FOV Manifest"
version = "15.0.0"
description = "Manifest for running MetriCal for a camera with a narrow field of view."
workspace = "metrical-results"

## === VARIABLES ===
[project.variables.dataset]
description = "Path to the input dataset containing calibration data."
value = "narrow_fov_observations"

[project.variables.object-space]
description = "Path to the input object space JSON file."
value = "narrow_fov_objects.json"

## === STAGES ===
[stages.survey-camera-init]
command = "init"
dataset = "{{variables.dataset}}"
topic-to-model = [["survey_camera", "opencv-radtan"]]
... # ...more options...
initialized-plex = "{{auto}}"

[stages.nfov-camera-init]
command = "init"
dataset = "{{variables.dataset}}"
topic-to-model = [["nfov_camera", "opencv-radtan"]]
... # ...more options...
initialized-plex = "{{auto}}"

[stages.survey-camera-calibrate]
command = "calibrate"
dataset = "{{variables.dataset}}"
input-plex = "{{survey-camera-init.initialized-plex}}"
input-object-space = "{{variables.object-space}}"
camera-motion-threshold = "disabled"
... # ...more options...
detections = "{{auto}}"
results = "{{auto}}"

[stages.consolidate]
command = "consolidate"
input-object-space = "{{survey-camera-calibrate.results}}"
consolidated-object-space = "{{auto}}"
overwrite = true

[stages.nfov-camera-calibrate]
command = "calibrate"
dataset = "{{variables.dataset}}"
input-plex = "{{nfov-camera-init.initialized-plex}}"
input-object-space = "{{consolidate.consolidated-object-space}}"
camera-motion-threshold = "disabled"
... # ...more options...
detections = "{{auto}}"
results = "{{auto}}"

Running the Manifest

Run your manifest as usual with MetriCal:

metrical run narrow_fov_manifest.toml

Phase 1: Survey

Our first stages, survey-camera-init and nfov-camera-init, will create individual plexes for each of the cameras. Normally, we suggest creating a plex for all your sensors at once, but we only want to use the surveying camera (our wide field-of-view camera) for the surveying phase of this procedure. Thus, we will deliberately leave the narrow field-of-view camera out of the first calibration phase.

Survey, then Whatever

Once your surveying camera is calibrated, and the target field is surveyed, you can use the consolidated object space to calibrate any type of camera, not just narrow field-of-view. However, narrow field-of-view cameras are the only type that really require the surveyed space.

Next, the survey-camera-calibrate stage does two things (as all calibrate stages do): it calibrates the wide field-of-view camera and surveys the target field, aka optimizing the plex and object space.

However, an optimized object space is still made up of separate objects; we run the consolidate stage to merge it into one consolidated object space configuration, a single feature that our narrow field-of-view camera can use to calibrate against.

If you requested visualization during the consolidate command, Rerun will pop up and you should see something like this:

Targets will be grouped together based on the mutual target observations in the data. As previously noted, we can only consolidate targets that were observed together in the same images. MetriCal will stitch together target groups even if they were not directly observed together, as long as there is a chain of mutual observations connecting them. So if we see target A and B together in some images, and targets B and C together in other images, MetriCal can infer the relationship between A and C, even if they were never observed together. If there are no mutual observations of a target, it will be left unconsolidated. Also it's possible to have multiple disjoint groups of consolidated targets if there are no mutual observations connecting them.

It's useful to inspect Rerun or the consolidated object space JSON to ensure that the intended consolidation takes place. If something looks wrong, you may need to recapture survey data with better target overlap.

Phase 2: NFOV Calibration

From here, calibration mostly proceeds as normal. The nfov-camera-calibrate stage uses the narrow field-of-view camera data and the consolidated object space configuration to calibrate the camera.

Sure enough, we get good results:

Such narrow! So fov! Wow. For reference, the simulated narrow field-of-view camera had a focal length of 3248px. This means that, on our synthetic dataset, MetriCal produced a focal length error of around 0.097%. Not bad!

Surveying: It's Worth It

This procedure may seem a bit involved, but we believe this is one of the most practical ways to calibrate narrow field-of-view cameras with high accuracy. By leveraging a more easily calibrated camera to survey the target field, we can create a rich object space that allows us to accurately calibrate cameras that would otherwise be very challenging to handle.

Don't think it makes a difference? Try for yourself: modify the manifest to skip the surveying phase and use our regular ol' object space for the narrow field-of-view calibration. Let us know how it goes!