metrical evaluate

Purpose

• Evaluate the quality of a calibration on a test dataset.

Usage

metrical evaluate [OPTIONS] <INPUT_DATA_PATH> <PLEX_PATH> <OBJECT_SPACE_PATH>

Description

Evaluate can apply a calibration to a given dataset and produce metrics to validate the quality of the calibration. This is commonly referred to validating calibration on a "test" dataset, rather than using the "training" dataset that produced the calibration in the first place.

An evaluation optimization runs the same adjustment as a metrical calibrate run, with one crucial difference: it will not solve for any values in the plex. Instead, it will instead fix these values and use them to generate metrics.

When evaluating, make sure to use an optimized plex JSON extracted from a previous MetriCal run for your system. Use jq to extract this data into its own file for easy analysis and comparison to the original inputs.

Install jq using apt:

sudo apt install jq

Then use jq to extract the relevant optimized data.

jq .plex results.json > optimized_plex.json
jq .object_space results.json > optimized_obj.json

Arguments

<INPUT_DATA_PATH>

Input data path for this evaluation.

MetriCal accepts a few data formats:

• Ros1 bags, in the form of a .bag file.
• Ros2 bags, in the form of a .mcap file.
• Folders, as a flat directory of data. The folder passed to MetriCal should itself hold one folder of data for every component.

In all cases, the topic/folder name must match a named component in the plex in order to be matched correctly. If this is not the case, there's no need to edit the plex directly; instead, one may use the --topic_to_component flag.

[PLEX_PATH]

The plex.json used as a prior for MetriCal's adjustment.

[OBJECT_SPACE_PATH]

A path pointing to a description of the object space for the adjustment. This should be a JSON file.

Options

-d, --disable-filter

Disable data filtering based on scene motion. This is useful for datasets that are a series of snapshots, rather than continuous motion.

-f, --cache-filtered-data

Enable caching of filtered images. The application caches filtered images as pngs on disc in the default directory cached_data and in subdirectories corresponding to the component name as stored in the plex. This default directory can be overridden by using the --cache-dir option.

-F, --cache-dir <CACHE_DIR>

The output directory for cached filtered images. [default: "cached_data"]

-t, --stillness-threshold <STILLNESS_THRESHOLD>

The upper bound of the average flow to classify an image as still. This threshold is used for filtering data based on scene motion. This threshold applies to the average of the magnitudes of the flow field between subsequent images. The units for this threshold are in pixels/frame. [default: 1]

-o, --output-json <OUTPUT_JSON>

The output path to save the final JSON output of the program. [default: results.json]

-T, --topic-to-component <topic_name:component_name>

A mapping of ROS topic/folder names to component names/UUIDs in the input plex.

MetriCal only parses data that has a topic-component mapping. Ideally, topics and components share the same name. However, if this is not the case, use this flag to map topic names from the dataset to component names in the plex.

-v, --render

Whether to visualize the current command using Rerun.

--render-socket <RENDER_SOCKET>

The web socket address on which Rerun is listening.

This should be an IP address and port number separated by a colon, e.g. --render-socket="127.0.0.1:3030". By default, Rerun will listen on socket host.docker.internal:9876.

When running Rerun from its CLI, the IP would correspond to Rerun's --bind option and the port would correspond to its --port option.

Examples

1. Run an evaluation, and render the calibration process and results

metrical evaluate -v    \
    --render-results    \
    $EVAL_DATA $OPTIMIZED_PLEX $OBJSPC