Plexes

MetriCal uses what is called a Plex as a description of the spatial, temporal, and semantic relationships within your perception system. In short, a Plex is a representation of the physical system that you are calibrating. It is a graph of components and constraints that fully describe the perception system being optimized by the Tangram Vision Platform.

Systems and Plexes have a one-to-one relationship. This means that a Plex can only describe one perception system, and a system should be fully described by one Plex. For example, the above plex description could be two cameras:

Why Plex?

A common occurrence in the perception industry is that the words sensor and device are often ambiguous and used interchangeably. This can often lead to confusion, which is the last thing one wants in a complicated perception system. To that end, we don't use these terms at all! "Plex" is sufficient.

Here's a quick example: While a Tangram Vision HiFi could be referred to as a single "device" or "sensor", it's really a combination of several constituent components:

• Two infrared cameras
• A color camera
• An accelerometer and gyroscope (6-DoF IMU)
• A "depth" stream, synthesized from observations of the two infrared cameras

Each of these components are represented independently in the Plex, with relations denoted separately as well.

We can even add an entire new HiFi module to the Plex with very little effort. Just create the components and constraints for the new HiFi!

This formulation helps prevent ambiguity within MetriCal. For example, one might ask for "the extrinsics between two HiFi. This is not a useful question, since there are many ways to define what that extrinsic may be dependent upon an assumed convention. Instead, with a plex we can ask for the "extrinsics between color camera with UUID X and color camera with UUID Y", which is specific and unambiguous.

How is the Plex used?

Plexes are used by MetriCal as a representation of your system. For this reason, plexes are both input into MetriCal during calibration, but also generated by MetriCal as a calibration result. By using the plex, we can gauge the following prior to a calibration:

• Initial intrinsic values, as well as their prior (co)variances
• Initial extrinsic values, as well as their prior (co)variances
• The temporal relationships between components in the system, which informs how MetriCal estimates "synchronized" observations (used to infer spatial relationships).
• Semantic relationships between components (e.g. a pair of cameras comprising a stereo pair), which can inform MetriCal e.g. which cameras to generate stereo pair metrics between.

This makes the plex very flexible, in that it can be used to configure and describe parts of the calibration problem as a function of the spatial, temporal, and semantic relationships present within your system. It is a declarative means to describing how components are related.

Likewise, MetriCal will also use the exact same plex format as part of the output to calibration (e.g. when you call metrical calibrate). From the output plex, we can gauge the following:

• Final calibrated intrinsics values, as well as their posterior covariances
• Final calibrated extrinsics values, as well as their posterior covariances

The plex is a convenient means to serialize, copy, share, and parse known information about a system. Many of MetriCal's features directly produce or consume a plex.

Sub-Plexes

MetriCal always interprets a Plex at face value —we never assume to know your system better than you do; However, this can lead to some awkward situations when trying to profile your system. Here's the same HiFi that we've been looking at with only a few constraints added:

Instead of one fully connected Plex, we have many subplexes.

This can be interpreted by MetriCal a few ways:

1. The missing constraints between e.g. color and infrared cameras cannot be known.
2. The missing constraints between e.g. color and infrared cameras are unimportant / should not be calibrated ever.
3. The missing constraints between e.g. color and infrared cameras were not added for some other reason (e.g. they were forgotten, sync was not working, etc.).

In any case, this will inform MetriCal's behaviour. Calibration, for instance, will look to fill in all possible constraints between components; As a conseuqence, if two components do not have any possible spatial constraint path between them, then MetriCal will not make attempts to infer a relative extrinsic or spatial constraint between them in the output.

Plex Conversion

While plexes are a convenient format when working with MetriCal, other software may not support Plexes as a format. In these instances, it can be useful to convert parts or the whole plex into a more usable format. Some examples can be:

• Converting the intrinsics and distortion parameters for a given camera into a look-up table for fast image rectification.
• Converting the intrinsics and distortion parameters for a given stereo pair into a look-up table for fast stereo rectification.
• Converting a plex into a URDF for use with ROS / ROS2 tooling.

See the metrical shape docs for more information on how to extract or convert information from the plex.

Serialization and Schema

Plex data is serialized as JSON for the convenience of being able to store and edit the plex in a text-based format. This can bring about some complications: while JSON is a plaintext format that is well-suited to manipulation (and to some extent, reading), it is loosely defined and the internal representation of what is inside a JSON file or object is not always easy to understand.

Example JSON

{
  "uuid": "5d71519a-31c0-4b71-b629-1195470330f1",
  "creation_timestamp": 1634855338595628000,
  "components": [
    // array of components
  ],
  "spatial_constraints": [
    // array of spatial constraints
  ],
  "temporal_constraints": [
    // array of temporal constraints
  ],
  "semantic_constraints": [
    // array of semantic constraints
  ]
}

Field Descriptions

Field	Type	Description
"uuid"	String	A universally unique identifier for the Plex.
"creation_timestamp"	integer	The creation timestamp of the Plex, from unix epoch, in nanoseconds.
"components"	Component Object	An array of JSON objects describing components within the Plex.
"spatial_constraints"	Array of Spatial Constraint Objects	An array of JSON objects describing spatial constraints within the Plex.
"temporal_constraints"	Array of Temporal Constraint Objects	An array of JSON objects describing temporal constraints within the Plex.
"semantic_constraints"	Array of Semantic Constraint Objects	An array of JSON objects describing semantic constraints within the Plex.

danger

It is an error if the components array is empty and either of the constraints arrays are non-empty.