Constraints
A constraint is a spatial, temporal, or semantic relation between any two components. In the context of interpreting plexes as a multigraph of relationships over our system (comprised of different components), constraints are the edge-relationships of the graph.
Conventions
From and To, To and From
Constraints are always across two components. MetriCal will often refer to each of these components in a directional sense using the "from" and "to" specifiers, which reference the UUIDs of the components.
This directional specifier is useful for spatial and temporal constraints, because it allows us to know how the extrinsics from the spatial constraints, or the synchronization data from the temporal constraints can be applied to data to put two observations into the same frame of reference.
Our extrinsics type is essentially a way to describe how to transform points in one coordinate
system into another. Anyone who has ever worked with transforms has experienced confusion in
convention. In order to cut through the ambiguity of extrinsics, every spatial constraint has a
from
and to
field. Let's dive into how this works.
We can think of an extrinsics transform between components and using the following notation:
If we wanted to move a point from the frame of reference of component to that of , we would use the following math:
...also read as " equals by transforming to from ".
Thus, when constructing a spatial constraint, the reference frame for the extrinsics transform is in the coordinate frame of component , and would move points from the coordinate frame of component .
Similar examples can be made for converting timestamps from e.g. into the same "clock" as using temporal constraints.
Coordinate Bases in MetriCal
It's common to represent a transform in a certain convention, such as FLU (X-Forward, Y-Left, Z-Up) or RDF (X-Right, Y-Down, Z-Forward). One might then wonder what the default coordinate system is for MetriCal. Short answer: it entirely depends on the data you're working with.
In MetriCal, spatial constraints are designed to transform observations (not components!) from the
from
frame to the to
frame. In the case of a camera-LiDAR extrinsics transform ,
the solved extrinsics will move LiDAR observations (which may be in FLU) to a camera
observations's coordinate frame (which may be in RDF):
This makes it simple to move observations from one component to another for sensor fusion tasks. This is what is meant when MetriCal is said to have no "default" component coordinate system; it operates directly on the provided observations!
However, for those users that really need the extrinsics in a specific coordinate basis, MetriCal provides. There are two relevant global options that can be passed to any mode that produces or displays a calibrated plex (Calibrate, Shape, Pretty Print, and Evaluate). These are:
--topic-to-observation-basis
(or-z
): Assign a topic a specific coordinate basis for its observations, i.e. the data it produces. For cameras, this is usually RDF.--topic-to-component-basis
(or-Z
): Assign a topic a specific coordinate basis for its component, i.e. the sensor itself.
If both of these options are set for every component, MetriCal will transform the extrinsics into the proper basis for you! Let's take a look at an example:
metrical calibrate -z ir_cam:RDF -z lidar:FLU -Z *:FLU $DATA $PLEX $OBJ
Here, we're telling MetriCal that the observations from the camera topic are in RDF, while the data from our LiDAR is in FLU. We're also telling MetriCal that we need the output of the entire plex to be in FLU, probably because our system expects that basis natively.
In this case, MetriCal will produce two plexes in the results JSON:
results.plex
- The first plex is what is natively produced by the calibration process, with extrinsics in the basis of the observations.results.changed_basis_plex
- The second plex is the same as the first, but with all extrinsics transformed into the desired component bases.
Note that this second plex is for your use only; if you use this in Init or Calibrate mode, MetriCal
will probably start a little confused. It needs results.plex
for proper operation.
Spatial Constraints
It is common to ask for the spatial relationship or extrinsics between two given components. A Plex incorporates this information in the form of what is called spatial constraints. A spatial constraint can be broken down into:
Field | Type | Description |
---|---|---|
Extrinsics | An extrinsics object | The extrinsics describing the "To" from "From" transformation. |
Covariance | A matrix of floats | The 6×6 covariance of the extrinsics described by this constraint. |
From | UUID | The UUID of the component that describes the "From" or base coordinate frame. |
To | UUID | The UUID of the component that describes the "To" coordinate frame, which we are transforming into. This can be considered the "origin" of the extrinsics matrix |
Spatial Covariance
Spatial covariance is generally presented as a 6×6 matrix relating the variance-covariance of an se3 lie group:
When traversing for spatial constraints within the Plex, the constraint returned will always contain the extrinsic with the minimum overall covariance. This ensures that users will always get the extrinsic that has the smallest covariance (thus, the highest confidence / precision), even if multiple spatial constraints exist between any two components.
Since all spatial constraints in a plex form an undirected (maybe even fully connected) graph, it can be confusing to figure out how to traverse that graph to find the "best" extrinsics between two components. MetriCal itself provides the Shape mode to help with this (with a few helpful options in Pretty Print), but sometimes it's useful to do your own thing.
To help would-be derivers, the
MetriCal Sensor Calibration Utilities
repository contains the spatial_constraint_traversal
directory, which demonstrates the right way
to derive optimal extrinsics straight from the Plex JSON via the magic of python.