Constraints

A constraint is a spatial, temporal, or semantic relation between any two components. In the context of interpreting plexes as a multigraph of relationships over our system (comprised of different components), constraints are the edge-relationships of the graph.

Conventions

From and To, To and From

Constraints are always across two components. MetriCal will often refer to each of these components in a directional sense using the "from" and "to" specifiers, which reference the UUIDs of the components.

This directional specifier is useful for spatial and temporal constraints, because it allows us to know how the extrinsics from the spatial constraints, or the synchronization data from the temporal constraints can be applied to data to put two observations into the same frame of reference.

Our extrinsics type is essentially a way to describe how to transform points in one coordinate system into another. Anyone who has ever worked with transforms has experienced confusion in convention. In order to cut through the ambiguity of extrinsics, every spatial constraint has a from and to field. Let's dive into how this works.

We can think of an extrinsics transform between components $A$ and $B$ using the following notation:

$$\Gamma_{B}^{A} \mathrel{:=} \Gamma_{from\ B}^{to\ A}$$

If we wanted to move a point $p$ from the frame of reference of component $B$ to that of $A$, we would use the following math:

$$\vec{p}_{W}^{A} = \Gamma_{B}^{A} \cdot \vec{p}_{W}^{B}$$

...also read as "$p_A$ equals $p_B$ by transforming to $A$ from $B$".

Thus, when constructing a spatial constraint, the reference frame for the extrinsics transform is in the coordinate frame of component $A$, and would move points from the coordinate frame of component $B$.

Similar examples can be made for converting timestamps from e.g. $B$ into the same "clock" as $A$ using temporal constraints.

Which way is Up?

It's common to represent a transform in a certain convention, such as FLU (X-Forward, Y-Left, Z-Up) or RDF (X-Right, Y-Down, Z-Forward). One might then wonder what the default coordinate system is for MetriCal. To that, there are two answers:

Simple answer: It's probably RDF.

More complete answer: MetriCal can work with any coordinate system as long as it's right-handed. The default coordinate basis is that of the data passed in. For camera systems, this means RDF. However, note that this can change depending on the modality, e.g. LiDAR-only systems that use FLU.

• Cameras are always RDF
• IMU is basis-dependent
• LiDAR are often FLU. However, Camera-LiDAR calibrations will be output in RDF.

Since MetriCal can't intuit the coordinate basis of your data (and sometimes you may not even know yourself!), we just treat all extrinsics like what they are: transforms between coordinate systems. If you know the basis of the modality going in, you should understand the basis coming out.

Spatial Constraints

It is common to ask for the spatial relationship or extrinsics between two given components. A Plex incorporates this information in the form of what is called spatial constraints. A spatial constraint can be broken down into:

Field	Type	Description
Extrinsics	An extrinsics object	The extrinsics describing the "To" from "From" transformation.
Covariance	A matrix of floats	The 6×6 covariance of the extrinsics described by this constraint.
From	UUID	The UUID of the component that describes the "From" or base coordinate frame.
To	UUID	The UUID of the component that describes the "To" coordinate frame, which we are transforming into. This can be considered the "origin" of the extrinsics matrix

Spatial Covariance

Spatial covariance is generally presented as a 6×6 matrix relating the variance-covariance of an se3 lie group:

$$\begin{bmatrix} v_1 & v_2 & v_3 & \omega_1 & \omega_2 &\omega_3 \end{bmatrix}$$

When traversing for spatial constraints within the Plex, the constraint returned will always contain the extrinsic with the minimum overall covariance. This ensures that users will always get the extrinsic that has the smallest covariance (thus, the highest confidence / precision), even if multiple spatial constraints exist between any two components.

Temporal Constraints

Time is a tricky thing in perception, but of crucial importance to get right. We've developed our temporal constraint to be flexible enough to describe many of the most common timing scenarios between components.

Field	Type	Description
Synchronization	A synchronization object	The strategy to achieve known synchronization between these two components in the Plex.
Resolution	float	The resolution to which synchronization should be applied.
From	UUID	The UUID of the component that the synchronization strategy must be applied to.
To	UUID	The UUID of the component whose clock we synchronize into by applying our synchronization strategy (to the from component).

The Problem With Clocks

In the world of hardware, measuring time can be a challenge. Two clocks might differ in several different ways; without taking these nuances into account, many higher-level perception tasks can fail.

Let's take the example below: two different clocks, possibly from two different hosts, that might be informing separate components in our plex.

Temporal constraints can balance these different clocks across a plex in order to make sure time confusion never occurs. It achieves this through Synchronization.

Synchronization

Synchronization describes the following relationship between two clocks:

$$C_{\text{to}} = (1e9 + \text{skew}) \cdot C_{\text{from}} + \text{offset}$$

Field	Type	Description
offset	Integer	The epoch offset between two clocks, in units of integer nanoseconds.
skew	Integer	The scale offset between two clocks. Unitless.

Offset

Unless two components are using the same clock, there's a chance that they are offset in time. This means that time t in one clock does not align with time t in the other. Fixing this is rather simple: just shift the time values in the from clock by the offset parameter until their two t times match.

Skew

Skew compensates for the difference in the duration of a time increment between two clocks. In other words, a second in one clock might be a different length than a second in another! These differences can be very subtle, but they will result in some unwanted drift.

Applying skew to a from clock's timestamps will match the duration of a second to that of the to clock.

Between skew and offset, we have the tools we need to synchronize time between two clocks! Note that components that use the same host clock will need no synchronization; their skew and offset remains 0.0.

Further Reading

MetriCal has adopted the terminology from this paper from the University of New Hampshire's InterOperability Laboratory.

Resolution

Resolution helps MetriCal identify observations that are meant to be synchronized between two components.

Say we have two camera components. The first is producing one image every 5 seconds; the second produces a new image every 1.3 seconds. We want to pair up observations from the two separate streams that we know are uniquely synced in time as a one-to-one set.

Our resolution tells the Platform how far from an observation we want to search for a synced pair. In the case of our first camera, we know that one new frame comes every 5 seconds. This means that there's a span of 2.5 seconds on either side of this image that could hold a matching observation from our second camera. So, we set resolution to 2.5 * 1e9 (for nanoseconds).

The Platform will then look for any synced observation candidates in camera two and find the observation that matches most closely in time to the image in camera one.

All that being said, resolution is a concept better shown than told:

If one is confident that two observation streams are in-sync, one may set the resolution to be fairly small. However, given the way some components can behave, it's generally not necessary or recommended.

Semantic Constraints

Semantic constraints are a bit different from spatial and temporal constraints. While spatial and temporal constraints exist to model the physical realities of components in a system, semantic constraints exist to model relationships in the system that don't fall within those boundaries.

Semantic constraints are defined by the following fields:

Field	Type	Description
Components	An array of component UUIDs	A set of unique UUIDs (corresponding to components that exist within the plex) that are grouped under this semantic.
Name	String	The name for these semantics. Usually indicates the purpose or function of the group.
UUID	UUID	The unique identifier for the semantic constraint.

Semantic constraints can label subplexes within a plex, label individual OEM pieces of hardware (e.g. labeling a single Tangram Vision HiFi within a plex), or to group together components that share some common function (e.g. grouping a stereo pair of cameras together).

Some semantic constraints can be used by MetriCal to generate unique kinds of metrics after calibration. These are identified purely through the semantic constraint's name field. For example, MetriCal currently understands the following semantic constraint kinds:

• "stereo_pair": This denotes that two cameras are part of a stereo pair. This also tells MetriCal to compute stereo rectification metrics after a calibration is complete.