# Temporal Constraints

Time is a tricky thing in perception, but of crucial importance to get right. We've developed our temporal constraint to be flexible enough to describe many of the most common timing scenarios between components.

Field
Type
Description
Synchronization Synchronization The strategy to achieve known synchronization between these two components in the Plex.
Resolution f64 The resolution to which synchronization should be applied.
From Uuid The UUID of the component that the synchronization strategy must be applied to.
To Uuid The UUID of the component whose clock we synchronize into by applying our synchronization strategy (to the from component).

## The Problem With Clocks

In the world of hardware, measuring time can be a challenge. Two clocks might differ in several different ways; without taking these nuances into account, many higher-level perception tasks can fail.

Let's take the example below: two different clocks, possibly from two different hosts, that might be informing separate components in our plex.

Temporal constraints can balance these different clocks across a plex in order to make sure time confusion never occurs. It achieves this through Synchronization.

## Synchronization

Synchronization describes the following relationship between two clocks:

$C_{\mbox{to}} = (1e9 + \mbox{skew}) * C_{\mbox{from}} + \mbox{offset}$
Field
Type
Description
offset i64 The epoch offset between two clocks, in units of integer nanoseconds.
skew i64 The scale offset between two clocks. Unitless.

### Offset

Unless two components are using the same clock, there's a chance that they are offset in time. This means that time t in one clock does not align with time t in the other. Fixing this is rather simple: just shift the time values in the from clock by the offset parameter until their two t times match.

### Skew

Skew compensates for the difference in the duration of a time increment between two clocks. In other words, a second in one clock might be a different length than a second in another! These differences can be very subtle, but they will result in some unwanted drift.

Applying skew to a from clock's timestamps will match the duration of a second to that of the to clock.

Between skew and offset, we have the tools we need to synchronize time between two clocks! Note that components that use the same host clock will need no synchronization; their skew and offset remains 0.0.

The Platform has adopted the terminology from this paper from the University of New Hampshire's InterOperability Laboratory.

## Resolution

Resolution helps the Tangram Vision Platform identify observations that are meant to be synchronized between two components.

Say we have two camera components. The first is producing one image every 5 seconds; the second produces a new image every 1.3 seconds. We want to pair up observations from the two separate streams that we know are uniquely synced in time as a one-to-one set.

Our resolution tells the Platform how far from an observation we want to search for a synced pair. In the case of our first camera, we know that one new frame comes every 5 seconds. This means that there's a span of 2.5 seconds on either side of this image that could hold a matching observation from our second camera. So, we set resolution to 2.5 * 1e9 (for nanoseconds).

The Platform will then look for any synced observation candidates in camera two and find the observation that matches most closely in time to the image in camera one.

All that being said, resolution is a concept better shown than told:

If one is confident that two observation streams are in-sync, one may set the resolution to be fairly small. However, given the way some host cameras behave, it's generally not necessary or recommended.