# Plex as JSON

```
{
"uuid": "5d71519a-31c0-4b71-b629-1195470330f1",
"creation_timestamp": 1634855338595628000,
"components": [
// array of components
],
"spatial_constraints": [
// array of spatial constraints
],
"temporal_constraints": [
// array of temporal constraints
]
}
```

Field | Type | Description |
---|---|---|

`"uuid"` |
String | A universally unique identifier for the Plex. |

`"creation_timestamp"` |
integer | The creation timestamp of the Plex, from unix epoch, in nanoseconds. |

`"components"` |
Component Object | An array of JSON objects describing components within the Plex. |

`"spatial_constraints"` |
Spatial Constraint Object | An array of JSON objects describing spatial constraints within the Plex. |

`"temporal_constraints"` |
Temporal Constraint Object | An array of JSON objects describing temporal constraints within the Plex. |

Error

It is an error if the components array is empty and either of the constraints arrays are non-empty.

## Components

```
{
"camera": {
// Values specific to this component kind
}
}
```

Every component type has a UUID, Root UUID, and Name.

Each of the supported component kinds below are described in more detail.

### Cameras

Camera Component

```
{
"camera": {
"uuid": "5680072f-3e03-4f70-85d1-7fda3630f59b",
"root_uuid": "071380b8-cf70-446c-afdc-d5ec13aad305",
"name": "Infrared camera #3 - Front side",
"intrinsics": {
"projection": {
"pinhole": {
"f": 382.8955078125,
"cx": 324.38397216796875,
"cy": 243.57395935058594
}
},
"distortion": {
"brown_conrady": {
"k1": 0,
"k2": 0,
"k3": 0,
"p1": 0,
"p2": 0
}
},
"affinity": {
"a1": 0
},
"width": 640,
"height": 480
},
"covariance": [
[
1000, 0, 0, 0, 0, 0, 0, 0, 0,
0, 1000, 0, 0, 0, 0, 0, 0, 0,
0, 0, 1000, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1000, 0, 0, 0, 0, 0,
0, 0, 0, 0, 1000, 0, 0, 0, 0,
0, 0, 0, 0, 0, 1000, 0, 0, 0,
0, 0, 0, 0, 0, 0, 1000, 0, 0,
0, 0, 0, 0, 0, 0, 0, 1000, 0,
0, 0, 0, 0, 0, 0, 0, 0, 1000
],
9,
9
],
"pixel_pitch": 1
}
}
```

Field | Type | Description |
---|---|---|

`"uuid"` |
String | A universally unique identifier for the Plex. |

`"root_uuid"` |
String | A universally unique identifier for the "root" component relative to this component. |

`"name"` |
String | A colloquial or friendly name for the component. This need not be unique. |

`"intrinsics"` |
Intrinsics Object | The intrinsic parameters corresponding to this camera component's model. |

`"covariance"` |
Covariance Object | A covariance matrix for the camera's intrinsics. Should be size M×M for M-number of intrinsics parameters. |

`"pixel_pitch"` |
number | The "pitch" or length of a single pixel in [units / pixel]. Can be 1.0 if unknown. |

#### Notes

- Pixel pitch: Pixel pitch is a measure of how large a pixel is in some units of [units / pixel]. For example, if you know that pixels have a 7mm pitch, then pixel pitch could be described as either 7mm-per-pixel or 0.007m-per-pixel.

Note

Pixel pitch is useful when comparing the reprojection error across different cameras. Because the size of a pixel may be different between two cameras, it is not an apples-to-apples comparison to compare a 1 pixel reprojection error in a camera that has 7mm pixels to a 1 pixel reprojection error in a camera with 40mm pixels. 1 pixel in the latter case is 5.7× as large!

When interpreting results from a calibration, it is important to remember the scale, since "a pixel" isn't well defined in this sense, and may mean different things for different sensors.

#### Intrinsics

Camera intrinsics are always represented with the following fields:

Camera Intrinsics

```
"intrinsics": {
"projection": {
"pinhole": {
"f": 382.896,
"cx": 324.384,
"cy": 243.574
}
},
"distortion": {
"brown_conrady": {
"k1": 0,
"k2": 0,
"k3": 0,
"p1": 0,
"p2": 0
}
},
"affinity": {
"a1": 0
},
"width": 640,
"height": 480
}
```

Field | Type | Description |
---|---|---|

`"projection"` |
Object | A JSON object describing the projection model of the camera. |

`"distortion"` |
Object | A JSON object describing the distortion model of the camera. |

`"affinity"` |
Object | A JSON object describing the affinity model of the camera. |

`"width"` |
integer | The width of the image format that the intrinsics parameters were specified for. |

`"height"` |
integer | The height of the image format that the intrinsics parameters were specified for. |

Warning

At present the Tangram Vision Platform's calibration pipeline only supports calibrating for the same image format (width × height) that the component streams at. For example, if the width and height of intrinsics are specified as 640×480, then we are unable to use this Plex to calibrate a dataset containing images from that component at a resolution e.g. 1920×1080.

Instead, we would require that a new Plex be generated with intrinsics values for the correct image format (in the example above, 1920×1080).

#### Projection

Projection is the first piece of our camera model. At present, Tangram Vision only supports a pinhole projection model. This looks like:

Camera Projection Model

Field | Type | Description |
---|---|---|

`"f"` |
number | The focal length of the camera's lens. |

`"cx"` |
number | The x-component of the principal point offset. |

`"cy"` |
number | The y-component of the principal point offset. |

```
"projection": {
"pinhole": {
"f": 382.896,
"cx": 324.384,
"cy": 243.574
}
}
```

#### Distortion

Distortions are the second component of our camera intrinsics. For these, we support one of two models: Brown-Conrady or Kannala-Brandt. This looks like:

Camera Distortion Model

Field | Type | Description |
---|---|---|

`"k1"` |
number | First order radial distortion parameter. |

`"k2"` |
number | Second order radial distortion parameter. |

`"k3"` |
number | Third order radial distortion parameter. |

`"p1"` |
number | Tangential / decentering distortion parameter. |

`"p2"` |
number | Tangential / decentering distortion parameter. |

```
"distortion": {
"brown_conrady": {
"k1": 0,
"k2": 0,
"k3": 0,
"p1": 0,
"p2": 0,
}
}
```

Field | Type | Description |
---|---|---|

`"k1"` |
number | First order radial distortion parameter. |

`"k2"` |
number | Second order radial distortion parameter. |

`"k3"` |
number | Third order radial distortion parameter. |

`"k4"` |
number | Fourth order radial distortion parameter. |

```
"distortion": {
"kannala_brandt": {
"k1": 0,
"k2": 0,
"k3": 0,
"k4": 0,
}
}
```

```
// No Distortion field in the Intrinsics Object
```

Warning

Generally speaking you will not want to specify your images as having "no distortion" unless your data set is already pre-rectified.

#### Affinity

Our final component of camera intrisnics is affinity. Affinity can look like one of three possible pairings: Scale, shear, or scale & shear.

Camera Affinity Model

Field | Type | Description |
---|---|---|

`"a1"` |
number | Ratio of the scale offset between the y- and x-axes of the imaging surface. |

`"a2"` |
number | Ratio of the shear effect (non-orthogonality) between the y- and x-axes of the imaging surface. |

```
"affinity": {
"a1": 0, // Scale. Optional.
"a2": 0, // Shear. Optional.
}
```

```
// No Affinity field in the Intrinsics Object
```

#### Covariance

Covariance can be fairly straightforward as far as the JSON representation goes, but there are a few mistakes that can be easy to pass over. The main things to remember when constructing the intrinsics covariance matrix are:

- The covariances matrix is always square and symmetric.
- The covariance matrix must have the same number of columns and rows as there are parameters in your intrinsics.

The most important of these is the second point above, in that the matrix must have an equal number of columns and rows as the number of intrinsics parameters. We provide some examples below:

Camera Intrinsics Covariance

Covariance starts as an array of JSON objects. In particular, the array will always have three elements: a sub-array containing the elements of the covariance matrix, the number of rows in the matrix, and the number of columns in the matrix. For a 3×3 covariance matrix, this might look like:

```
"covariance": [
// The entries of the covariance matrix.
//
// NOTE: We do not have to worry about row / column major ordering
// since the matrix is symmetric, so you can enter it in either order.
[
1000, 0, 0,
0, 1000, 0,
0, 0, 1000
],
// The number of rows
3,
// The number of columns
3
]
```

Notice that above we've written out the first element of this array (the covariance entries) arranged as a matrix. This is not necessary and is done for illustrative purposes. We could instead write:

```
"covariance": [
[1000, 0, 0, 0, 1000, 0, 0, 0, 1000],
3,
3,
]
```

Which is equally valid JSON.

If we do not have any distortion or affinity, our only intrinsics
parameters would be the projection model. If we're using `"pinhole"`

projection, this gives us 3 parameters:

`f`

: the focal length.`cx`

: the x-component of the principal distance.`cy`

: the y-component of the principal distance.

This would give us a 3×3 covariance matrix:

```
"covariance": [
[
// f, cx, cy
1000, 0, 0,
0, 1000, 0,
0, 0, 1000
],
3,
3
]
```

With projection and distortion, we could have a varying number of
parameters, depending on the distortion model we choose. For projection
we have the same number of parameters as in the projection-only case,
plus the parameters from our distortion model. For Brown-Conrady
distortion, this would be `k1`

, `k2`

, `k3`

, `p1`

, and `p2`

, so 5 extra
parameters, for a total of 8.

This would then give us a covariance entry such as:

```
"covariance": [
[
// f, cx, cy, k1, k2, k3, p1, p2
1000, 0, 0, 0, 0, 0, 0, 0,
0, 1000, 0, 0, 0, 0, 0, 0,
0, 0, 1000, 0, 0, 0, 0, 0,
0, 0, 0, 1000, 0, 0, 0, 0,
0, 0, 0, 0, 1000, 0, 0, 0,
0, 0, 0, 0, 0, 1000, 0, 0,
0, 0, 0, 0, 0, 0, 1000, 0,
0, 0, 0, 0, 0, 0, 0, 1000
],
8,
8
]
```

Or for Kannala-Brandt (`k1`

, `k2`

, `k3`

, `k4`

):

```
"covariance": [
[
// f, cx, cy, k1, k2, k3, k4,
1000, 0, 0, 0, 0, 0, 0,
0, 1000, 0, 0, 0, 0, 0,
0, 0, 1000, 0, 0, 0, 0,
0, 0, 0, 1000, 0, 0, 0,
0, 0, 0, 0, 1000, 0, 0,
0, 0, 0, 0, 0, 1000, 0,
0, 0, 0, 0, 0, 0, 1000
],
7,
7
]
```

Notice that the conventional order of projection → distortion is always the same.

As in the previous two examples, we now have a varying number of parameters due to affinity, in addition to projection and distortion. For these examples, we'll demonstrate the ordering of the covariance matrix depending on various configurations.

First, let's start with Pinhole projection (3 parameters; `f`

, `cx`

, &
`cy`

), Kannala-Brandt distortion (4 parameters; `k1`

, `k2`

, `k3`

, &
`k4`

), and Scale affinity (1 parameter; `a1`

) for a total of 8
parameters:

```
"covariance": [
[
// f, cx, cy, k1, k2, k3, k4, a1,
1000, 0, 0, 0, 0, 0, 0, 0,
0, 1000, 0, 0, 0, 0, 0, 0,
0, 0, 1000, 0, 0, 0, 0, 0,
0, 0, 0, 1000, 0, 0, 0, 0,
0, 0, 0, 0, 1000, 0, 0, 0,
0, 0, 0, 0, 0, 1000, 0, 0,
0, 0, 0, 0, 0, 0, 1000, 0,
0, 0, 0, 0, 0, 0, 0, 1000
],
8,
8
]
```

Our convention for ordering continues to go projection → distortion → affinity.

This holds even if we were to have no distortion. If distortion in our
example above is instead set to `"no_distortion"`

, then we would get:

```
"covariance": [
[
// f, cx, cy, a1,
1000, 0, 0, 0,
0, 1000, 0, 0,
0, 0, 1000, 0,
0, 0, 0, 1000
],
4,
4
]
```

## Spatial Constraints

Spatial constraints describe the spatial relationship between two distinct components. The JSON format for spatial constraints takes the form:

Spatial Constraints

Field | Type | Description |
---|---|---|

`"extrinsics"` |
Object | A JSON object describing the `to` ← `from` extrinsic. Origin: "to" |

`"covariance"` |
Object | A matrix describing the variance-covariance of the extrinsics. |

`"from"` |
String | The universally unique identifer of the component the extrinsics are "from." |

`"to"` |
String | The universally unique identifer of the component the extrinsics are "to." |

```
{
"extrinsics": {
"rotation": [0, 0, 0, 1],
"translation": [-0.0499, 0.032, 0]
},
"covariance": {
"raw_se3": [
10, 0, 0, 0, 0, 0,
0, 10, 0, 0, 0, 0,
0, 0, 10, 0, 0, 0,
0, 0, 0, 10, 0, 0,
0, 0, 0, 0, 10, 0,
0, 0, 0, 0, 0, 10
]
},
"from": "03da2e68-207a-44d7-a7e8-fa9b501e0ca6",
"to": "4a0f04a2-839b-409e-9b3e-d4c3074540e3"
}
```

## Temporal Constraints

Temporal constraints describe the temporal relationship between two distinct components. The JSON format for temporal constraints takes the form:

Temporal Constraints

Field | Type | Description |
---|---|---|

resolution | Object | A time span delta around which to search for matching observations across component streams. |

synchronization | Object | An object describing how to synchronize time from component "from" to component "to". |

from | String | The universally unique identifer of the component the synchronization is adjusting. |

to | String | The universally unique identifer of the component the synchronization is matching. |

Synchronization describes the following relationship between two clocks:

Field | Type |
Description |
---|---|---|

offset | integer | The epoch offset between two clocks, in units of integer nanoseconds. |

skew | integer | The scale offset between two clocks. Unitless. |

```
{
"from": "5c3bc902-1524-4d25-a91b-cdd9b494f568",
"resolution": 16666666,
"synchronization": {
"offset": 0,
"skew": 0
},
"to": "96566821-efef-41b9-a817-f38c2a802f4b"
}
```