Skip to content

Posterior Variance

Precision, Not Accuracy

Uncertainty is necessarily a measure of precision, not accuracy. Prior and posterior variance tell us about the data that we observed and its relation to the model we chose for our calibration, but doesn't say anything about the accuracy of the model itself.

First, it is important to discuss posterior variance together in context with prior variance, or the "a-priori variance factor," as it is sometimes called. Prior variance in this context is a global scale on the uncertainty of our input data. This could be considered a relative measure of confidence in a given "geometric network" of data input into our calibration.

Tangram Vision always uses a prior variance of 1.0 in the adjustment — in other words, no calibration is considered special with regards to its input uncertainty. TVCal will just use default uncertainties for any given observed quantity and scale the whole "network" with 1.0.

This means that when we discuss posterior variance, we're comparing this variance to the prior, or 1.0. Posterior variance \((\hat{\sigma}_0^2)\) can be computed in any least-squares adjustment by using the following formula:

\[ \hat{\sigma}_0^2 = \frac{r^T \cdot C_l^{-1} \cdot r}{\mathsf{D.o.F}} \]

where \(r\) is the residuals vector, \(C_l\) is the covariance matrix of the observations in the calibration, and D.o.F. refers to the total degrees of freedom in the entire adjustment. The upper part of the above fraction is the cost function of a least-squares process (the weighted square sum of residuals), which is why this is sometimes referred to as "normalized cost."

Posterior vs Prior Variance

The trick here is in interpreting this value relative to our prior variance of 1.0. There are three possible scenarios that can occur:

  1. Posterior variance is approximately 1.0, to some level of significance.
  2. Posterior variance is less than the prior variance (\(\hat{\sigma}_0^2 < 1.0\))
  3. Posterior variance is greater than the prior variance (\(\hat{\sigma}_0^2 > 1.0\))

Posterior Variance ~ 1.0

The first scenario is the simplest, but also the least interesting. If the posterior variance matches the prior variance well, then our uncertainty has been correctly quantified, and that the final variances of our estimated parameters match expectations.

This is why this result is so uninteresting: while the data collected may have informed our parameters, it did not fundamentally minimize error any more than what was known at the outset.

Posterior < Prior

The residual error across the data set is now smaller than what was expected. This could mean the problem was pessimistic in its initial estimate of uncertainty in the network / data set / problem definition. Taking a more Bayesian approach, it can be interpreted as having more information or certainty in the results of the calibration using this data set than it had going in.

Posterior > Prior

The posterior variance is now larger than what was expected at the outset. This implies the opposite of Posterior < Prior: the problem was optimistic in its initial estimate of uncertainty. In other words, we now have more uncertainty (less certainty) in the results using this data set than we thought we ought to have, after considering the data.

What's Best?

From the latter two scenarios, it might be tempting to say that posterior variance should always be less than or equal to 1.0. After all, it's better to be pessimistic or realistic with regards to our uncertainty than it is to be optimistic and have more error, right?

Unfortunately, that's a very broad brush to paint posterior variance with, and may lead to some biased inferences. In particular, there's a good number of reasons why posterior variance might be smaller than prior variance:

  • We set our prior variances to be very large, and that was unrealistic.
  • The data set contained much more data than was necessary to estimate the parameters to the appropriate level of significance. This relates to the observability of our parameters as well as the number of parameters we are observing.

Conversely, there's a number of good reasons for why posterior variance may be larger than prior variance:

  • The prior variance was set to be very small, and that was unrealistic. This can occur if the data set is good, but observations from the data are qualitatively bad for some reason (e.g. a blurry lens that was installed incorrectly). The model and data would not agree, so residual error increases.
  • The data set did not contain enough degrees of freedom (D.o.F) to be able to minimize residuals to the level of desired significance. This can occur when individual frames in a camera do not detect enough points to account for the number of parameters we have to estimate for that pose / frame / intrinsics model / etc.
  • The data actually disagrees with prior estimates of our parameters. This can occur if parameters are "fixed" to incorrect values, and the data demonstrates this through larger residual error. This can also occur when there are large projective compensations in our model, and our data set does not contain frames or observations that would help discriminate correlations across parameters.

Holistic Interpretations

Of course, these interpretations cannot exist on their own, and are fairly "global" across the entire calibration. It's easy to misattribute any one of these causes to a problem in the calibration. For instance, if the model and correspondent covariances in the plex are acceptable, and the other calibration outputs don't show any signs that the calibration is invalid in some way, then posterior variance probably isn't telling you much relatively speaking.

When should I worry about posterior variance?

Generally speaking, posterior variance needs to differ by quite a large margin before it is worth worrying about, and you'll likely see other problems in the calibration process that will lead to more fruitful investigations if something is "wrong" or can be improved upon.

As a rule of thumb, if posterior variance isn't less than \(\frac{1}{3}\) or greater than 3.0 (a factor of 3 between posterior and prior variance), then you shouldn't worry about it.

Comparing Data Collection Strategies

One extremely useful way to utilize posterior variance is to compare data collection strategies (factory or otherwise) for your system. For example, if you have:

  1. The same input plex.
  2. Multiple different data sets, collected in different ways (e.g. different motion paths, different object space).

...then multiple calibrations might give different results not necessarily different enough to tell which is more appropriate to use. However, pairing these results with the posterior variance (relative to the same input plex) can provide a heuristic as to whether a data collection strategy generally provides more certainty in results (e.g. lower residuals) relative to another data collection strategy.

This naturally needs to be balanced against degrees of freedom in the data sets. Comparing a data set with infinitely more data (e.g. collect the same motion path for 20 min instead of 2 min) will likely result in a lower posterior variance, but this is expected. Comparing different object space configurations, different collection methodologies, etc. is much more interesting. When done at scale (comparing two data collection strategies across many input plexes), this can help inform ways to reduce the data collection period, or shrink data collection procedures while still retaining the same posterior variance on average in order to cut data collection costs during the calibration process.