To evaluate the performance of state estimators, the estimation error \( \tilde{\mathbf{x}}(k) = \mathbf{x}(k) - \mathbf{\hat{x}}(k) \) is evaluated. The root mean square error (RMSE), which is a widely used quality measure, is suitable for this purpose. The basis is the estimation error \(\tilde{\mathbf{x}}(k)\) for each time step \( k \in {1…K} \).

In simulation

For a simulation with a length of \( K\) time increments the RMSE is averaged by \( N\) Monte Carlo runs in order to achieve a high statistical significance \[ \text{RMSE}(\tilde{\mathbf{x}}(k)) = \sqrt{\frac{1}{N} \sum^N_{i=1} (\tilde{x}^i_1(k)^2 + … + \tilde{x}^i_n(k)^2)} \] where \( n = \text{dim}(\tilde{\mathbf{x}}(k)) \) is.

The RMSE should always be calculated only for elements with the same unit, because it is not correct to add e.g. velocity and acceleration. In this case, RMSE’s are determined per state or unit. If for example a velocity is chosen as a state vector, it is called the RMSE of the velocity. In general, the smaller the RMSE, the better the state estimator.

In real-world applications

If real measurement data are to be evaluated, the RMSE can only be calculated from a single run. The RMSE thus has no high statistical significance. The RMSE can therefore alternatively be calculated via the mean estimation error

\[ \mu_{\tilde{x}} = \frac{1}{K} \sum^K_{k=1} \tilde{\mathbf{x}}(k) \]

and the standard deviation of the estimation error

\[ \sigma_{\tilde{x}} = \sqrt{ \frac{1}{K} \sum^K_{k=1} (\tilde{\mathbf{x}}(k) - \mu_{\tilde{x}} )^2 } \]

with

\[ \text{RMSE}(\tilde{\mathbf{x}}(k)) = \sqrt{ \mu_{\tilde{x}} + \sigma_{\tilde{x}} } . \]

However, since the true state \( \mathbf{x} \), i.e. a “ground truth”, must be available, this can only be done in practice with highly accurate reference data. It may also be the case that not all states can be measured by reference measuring devices. It is therefore legitimate to calculate the RMSE only for the states for which reference data are available.