| Title: | Performance metrics for probabilitic forecasts |
|---|---|
| Description: | This package implements various metrics for assessing the performance of probabilitic forecasts. |
| Authors: | Sangeeta Bhatia [aut, cre] |
| Maintainer: | Sangeeta Bhatia <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 1.0.0 |
| Built: | 2024-10-31 20:29:03 UTC |
| Source: | https://github.com/mrc-ide/assessr |
Median absolute deviation about the median
abs_madm(pred)abs_madm(pred)
pred |
T X N Matrix of predictions. Each column is a simulation. |
Median absolute deviation about the median is a measure of how clustered the forecasts are. A value of 0 indicates that all the predicted values are the same, thus highly clustered. Large values indicate more diffuse predictions.
vector of length T.
https://bit.ly/2vPO0I9
Residual averaged acorss simulations
avg_residual(obs, pred)avg_residual(obs, pred)
obs |
observed vector T X 1 |
pred |
matrix of predicted observations. Each column is a simulation. T X N where N is the number of simulations. |
error T X 1. Each entry is the error averaged across the simulations.
Sangeeta Bhatia
Bias in probabilistic forecasts
bias(obs, pred)bias(obs, pred)
obs |
observed vector T X 1 |
pred |
Simulated predictions T X N. Each column is a simulation. |
Bias is measured as
where heaviside returns 1 if the arg is positive, 0 if this negative and 0.5 if it is 0. The average is taken over all simulations.
vector of length T.
Sangeeta Bhatia
https://doi.org/10.1371/journal.pcbi.1006785
Mean absolute error
mae(obs, pred)mae(obs, pred)
obs |
T X 1 vector of observations. |
pred |
T X N matrix of predictions where each column is a simulation. |
T X 1 vector of mean absolute error
Sangeeta Bhatia
Proportion of observations in given credible interval
prop_in_ci(obs, min, max)prop_in_ci(obs, min, max)
obs |
vector of observed values |
min |
vector of the lower end of the interval. Either length 1 vector or the same length as the that of obs. |
max |
vector of the upper end of the interval. Either length 1 or the same length as that of the obs vector. |
Proportion of observed values that fall within a given interval
proportion of values in obs vector that are greater than or equal to min and less than or equal to max.
Sangeeta Bhatia
Relative sharpness: median absolute deviation about the median
rel_madm(pred)rel_madm(pred)
pred |
T X N Matrix of predictions. Each column is a simulation. |
vector of length T.
https://bit.ly/2vPO0I9
Relative mean absolute error
rel_mae(obs, pred)rel_mae(obs, pred)
obs |
T X 1 vector of observations. |
pred |
T X N matrix of predictions where each column is a simulation. |
Relative mean absolute error is defined as
T X 1 vector of mean absolute error normalised by the observed value.
Sangeeta Bhatia
Relative mean absolute deviation about the median
rel_mean_dvtn(pred)rel_mean_dvtn(pred)
pred |
T X N Matrix of predictions. Each column is a simulation. |
vector of length T.
https://bit.ly/2vPO0I9
Relative mean squared error averaged acorss simulations
rel_mse(obs, pred)rel_mse(obs, pred)
obs |
observed vector T X 1 |
pred |
matrix of predicted observations. Each column is a simulation. T X N where N is the number of simulations. |
Relative average mean square error is
We add 1 to the observed vector to avoid dividing by 0.
error T X 1. Each entry is the error averaged across the simulations
Sangeeta Bhatia