Comparison using WAIC

 P(Model 1 better than Model 2)=Φ(elpd_diff​/se_diff)

prob_model_better <- function(elpd_diff, se_diff) {

  z <- elpd_diff / se_diff

  p <- pnorm(z)

  return(p)

}

elpd_diff <- 5
se_diff   <- 3

prob_model_better(elpd_diff, se_diff)

Result:

[1] 0.952

So there is approximately 95% probability that model 1 predicts better than model 2.

It is not the probability that the model is better — only that its predictive accuracy is higher.

You can use also the function Probability_Best_Model_WAIC() in HelpersMG packages

How to cite this method:

Model predictive performance was evaluated using leave-one-out cross-validation (PSIS-LOO). Uncertainty in model comparison was assessed using resampling of the pointwise log predictive densities following the approach described by Vehtari, Gelman and Gabry (2017) and Yao et al. (2018).

Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413-1432. https://doi.org/10.1007/s11222-016-9696-4 

Yao, Y., Vehtari, A., Simpson, D., & Gelman, A. (2018). Using stacking to average Bayesian predictive distributions (with discussion). Bayesian Analysis, 13(3), 917-1007. https://doi.org/10.1214/17-BA1091918 



Commentaires

Enregistrer un commentaire

Posts les plus consultés de ce blog

Standard error from Hessian Matrix... what can be done when problem occurs

An Hessian matrix is a square matrix of partial second order derivative. From the square-root of the inverse of the diagonal, it is possible to estimate the standard error of parameters. Let take an example: val=rnorm(100, mean=20, sd=5) # Return -ln L of values in val in Gaussian distribution with mean and sd in par fitnorm<-function(par, val) {   -sum(dnorm(val, par["mean"], par["sd"], log = TRUE)) } # Initial values for search p<-c(mean=20, sd=5) # fit the model result <- optim(par=p, fn=fitnorm, val=val, method="BFGS", hessian=TRUE) # Inverse the hessian matrix to get SE for each parameters mathessian <- result$hessian inversemathessian <- solve(mathessian) res <- sqrt(diag(inversemathessian)) # results data.frame(Mean=result$par, SE=res)   library(MASS) (r<-fitdistr(val, "normal")) It works well. However, the inverse of the matrix cannot be calculated if the second order derivative for some paramete...
Lire la suite

stepAIC from package MASS with AICc

Due to a bug in dropterm() and addterm() in package MASS, it is impossible to use AICc with stepAIC() . Here is a solution. Enter these commands and now you can use: stepAIC(g0, criteria="AICc") or stepAIC(g0, criteria="BIC") For example: datax <- data.frame(y=rnorm(100), x1=rnorm(100),                     x2=rnorm(100), x3=rnorm(100), x4=rnorm(100)) g0 <- glm(y ~ x1+x2+x3+x4, data=datax) gx_AICc <- stepAIC(g0, criteria="AICc") gx_AIC <- stepAIC(g0, criteria="AIC") gx_AIC <- stepAIC(g0, criteria="BIC")
Lire la suite

Multivariable analysis and correlation of iconography

Image
Introduction Correlation iconography is a not very well known method to study multivariate data. It was developed long-time ago by Michel Lesty: Lesty M (1999) Une nouvelle approche dans le choix des régresseurs de la régression multiple en présence d'intéractions et de colinearités. La revue de Modulad 22:41-77 It is also well described in a French Wikipedia page: https://fr.wikipedia.org/wiki/Iconographie_des_corrélations After checking the possibilities of this method, I think that it deserves more attention. Let take the example of the wikipedia page: dta <- read.table(text=gsub(",", ".", "Élève Poids Âge Assiduité Note e1 52 12 12 5 e2 59 12,5 9 5 e3 55 13 15 9 e4 58 14,5 5 5 e5 66 15,5 11 13,5 e6 62 16 15 18 e7 63 17 12 18 e8 69 18 9 18"), header=TRUE) > dta   Élève Poids  Âge Assiduité Note 1    e1    52 12.0        12  5.0 2    e...
Lire la suite