Why be a Bayesian

A non-technical introduction to the Bayesian approach

Daniel Skak Mazhari-Jensen

2026-08-04

What is this Bayesian thing anyway?

Chat with your neighbor for 3 minutes:

  • Have you heard about Bayesian models/inference/approach?
  • If yes, in which context?
  • What do you think it means (no guarantee it’s the “right” answer, but give it a try!)

How many have heard about Bayesian inference? 🙋‍♀️🙋

How many already use a Bayesian workflow? 🙋‍♀️🙋

How many have read a paper using Bayesian stats? 🙋‍♀️🙋

How many (of you) understood that paper using Bayesian stats? 🙋‍♀️🙋

Are you ready to get a non-technical introduction? 🎉🥳

Bayes in the style of xkcd

But why is this different from other stats?

Is there a fight?

Is there a fight?

The Core Question

Why do we need a Bayesian approach to regression?

Can’t we just use ordinary least squares?

The answer is not (only):

  • philosophy
  • elegance
  • a matter of preference

Real regression problems are often weakly identified, noisy, unstable, and geometrically pathological.

The ultra-quick answer

Bayesian regression stabilizes inference by:

  1. Explicitly stating expectations
  2. Constraining parameters, offering regularization
  3. Constraining model, offering stability in parametrization of the model

When to use Frequentist and when to use Bayesian approaches?

Frequentist are often sufficient and comparable to Bayesian stats!

The data

Code 
set.seed(18062026)

# Data
n <- 100                 # samples
x <- rnorm(n)            # x = N(0,1)      
y <- 2 + 3*x + rnorm(n)  # α = 2, β = 3, ε = N(0,1)

The models

Frequentist:

Code 
# Frequentist OLS
fit_freq <- lm(y ~ x)

Bayesian:

Code 
# Bayesian regression with flat priors
library(rstanarm)
fit_bayes <- stan_glm(
  y ~ x,
  prior = NULL,            # flat prior on coefficients
  prior_intercept = NULL,
  chains = 2, iter = 2000, refresh = 0
)

What did we find?

Frequentist:

Code 
parameters::parameters(fit_freq, ci = 0.95) |> 
  dplyr::select(Parameter, Coefficient, CI, CI_low, CI_high)
# Fixed Effects

Parameter   | Coefficient |       95% CI
----------------------------------------
(Intercept) |        2.09 | [1.91, 2.28]
x           |        3.01 | [2.82, 3.20]

Bayesian:

Code 
parameters::parameters(fit_bayes, ci = 0.95) |> 
  dplyr::select(Parameter, Median, CI, CI_low, CI_high)
# Fixed Effects

Parameter   | Median |       95% CI
-----------------------------------
(Intercept) |   2.09 | [1.91, 2.27]
x           |   3.00 | [2.82, 3.20]

THEN WHY DO WE BOTHER!?

When to use Frequentist and when to use Bayesian approaches?

A More Realistic example

Suppose we have:

  • 30 observations
  • 30 predictors
  • strong predictor correlation
  • weak signal
  • uncertain mechanisms

This is extremely common in:

  • social science
  • biology
  • psychology
  • economics
  • policy research

Simulating Correlated Predictors

Figure 1: Correlation structure among predictors.

The Data Generating Process

The truth is simple.

Only one predictor matters.

\[ y = 2x_1 + \epsilon \\ \epsilon = Normal(0,2) \]

All other predictors have zero effect.

Yet predictors are highly correlated.

Talk to your neighbor

what do you think will happen?

  • think about the model specification
  • does the same happen in the Frequentist and Bayesian models?
  • 30 observations (humans?)
  • 30 predictors (genes?)
  • strong predictor correlation (ρ ~ .8)
  • weak signal (a.u. 2)
  • noise at \(\mathcal{N}(0,2)\)
  • data generating process:

\[ y = 2x_1 + \epsilon \\ \epsilon = Normal(0,2) \]

The model results when p=30 and n=30

x30 doesn’t even have an estimate due to unindentifiability!

Let’s try again - with more data!

what do you think will happen?

  • 30 observations (humans?)
  • 30 predictors (genes?)
  • strong predictor correlation (ρ ~ .8)
  • weak signal (a.u. 2)
  • noise at \(\mathcal{N}(0,2)\)
  • data generating process:

\[ y = 2x_1 + \epsilon \\ \epsilon = Normal(0,2) \]

The model results when p=30 and n=32

collinearity and inflates CIs and unstable coefficients!

The Frequentist Ideal

Ordinary Least Squares (OLS):

  • minimizes squared residuals
  • produces unbiased estimators
  • provides confidence intervals and p-values
  • works beautifully asymptotically

The classical linear model:

\[ y = X\beta + \epsilon \]

with

\[ \epsilon \sim \mathcal{N}(0, \sigma^2) \]

OLS estimator:

\[ \hat{\beta}_{OLS} = (X^TX)^{-1}X^Ty \]

The Hidden Assumption

OLS works well when:

  • predictors are not highly collinear
  • sample size is large
  • signal-to-noise ratio is reasonable
  • parameters are strongly identified

But real data often violate all of these simultaneously.

The Mechanism of Failure

With correlated predictors:

  • many coefficient combinations fit equally well
  • the likelihood becomes nearly flat
  • tiny noise changes estimates dramatically

OLS asks:

Which coefficients minimize prediction error?

But it never asks:

Are these coefficient magnitudes plausible?

The Geometry Problem

Collinearity creates unstable directions in parameter space.

The matrix:

\[ (X^TX)^{-1} \]

becomes nearly singular.

As a result:

  • coefficients explode
  • signs flip
  • uncertainty becomes enormous
  • interpretation collapses

The Bayesian Solution

Bayesian regression adds prior information:

\[ \beta_j \sim \mathcal{N}(0,1) \]

Posterior:

\[ p(\beta \mid y) \propto p(y \mid \beta)p(\beta) \]

The prior regularizes weakly identified directions.

This changes the geometry of inference.

Important Clarification

Bayesian priors are not magic.

They encode structural skepticism.

The prior says:

Large coefficients should require strong evidence.

This is often scientifically reasonable.

Explaining the simulation Setup

We simulated the failure directly:

  • small sample
  • many predictors
  • high collinearity
  • weak signal

Frequentist OLS

Why This Happens

Predictors are interchangeable.

The model can fit equally well using:

  • huge positive x1
  • huge negative x2

OLS has no mechanism preventing absurd parameter values.

It only optimizes fit.

The Frequentist Interpretation Problem

Researchers now face impossible interpretation.

Questions become unstable:

  • Which variables matter?
  • Which signs are trustworthy?
  • Which effects are real?

p-values fluctuate wildly across samples.

Bayesian Regression

How did it do that?

Let’s have a look

Scientific sceptisism saves us from statistical model stupidity

What the Prior Does

The prior shrinks implausible coefficients toward zero.

Not aggressively.

Just enough to stabilize weakly identified directions.

This is called:

  • regularization
  • shrinkage
  • partial pooling of uncertainty

Bias-Variance Tradeoff

Frequentist OLS emphasizes unbiasedness.

But unbiased estimators can have enormous variance.

Bayesian regression accepts:

  • tiny bias
  • dramatically lower variance

This often improves:

  • prediction
  • calibration
  • scientific interpretation

In a nutshell…

OLS treats the model as fixed truth.

Bayesian workflow treats models as uncertain approximations.

This distinction is profound.

Posterior Interpretation

Frequentist confidence interval:

Does NOT mean: “There is a 95% probability the parameter lies here.”

Bayesian posterior interval:

Literally means: “Given model and data, there is 95% posterior probability the parameter lies here.”

Weak Identification

The Bayesian posterior honestly expresses:

  • uncertainty
  • partial identifiability
  • lack of information

Sometimes the correct scientific answer is:

“We do not know very much.”

OLS often hides this instability behind noisy point estimates.

Posterior Predictive Thinking

Bayesian models naturally support:

  • posterior predictive checks
  • hierarchical models
  • measurement error
  • causal modeling
  • uncertainty propagation

This enables model criticism rather than blind estimation.

A More Extreme Failure

Increase:

  • predictors from 10 to 20
  • correlation from 0.9 to 0.95

OLS becomes nearly meaningless.

Coefficients:

  • explode
  • reverse sign
  • become sample-dependent noise

Bayesian regularization still produces stable inference.

Final Takeaway

Bayesian regression matters because:

  • real models are weakly identified
  • finite samples are noisy
  • predictors are correlated
  • unconstrained likelihoods become pathological

Priors repair inferential geometry.

Final Thought

The Bayesian question is not:

“What coefficient minimizes error?”

The Bayesian question is:

“What parameter values remain plausible after combining data with scientific structure?”

That is usually the more meaningful scientific question.

References

  • Gelman, A., et al. Bayesian Data Analysis
  • McElreath, R. Statistical Rethinking
  • Vehtari, A., Gelman, A., Gabry, J. (2017) “Practical Bayesian model evaluation using leave-one-out cross-validation”
  • Hastie, Tibshirani, Friedman. Elements of Statistical Learning

Exercises:

See exercises at https://danmazjen.github.io/blog/why_we_bayesian.html