Why be a Bayesian

A non-technical introduction to the Bayesian approach

Daniel Skak Mazhari-Jensen

2026-10-08

What is this Bayesian thing anyway?

Chat with your neighbor for 3 minutes:

  • Have you heard about Bayesian models/inference/approach?
  • If yes, in which context?
  • What do you think it means (no guarantee it’s the “right” answer, but give it a try!)

How many have heard about Bayesian inference? 🙋‍♀️🙋

How many already use a Bayesian workflow? 🙋‍♀️🙋

How many have read a paper using Bayesian stats? 🙋‍♀️🙋

How many (of you) understood that paper using Bayesian stats? 🙋‍♀️🙋

Are you ready to get a non-technical introduction? 🎉🥳

Bayes in the style of xkcd

But why is this different from other stats?

But why is this different from other stats?

Bayesian prior

Bayesian prior 🐉

How priors are made (up)

The fear of the prior

frequentists are often suspicious of priors because a prior can look like smuggling assumptions into the analysis before seeing the data.

Their instinct is:

Let the data speak for itself

or

The prior biases the model a priori!

Is there a fight?

Is there a fight?

The Core Question

Why do we need a Bayesian approach to regression?

Can’t we just use ordinary least squares?

The answer is not (only):

  • philosophy
  • elegance
  • a matter of preference

Real regression problems are often weakly identified, noisy, unstable, and geometrically pathological.

The ultra-quick answer

Spoiler alert!

Bayesian regression stabilizes inference by:

  1. Explicitly stating expectations
  2. Constraining parameters, offering regularization
  3. Not using quardratic approximation of likelihood (for SD etc.)

When to use Frequentist and when to use Bayesian approaches?

Frequentist stats are often sufficient and comparable to Bayesian stats!

The data

Code 
set.seed(18062026)

# Data
n <- 100                 # samples
x <- rnorm(n)            # x = N(0,1)      
y <- 2 + 3*x + rnorm(n)  # α = 2, β = 3, ε = N(0,1)

The models

Frequentist:

Code 
# Frequentist OLS
fit_freq <- lm(y ~ x)

Bayesian:

Code 
# Bayesian regression with flat priors
library(rstanarm)
fit_bayes <- stan_glm(
  y ~ x,
  prior = NULL,            # flat prior on coefficients
  prior_intercept = NULL,
  chains = 2, iter = 2000, refresh = 0
)

What did we find?

Frequentist:

Code 
parameters::parameters(fit_freq, ci = 0.95) |> 
  dplyr::select(Parameter, Coefficient, CI, CI_low, CI_high)
# Fixed Effects

Parameter   | Coefficient |       95% CI
----------------------------------------
(Intercept) |        2.09 | [1.91, 2.28]
x           |        3.01 | [2.82, 3.20]

Bayesian:

Code 
parameters::parameters(fit_bayes, ci = 0.95) |> 
  dplyr::select(Parameter, Median, CI, CI_low, CI_high)
# Fixed Effects

Parameter   | Median |       95% CI
-----------------------------------
(Intercept) |   2.09 | [1.91, 2.27]
x           |   3.00 | [2.82, 3.20]

THEN WHY DO WE BOTHER!? 👀

When to use Frequentist and when to use Bayesian approaches?

Is he aiming for the cake or the table? 👀

A More Realistic example

Suppose we have:

  • 30 observations
  • 30 predictors
  • strong predictor correlation
  • weak signal
  • uncertain mechanisms

This is extremely common in:

  • social science
  • biology
  • psychology
  • economics
  • policy research

Simulating Correlated Predictors

Figure 1: Correlation structure among predictors.

The Data Generating Process

The truth is simple.

Only one predictor matters.

\[ y = 2x_1 + \epsilon \\ \epsilon = Normal(0,2) \]

All other predictors have zero effect.

Yet predictors are highly correlated.

Talk to your neighbor

what do you think will happen?

  • think about the model specification
  • does the same happen in the Frequentist and Bayesian models?
  • 30 observations (humans?)
  • 30 predictors (genes?)
  • strong predictor correlation (ρ ~ .8)
  • weak signal (a.u. 2)
  • noise at \(N(0,2)\)
  • data generating process:

\[ y = 2x_1 + \epsilon \\ \epsilon = Normal(0,2) \]

The model results when p=30 and n=30

What do you notice? 👀

x30 doesn’t even have an estimate due to unindentifiability!

Let’s try again - with more data!

what do you think will happen?

  • 32 observations (humans?)
  • 30 predictors (genes?)
  • strong predictor correlation (ρ ~ .8)
  • weak signal (a.u. 2)
  • noise at \(N(0,2)\)
  • data generating process:

\[ y = 2x_1 + \epsilon \\ \epsilon = Normal(0,2) \]

The model results when p=30 and n=32

What do you notice? 👀

collinearity and inflates CIs and unstable coefficients!

The Frequentist Ideal

Ordinary Least Squares (OLS):

  • minimizes squared residuals
  • produces unbiased estimators
  • provides confidence intervals and p-values
  • works beautifully asymptotically

The classical linear model:

\[ y = X\beta + \epsilon \]

with

\[ \epsilon \sim N(0, \sigma^2) \]

OLS estimator:

\[ \hat{\beta}_{OLS} = (X^TX)^{-1}X^Ty \]

The Mechanism of Failure

With correlated predictors:

  • many coefficient combinations fit equally well
  • the likelihood becomes nearly flat
  • estimates dramatically change due to tiny noise

OLS asks:

Which coefficients minimize prediction error?

But it never asks:

Are these coefficient magnitudes plausible?

The Geometry Problem

Collinearity creates unstable directions in parameter space.

The matrix:

\[ (X^TX)^{-1} \]

becomes nearly singular.

As a result:

  • coefficients explode
  • signs flip
  • uncertainty becomes enormous
  • interpretation collapses

The Bayesian Solution

Bayesian regression adds prior information:

\[ \beta_j \sim N(0,1.5) \\ \alpha \sim N(0,0.25) \\ \]

Posterior:

\[ p(\alpha, \beta \mid y,x) \propto p(y \mid x,\alpha, \beta)p(\alpha, \beta) \]

The prior regularizes weakly identified directions.

This changes the geometry of inference.

Same thing here!

Important Clarification

Bayesian priors are not magic.

They encode structural skepticism.

The prior says:

Large coefficients should require strong evidence.

This is often scientifically reasonable.

Frequentist OLS

Why This Happens

Predictors are interchangeable.

The model can fit equally well using:

  • huge positive x1
  • huge negative x2

OLS has no mechanism preventing absurd parameter values.

It only optimizes fit.

The Frequentist Interpretation Problem

Researchers now face impossible interpretation.

Questions become unstable:

  • Which variables matter?
  • Which signs are trustworthy?
  • Which effects are real?

p-values fluctuate wildly across samples.

Bayesian Regression

How did it do that?

Let’s have a look

Scientific sceptisism saves us from statistical model stupidity

Bias-Variance Tradeoff

Frequentist OLS emphasizes unbiasedness.

But unbiased estimators can have enormous variance.

Bayesian regression accepts:

  • tiny bias
  • dramatically lower variance

This often improves:

  • prediction
  • calibration
  • scientific interpretation

In a nutshell…

OLS treats the model as fixed truth.

Bayesian workflow treats models as uncertain approximations.

This distinction is profound.

Even More Extreme Failures…

Add complexity of:

  • GLMs (often not multinomial Gaussian)
  • Over-dispersion
  • Outliers

OLS becomes nearly meaningless.

Coefficients:

  • explode
  • reverse sign
  • inference just become sample-dependent noise

Meanwhile, Bayesian regularization still produces stable inference.

Final Thoughts

Don’t fear the prior!

A prior is not some dangerous and mysterious trick that biases your model.

The Bayesian prior enforces that parameter values remain plausible after combining data with scientific structure

A prior can save you from foolish inference

Cheers!

References

  • Gelman, A., et al. Bayesian Data Analysis
  • McElreath, R. Statistical Rethinking
  • Vehtari, A., Gelman, A., Gabry, J. (2017) “Practical Bayesian model evaluation using leave-one-out cross-validation”
  • Hastie, Tibshirani, Friedman. Elements of Statistical Learning

Exercises:

See exercises at https://danmazjen.github.io/blog/why_we_bayesian.html