Probability and Statistics

Canonical T1 reference spanning probability theory and statistical inference. Companion deep notes:

• probability-fundamentals — axioms, conditioning, expectation, CLT/LLN, inequalities.
• probability-distributions — distribution zoo with parameter / moment / use-case tables.
• hypothesis-testing-mle — frequentist testing and maximum likelihood theory.
• bayesian-inference — priors, posteriors, conjugacy.
• mcmc-sampling — Metropolis-Hastings, Gibbs, HMC, NUTS.
• variational-inference — VI as deterministic alternative to MCMC.
• gaussian-processes, markov-chains-and-hmm, stochastic-calculus, causal-inference.

1. At a glance

Probability is the mathematics of uncertainty; statistics is the inverse problem — given data sampled from some unknown distribution, infer that distribution (or features of it). The two are coupled: every statistical method is a probabilistic model plus an inference rule.

• Probability theory provides the calculus of random variables, distributions, expectations, limit theorems, and stochastic processes.
• Statistics uses that calculus to estimate parameters, test hypotheses, quantify uncertainty, build predictive models, and reason about cause.

Three interpretive lenses, all consistent with the same Kolmogorov axiomatisation:

• Frequentist — probability = long-run relative frequency. Parameters are fixed unknown constants; randomness is in the data.
• Bayesian — probability = degree of belief. Parameters carry distributions; randomness encodes epistemic uncertainty.
• Measure-theoretic / mathematical — probability is a special case of measure theory, with $P(\Omega )=1$ (see measure-theory-and-integration).

This note traces the path from axioms to inference to modern applied statistics. Routine derivations are sketched; the heavy lifting lives in the companion notes above.

2. Probability axioms (Kolmogorov 1933)

A probability space is a triple $(\Omega ,\mathcal{F},P)$:

• $\Omega$ — the sample space, the set of all possible outcomes.
• $\mathcal{F}$ — a $\sigma$-algebra of subsets of $\Omega$, the events — closed under complement and countable union.
• $P:\mathcal{F}\to [0,1]$ — a probability measure satisfying:
  1. Non-negativity — $P(A)\ge 0$ for every $A\in \mathcal{F}$.
  2. Normalisation — $P(\Omega )=1$.
  3. Countable additivity ($\sigma$-additivity) — for pairwise disjoint $\{A_i\}$, $P({\bigcup }_i{A}_i)={\sum }_{i}P(A_i)$.

Kolmogorov’s Grundbegriffe (1933) put probability on the modern measure-theoretic foundation, dissolving century-old paradoxes (Bertrand’s, Borel-Kolmogorov) and unifying discrete and continuous treatments. Everything below — distributions, expectation, CLT, stochastic processes — follows from these three rules plus the measure-theoretic toolkit.

Derived rules.

• $P(\varnothing )=0$, $P(A^c)=1-P(A)$.
• $P(A\cup B)=P(A)+P(B)-P(A\cap B)$ (inclusion-exclusion).
• Continuity — $A_n\uparrow A\Rightarrow P(A_n)\uparrow P(A)$.
• Monotonicity — $A\subseteq B\Rightarrow P(A)\le P(B)$.
• Union bound (Boole) — $P({\bigcup }_i{A}_i)\le {\sum }_{i}P(A_i)$.

Why the $\sigma$-algebra matters. In continuous sample spaces (e.g. $\Omega =[0,1]$) you cannot consistently assign probability to every subset — Vitali’s construction (1905) builds non-measurable sets using the axiom of choice. Restricting to $\mathcal{F}$ (e.g. the Borel sets generated by intervals) makes the theory consistent. See measure-theory-and-integration.

3. Random variables and distributions

A random variable is a measurable function $X:\Omega \to \mathbb{R}$ — measurable meaning $\{X\le x\}\in \mathcal{F}$ for every $x$. The distribution (or law) of $X$ is the pushforward measure $P_X(B)=P(X^{-1}(B))$ on Borel sets of $\mathbb{R}$.

Cumulative distribution function (CDF). $F_X(x)=P(X\le x)$. Right-continuous, non-decreasing, ${\lim }_{x\to -\infty }F=0$, ${\lim }_{x\to \infty }F=1$. CDFs uniquely determine distributions.

Discrete random variable. $X$ takes countably many values; described by a probability mass function (PMF) $p_X(x)=P(X=x)$ with ${\sum }_x{p}_X(x)=1$.

Continuous random variable. $F_X$ is absolutely continuous; described by a probability density function (PDF) $f_X$ with $P(a\le X\le b)={\int }_a^b{f}_X(x)dx$ and ${\int }_{-\infty }^{\infty }{f}_X(x)dx=1$. Note: $f_X(x)$ is not a probability — it can exceed 1.

Mixed distributions (atoms + continuous part) and singular distributions (e.g. Cantor) exist but are rare in practice.

Multivariate random variables

For random vector $X=(X_1,\dots ,X_n)$:

• Joint CDF — $F(x_1,\dots ,x_n)=P(X_1\le x_1,\dots ,X_n\le x_n)$.
• Joint PDF (continuous) — $f(x_1,\dots ,x_n)={\partial }^{n}F/\partial x_1\cdots \partial x_n$.
• Marginal PDF — $f_{X_i}(x_i)=\int f(x_1,\dots ,x_n)d{x}_{-i}$ (integrate out the other variables).
• Conditional PDF — $f_{X_i\mid X_{-i}}(x_i\mid x_{-i})=f(x_1,\dots ,x_n)/f_{X_{-i}}(x_{-i})$.

4. Expectation, variance, moments

Expectation (mean) of $X$:

• Discrete: $E[X]={\sum }_{x}x{p}_X(x)$.
• Continuous: $E[X]=\int x{f}_X(x)dx$.
• Lebesgue-theoretic: $E[X]={\int }_{\Omega }XdP$.

For any measurable $g$: $E[g(X)]=\int g(x)d{P}_X(x)$ (LOTUS — Law Of The Unconscious Statistician).

Properties. Linearity: $E[\alpha X+\beta Y]=\alpha E[X]+\beta E[Y]$ (no independence needed). For independent X, Y: $E[XY]=E[X]E[Y]$.

Variance and standard deviation. $\mathrm{Var}(X)=E[(X-\mu {)}^2]=E[X^2]-{\mu }^2$. ${\sigma }_X=\sqrt{\mathrm{Var}(X)}$.

Covariance and correlation. $\mathrm{Cov}(X,Y)=E[(X-{\mu }_X)(Y-{\mu }_Y)]=E[XY]-E[X]E[Y]$. Correlation $\rho =\mathrm{Cov}(X,Y)/({\sigma }_X{\sigma }_Y)\in [-1,1]$. Independence $\Rightarrow$ zero covariance; the converse fails (e.g. $X\sim N(0,1)$, $Y=X^2$ — uncorrelated but not independent).

Moments and central moments. $m_k=E[X^k]$, ${\mu }_k=E[(X-\mu {)}^k]$. Higher moments:

• ${\mu }_3/{\sigma }^3$ = skewness — asymmetry.
• ${\mu }_4/{\sigma }^4$ = kurtosis — tail heaviness ($=3$ for Gaussian; “excess kurtosis” $={\mu }_4/{\sigma }^4-3$).

5. Moment generating and characteristic functions

Moment generating function (MGF). $M_X(t)=E[e^{tX}]$ when finite in a neighbourhood of 0. Then $M_X^{(k)}(0)=E[X^k]$ (hence the name). MGFs uniquely determine distributions (when they exist) and convert sums of independents to products: $M_{X+Y}(t)=M_X(t)M_Y(t)$.

Characteristic function (CF). ${\phi }_X(t)=E[e^{itX}]$. Always exists (it’s a Fourier transform of the distribution). Uniqueness: distributions are determined by their CFs. Lévy’s continuity theorem — convergence of CFs at every $t$ implies convergence in distribution (the workhorse of CLT-style proofs).

6. Common distributions

See probability-distributions for the full reference; here are the canonical members.

Discrete

Name	Support	PMF	Mean	Variance	Use
Bernoulli($p$)	$\{0,1\}$	$p^k(1-p{)}^{1-k}$	$p$	$p(1-p)$	One coin flip / binary event
Binomial($n,p$)	$\{0,\dots ,n\}$	$\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$	$np$	$np(1-p)$	Count of successes in $n$ trials
Geometric($p$)	$\{1,2,\dots \}$	$(1-p{)}^{k-1}p$	$1/p$	$(1-p)/p^2$	Trials until first success
Poisson($\lambda$)	$\{0,1,\dots \}$	$e^{-\lambda }{\lambda }^k/k!$	$\lambda$	$\lambda$	Count of rare events in fixed time
Negative binomial($r,p$)	$\{r,r+1,\dots \}$	$\left(\genfrac{}{}{0ex}{}{k-1}{r-1}\right)p^r(1-p{)}^{k-r}$	$r/p$	$r(1-p)/p^2$	Trials until $r$-th success; over-dispersed Poisson
Hypergeometric	$\{0,\dots ,K\}$	$\left(\genfrac{}{}{0ex}{}{K}{k}\right)\left(\genfrac{}{}{0ex}{}{N-K}{n-k}\right)/\left(\genfrac{}{}{0ex}{}{N}{n}\right)$	$nK/N$	---	Sampling without replacement

Continuous

Name	Support	PDF	Mean	Variance	Use
Uniform$(a,b)$	$[a,b]$	$1/(b-a)$	$(a+b)/2$	$(b-a{)}^2/12$	Maximum entropy on interval
Exponential($\lambda$)	$[0,\infty )$	$\lambda e^{-\lambda x}$	$1/\lambda$	$1/{\lambda }^2$	Memoryless waiting times
Gamma($\alpha ,\beta$)	$[0,\infty )$	${\beta }^{\alpha }{x}^{\alpha -1}{e}^{-\beta x}/\Gamma (\alpha )$	$\alpha /\beta$	$\alpha /{\beta }^2$	Sum of $\alpha$ exponentials; positive r.v.
Beta($\alpha ,\beta$)	$[0,1]$	$x^{\alpha -1}(1-x{)}^{\beta -1}/B(\alpha ,\beta )$	$\alpha /(\alpha +\beta )$	---	Distributions over probabilities
Normal($\mu ,{\sigma }^2$)	$\mathbb{R}$	$\frac{1}{\sigma \sqrt{2\pi }}{e}^{-(x-\mu {)}^2/2{\sigma }^2}$	$\mu$	${\sigma }^2$	CLT limit; default for noise
Multivariate normal	${\mathbb{R}}^d$	$\frac{1}{\sqrt{(2\pi {)}^d\det \Sigma }}{e}^{-\frac{1}{2}(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu )}$	$\mu$	$\Sigma$	Gaussian models everywhere
Chi-squared($k$)	$[0,\infty )$	---	$k$	$2k$	Sum of $k$ squared standard normals
Student’s $t(\nu )$	$\mathbb{R}$	---	0 (if $\nu >1$)	$\nu /(\nu -2)$ (if $\nu >2$)	Heavy-tailed; mean of small Gaussian sample
F$({\nu }_1,{\nu }_2)$	$[0,\infty )$	---	${\nu }_2/({\nu }_2-2)$	---	Ratio of variances; ANOVA
Cauchy	$\mathbb{R}$	$1/[\pi (1+x^2)]$	undefined	undefined	Pathological example; ratio of Gaussians
Dirichlet($\mathbf{\alpha }$)	simplex	$\frac{1}{B(\mathbf{\alpha })}{\prod }_i{x}_i^{{\alpha }_i-1}$	${\alpha }_i/{\sum }_j{\alpha }_j$	---	Distributions over simplices
Pareto	$[x_m,\infty )$	$\alpha x_m^{\alpha }/x^{\alpha +1}$	$\alpha x_m/(\alpha -1)$ (if $\alpha >1$)	---	Heavy-tailed; wealth, file sizes
Log-normal	$(0,\infty )$	---	$e^{\mu +{\sigma }^2/2}$	---	Multiplicative processes
Weibull	$[0,\infty )$	---	---	---	Reliability / survival

7. Conditional probability and Bayes

For events $A,B$ with $P(B)>0$:

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$$

Multiplication rule. $P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A)$.

Independence. $A\perp B$ iff $P(A\cap B)=P(A)P(B)$ iff $P(A\mid B)=P(A)$.

Law of total probability. For a partition $\{B_i\}$ of $\Omega$:

$$P(A)=\sum _{i}P(A\mid B_i)P(B_i).$$

Bayes’ theorem (Bayes 1763, Laplace 1774).

$$P(B\mid A)=\frac{P(A\mid B)P(B)}{P(A)}.$$

The posterior $P(B\mid A)$ equals the likelihood $P(A\mid B)$ times the prior $P(B)$, normalised by the evidence $P(A)={\sum }_{i}P(A\mid B_i)P(B_i)$. This is the engine of all Bayesian inference (Section 13).

Conditional expectation. $E[X\mid \mathcal{G}]$ — the best $\mathcal{G}$-measurable predictor of $X$ in $L^2$ sense. Satisfies the tower property $E[E[X\mid \mathcal{G}]]=E[X]$ and is the foundation of martingale theory (see stochastic-calculus).

8. Convergence

Random variables can converge in different senses, and the senses are nested:

• Almost sure (a.s.) convergence — $P(X_n\to X)=1$. Strongest.
• Convergence in $L^p$ — $E[\Vert X_n-X{\Vert }^p]\to 0$.
• Convergence in probability — $P(\Vert X_n-X\Vert >\epsilon )\to 0$ for every $\epsilon >0$.
• Convergence in distribution — $F_{X_n}(x)\to F_X(x)$ at every continuity point of $F_X$. Weakest.

Implications: a.s. ⇒ probability ⇒ distribution; $L^p$ ⇒ probability. None of the reverse implications hold in general.

9. Law of large numbers and Central Limit Theorem

Two limit theorems form the bedrock of statistical inference.

Law of Large Numbers (LLN)

Let $X_1,X_2,\dots$ be i.i.d. with $E|X_1|<\infty$ and mean $\mu$. The sample mean ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ satisfies:

• Weak LLN (Khinchin 1929) — ${\bar{X}}_n\to \mu$ in probability.
• Strong LLN (Kolmogorov 1933) — ${\bar{X}}_n\to \mu$ almost surely.

Sample averages converge to the true mean — the justification for estimating expectations by Monte Carlo.

Central Limit Theorem (CLT)

Let $X_i$ be i.i.d. with mean $\mu$, variance ${\sigma }^2<\infty$. Then

$$\sqrt{n}({\bar{X}}_n-\mu )\stackrel{d}{\to }N(0,{\sigma }^2).$$

Historical lineage. De Moivre (1733) for Bernoulli; Laplace (1810) for general i.i.d.; Lyapunov (1901) gave the first rigorous proof; Lindeberg (1922) and Lévy (1925) generalised. The Berry-Esseen theorem (1941) quantifies the rate: ${\sup }_x|F_n(x)-\Phi (x)|\le C\rho /({\sigma }^3\sqrt{n})$ where $\rho =E|X_1-\mu {|}^3$.

Variants.

• Multivariate CLT — $\sqrt{n}({\bar{X}}_n-\mu )\to N(0,\Sigma )$ for vector-valued $X_i$ with covariance $\Sigma$.
• Lindeberg-Feller CLT — for independent but not identically distributed summands, with the Lindeberg condition controlling individual contributions.
• Martingale CLT — for martingale differences (foundation of MLE asymptotic theory).
• Stable laws — when variance is infinite (heavy tails), CLT fails; the limit is an $\alpha$-stable distribution (Lévy 1925).

CLT explains why Gaussian noise is ubiquitous: any process whose noise is the sum of many small independent contributions is approximately Gaussian.

10. Concentration inequalities

Tail bounds without assuming exact distributions — the lifeblood of statistical learning theory and high-dimensional probability.

Markov. For $X\ge 0$: $P(X\ge a)\le E[X]/a$.

Chebyshev. $P(|X-\mu |\ge k\sigma )\le 1/k^2$. Universal but loose for Gaussian-like tails.

Jensen. $\phi (E[X])\le E[\phi (X)]$ for convex $\phi$ (reverse for concave). Hence $E[X^2]\ge E[X{]}^2$, $\log E[X]\ge E[\log X]$ (AM-GM in disguise), and KL divergence $\ge 0$.

Cauchy-Schwarz (probability form). $(E[XY]{)}^2\le E[X^2]E[Y^2]$.

Hoeffding (1963). For $X_i\in [a_i,b_i]$ independent:

$$P\left(\frac{1}{n}\sum _i(X_i-E{X}_i)\ge t\right)\le \exp \left(\frac{-2n^2{t}^2}{{\sum }_i(b_i-a_i{)}^2}\right).$$

The workhorse of PAC learning, A/B test sample size, bandit regret bounds.

Chernoff bound. $P(X\ge a)\le {\min }_{t>0}{e}^{-ta}{M}_X(t)$. The cleanest exponential tail bound when the MGF exists.

Bernstein, McDiarmid, Azuma-Hoeffding (martingale form) — refinements for variance-aware or dependent sums.

Sub-Gaussian / sub-exponential variables. $X$ is sub-Gaussian with parameter $\sigma$ if $P(|X|\ge t)\le 2e^{-t^2/(2{\sigma }^2)}$. Gaussian + bounded variables are sub-Gaussian; many results in high-dimensional probability are stated for this class.

See the catalog at named-inequalities-catalog for many more.

11. Point estimation

Given data $X_1,\dots ,X_n\stackrel{\mathrm{iid}}{\sim }{P}_{\theta }$, estimate the unknown parameter $\theta$.

Definitions.

• Estimator — any function $\hat{\theta }=\hat{\theta }(X_1,\dots ,X_n)$.
• Bias — $\mathrm{bias}(\hat{\theta })=E[\hat{\theta }]-\theta$.
• Variance — $\mathrm{Var}(\hat{\theta })$.
• Mean squared error (MSE) — $E[(\hat{\theta }-\theta {)}^2]=\mathrm{Var}(\hat{\theta })+\mathrm{bias}(\hat{\theta }{)}^2$. The bias-variance trade-off.
• Consistency — ${\hat{\theta }}_n\to \theta$ in probability.
• Efficiency — variance achieves the Cramér-Rao lower bound (CRLB).

Maximum Likelihood Estimation (MLE)

Fisher (1922) formalised the dominant point estimation method. Given likelihood $L(\theta )={\prod }_{i}f(X_i;\theta )$ (or PMF), maximise:

$${\hat{\theta }}_{\mathrm{MLE}}=\arg \underset{\theta }{\max }\log L(\theta )=\arg \underset{\theta }{\max }\sum _i\log f(X_i;\theta ).$$

Examples.

• Bernoulli($p$): ${\hat{p}}_{\mathrm{MLE}}={\bar{X}}_n$ (sample proportion).
• Normal($\mu ,{\sigma }^2$): $\hat{\mu }={\bar{X}}_n$, ${\hat{\sigma }}_{\mathrm{MLE}}^2=\frac{1}{n}\sum (X_i-\bar{X}{)}^2$ (biased; the unbiased $S^2$ divides by $n-1$).
• Poisson($\lambda$): $\hat{\lambda }={\bar{X}}_n$.

Asymptotic properties (under regularity conditions).

• Consistency — ${\hat{\theta }}_{\mathrm{MLE}}\to \theta$ in probability.
• Asymptotic normality — $\sqrt{n}({\hat{\theta }}_{\mathrm{MLE}}-\theta )\to N(0,I(\theta {)}^{-1})$ where $I(\theta )=-E[{\partial }^2\log f/\partial {\theta }^2]$ is the Fisher information.
• Efficiency — asymptotic variance achieves the Cramér-Rao lower bound $I(\theta {)}^{-1}$.
• Functional invariance — ${\widehat{g(\theta )}}_{\mathrm{MLE}}=g({\hat{\theta }}_{\mathrm{MLE}})$.

The conditions exclude pathological cases (support depending on $\theta$, boundary issues, identifiability failure). See hypothesis-testing-mle for proofs and exceptions.

Method of Moments (MoM)

Equate sample moments to population moments and solve. Easy to compute, often less efficient than MLE, but useful as initial values and for distributions where MLE is intractable. E.g. for Gamma($\alpha ,\beta$): set $\bar{X}=\alpha /\beta$ and $S^2=\alpha /{\beta }^2$, solve.

Maximum A Posteriori (MAP)

Bayesian point estimate: maximise the posterior. With prior $\pi (\theta )$:

$${\hat{\theta }}_{\mathrm{MAP}}=\arg \underset{\theta }{\max }[\log L(\theta )+\log \pi (\theta )].$$

A Gaussian prior is $L^2$ regularisation; a Laplace prior is $L^1$ (LASSO). Coincides with MLE when the prior is flat. MAP is not the posterior mean and is reparametrisation-non-invariant.

M-estimators

Generalise MLE: $\hat{\theta }=\arg {\min }_{\theta }\frac{1}{n}{\sum }_i\rho (X_i;\theta )$ for some loss $\rho$. Includes least squares ($\rho (x;\mu )=(x-\mu {)}^2$), MLE ($\rho =-\log f$), and Huber regression. Asymptotic theory parallels MLE (Huber 1964).

12. Confidence intervals and hypothesis testing

Confidence intervals

A $(1-\alpha )$-confidence interval for $\theta$ is a random interval $[L(X),U(X)]$ with

$$P(L(X)\le \theta \le U(X))\ge 1-\alpha .$$

The probability is over the data; $\theta$ is fixed. “95% CI” means: across repeated experiments, 95% of the constructed intervals contain $\theta$. It does not mean the probability the specific interval contains $\theta$ is 95% — that’s a Bayesian credible interval (Section 13).

Wald CI from MLE asymptotics. ${\hat{\theta }}_{\mathrm{MLE}}\pm z_{\alpha /2}\sqrt{I(\hat{\theta }{)}^{-1}/n}$.

Exact CIs exist for some distributions (e.g. Clopper-Pearson for Bernoulli, exact normal-theory CIs for Gaussian).

Hypothesis testing (Neyman-Pearson)

Decide between:

• $H_0$ — null hypothesis, the default / status-quo claim.
• $H_1$ — alternative, what we wish to demonstrate.

Asymmetry: we reject or fail to reject $H_0$; we never “accept” it.

Errors.

	$H_0$ true	$H_1$ true
Don’t reject $H_0$	OK	Type II ($\beta$)
Reject $H_0$	Type I ($\alpha$)	OK

Significance level $\alpha$ is the Type-I error rate we tolerate (conventionally 0.05). Power = $1-\beta$.

Neyman-Pearson lemma (1933). For simple $H_0:\theta ={\theta }_0$ vs $H_1:\theta ={\theta }_1$, the most powerful test of size $\alpha$ rejects when the likelihood ratio $L({\theta }_1)/L({\theta }_0)>k_{\alpha }$ for the threshold making the test size $\alpha$.

p-values. Probability under $H_0$ of seeing a test statistic at least as extreme as the observed value. Reject when $p<\alpha$. Common misuses: “the probability $H_0$ is true” (it is not); “the probability our result is due to chance” (also not). See the ASA statement (Wasserstein + Lazar 2016).

Classical tests.

Test	Setting	Statistic
z-test	Single mean, known $\sigma$	$(\bar{X}-{\mu }_0)/(\sigma /\sqrt{n})$
One-sample t-test	Single mean, unknown $\sigma$	$(\bar{X}-{\mu }_0)/(S/\sqrt{n})\sim t_{n-1}$
Two-sample t-test	Difference of means	---
Welch’s t-test	Different variances	---
Paired t-test	Matched pairs	---
${\chi }^2$ goodness-of-fit	Discrete distribution fit	$\sum (O_i-E_i{)}^2/E_i$
${\chi }^2$ independence	Two categorical variables	---
F-test	Equality of variances; ANOVA	$S_1^2/S_2^2\sim F$
ANOVA	Multiple means	---
Likelihood ratio test	Nested models	$-2\log \Lambda \stackrel{d}{\to }{\chi }_k^2$ (Wilks)
Wald test	$\hat{\theta }$ vs ${\theta }_0$ via Fisher information	---
Score (Lagrange multiplier) test	Without fitting full model	---

For deeper treatment including LR/Wald/Score asymptotic equivalence under $H_0$, see hypothesis-testing-mle.

Multiple testing correction. When testing $m$ hypotheses:

• Bonferroni — reject at $\alpha /m$ each, controls family-wise error rate (FWER). Conservative.
• Holm-Bonferroni (1979) — sequential, strictly more powerful than Bonferroni at same FWER.
• Benjamini-Hochberg (1995) — controls false discovery rate (FDR); modern default for high-dimensional testing.

13. Bayesian inference

Treat $\theta$ as random with prior $\pi (\theta )$ encoding pre-data belief. Given data $D$:

$$p(\theta \mid D)=\frac{p(D\mid \theta )\pi (\theta )}{p(D)}\propto L(\theta )\pi (\theta ).$$

The posterior combines likelihood and prior; inference consists of summarising the posterior (point estimates, credible intervals, posterior predictive).

Point summaries.

• Posterior mean $E[\theta \mid D]$ — minimises squared error.
• Posterior median — minimises absolute error.
• Posterior mode (MAP) — Section 11.

Credible intervals. A $(1-\alpha )$ credible interval $C$ satisfies $P(\theta \in C\mid D)=1-\alpha$. Direct probabilistic interpretation unlike frequentist CIs.

Conjugate priors

A prior is conjugate to a likelihood if the posterior belongs to the same family. The classical pairs:

Likelihood	Prior	Posterior
Bernoulli / Binomial	Beta($\alpha ,\beta$)	Beta($\alpha +k,\beta +n-k$)
Poisson($\lambda$)	Gamma($\alpha ,\beta$)	Gamma($\alpha +\sum X_i,\beta +n$)
Normal (known $\sigma$)	Normal	Normal
Normal (unknown $\mu ,\sigma$)	Normal-Inverse-Gamma	Normal-Inverse-Gamma
Multinomial	Dirichlet	Dirichlet
Exponential	Gamma	Gamma
Uniform $(0,\theta )$	Pareto	Pareto

For non-conjugate problems, computation requires MCMC (mcmc-sampling: Metropolis-Hastings, Gibbs, HMC, NUTS) or variational inference (variational-inference: minimise KL to a tractable family).

Improper and informative priors

• Improper — $\int \pi (\theta )d\theta =\infty$ (e.g. Jeffreys prior $1/\sigma$ on scale). Often OK if posterior remains proper.
• Jeffreys prior — $\pi (\theta )\propto \sqrt{|I(\theta )|}$. Parametrisation-invariant.
• Empirical Bayes — estimate prior hyperparameters from data; sits between full Bayes and frequentist.
• Hierarchical Bayes — priors on hyperparameters; pooling across groups (multilevel models).

Bayesian vs frequentist — a brief comparison

Aspect	Frequentist	Bayesian
Parameter	Fixed unknown	Random variable
Probability of $\theta$	Undefined	Posterior $p(\theta \mid D)$
Inference unit	Test / CI	Posterior
Prior info	Not used (or via shrinkage)	Explicit in prior
Computation	Often closed-form	MCMC / VI / conjugacy
Long-run guarantees	Yes (frequency interpretation)	Coherent updating but no frequency guarantees
Sequential updating	Awkward (multiple-testing issues)	Natural (yesterday’s posterior = today’s prior)

Modern practice frequently mixes the two: use Bayesian computation but evaluate with frequentist coverage; or run frequentist tests but interpret via likelihood ratios.

14. Regression and the linear model

Ordinary least squares (OLS). Model $Y_i=X_i^{\top }\beta +{\epsilon }_i$ with ${\epsilon }_i\stackrel{\mathrm{iid}}{\sim }N(0,{\sigma }^2)$. Estimator:

$$\hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }Y,\mathrm{Var}(\hat{\beta })={\sigma }^2(X^{\top }X{)}^{-1}.$$

Under Gauss-Markov assumptions, OLS is BLUE (best linear unbiased estimator). Under additional normality, OLS = MLE.

Diagnostics.

• R² — fraction of variance explained.
• Residual plots — check linearity, homoscedasticity, normality.
• Cook’s distance — leverage and influence of single observations.
• VIF — variance inflation factor for multicollinearity.

Generalised linear models (GLM) (Nelder + Wedderburn 1972). $g(E[Y_i])=X_i^{\top }\beta$ for a link $g$ and a response from the exponential family.

Response	Distribution	Link $g$
Continuous	Normal	Identity (= OLS)
Binary	Bernoulli	Logit (= logistic regression)
Count	Poisson	Log (= Poisson regression)
Positive continuous	Gamma	Log or inverse
Ordinal	Multinomial	Cumulative logit (proportional odds)

Fit by iteratively reweighted least squares (IRLS) — equivalent to Newton-Raphson on the log-likelihood.

Regularisation

Adding penalty $\lambda \Vert \beta {\Vert }_{\ast }$ to the loss controls overfitting:

• Ridge ($L^2$, Hoerl + Kennard 1970) — $\Vert \beta {\Vert }_2^2$. Closed-form $\hat{\beta }=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }Y$. Shrinks but keeps all coefficients. Bayesian: Gaussian prior on $\beta$.
• LASSO ($L^1$, Tibshirani 1996) — $\Vert \beta {\Vert }_1$. Induces sparsity (sets coefficients exactly to zero). Bayesian: Laplace prior.
• Elastic net (Zou + Hastie 2005) — $\alpha \Vert \beta {\Vert }_1+(1-\alpha )\Vert \beta {\Vert }_2^2$. Best of both.
• Group LASSO, fused LASSO, total variation — structured regularisation.

Choice of $\lambda$: cross-validation, information criteria (AIC, BIC), Bayesian marginal likelihood.

15. Nonparametric methods

When parametric assumptions are too strong:

Density estimation.

• Histogram — simplest; bin width is the bias-variance knob.
• Kernel density estimator (KDE) — $\hat{f}(x)=\frac{1}{nh}{\sum }_{i}K((x-X_i)/h)$ for kernel $K$ and bandwidth $h$. Silverman’s rule of thumb for $h$.
• Adaptive KDE, nearest-neighbour density — variable bandwidth.

Bootstrap (Efron 1979). Estimate the sampling distribution of $\hat{\theta }$ by resampling with replacement from the data:

1. Draw $B$ bootstrap samples $X^{\ast (b)}$.
2. Compute ${\hat{\theta }}^{\ast (b)}$ on each.
3. The empirical distribution of $\{{\hat{\theta }}^{\ast (b)}\}$ approximates the sampling distribution.

Gives standard errors, CIs (percentile, BCa, basic), tests. Foundational under mild conditions; can fail for heavy tails or boundary parameters. Block bootstrap for dependent data.

Permutation tests. Under exchangeability, recompute the test statistic on relabelled data. Exactly valid Type-I error; widely used in genomics and neuroimaging.

Other nonparametric tests.

Test	Setting
Wilcoxon rank-sum / Mann-Whitney U	Two-sample location
Wilcoxon signed-rank	Paired location
Kruskal-Wallis	$k$-sample location (ANOVA analogue)
Spearman rank correlation	Monotone relationship
Kendall’s $\tau$	Concordance-based correlation
Kolmogorov-Smirnov	Equality of distributions
Anderson-Darling	Tail-sensitive K-S variant

Nonparametric regression. Kernel regression (Nadaraya-Watson), local polynomial regression (Loess/Lowess), splines (smoothing, regression, P-splines), Gaussian processes (gaussian-processes).

16. Causal inference primer

Statistical association ≠ causation. Modern causal inference uses formal frameworks to identify causal effects from observational data when randomised trials are impossible.

Potential outcomes (Neyman-Rubin)

Each unit has potential outcomes $Y(0),Y(1)$ under treatment and control; observe only one. The individual treatment effect $Y_i(1)-Y_i(0)$ is fundamentally unobservable for any single unit. The average treatment effect (ATE) $\tau =E[Y(1)-Y(0)]$ is identifiable under assumptions.

Identification assumptions (the unconfoundedness story).

• SUTVA — no interference between units; well-defined treatment.
• Unconfoundedness / ignorability — $\{Y(0),Y(1)\}\perp T\mid X$. Treatment is as-if random given observed covariates.
• Overlap / positivity — $0<P(T=1\mid X)<1$ for all relevant $X$.

Under these, ATE is identified by:

• Stratification / regression adjustment on $X$.
• Propensity score matching/weighting (Rosenbaum + Rubin 1983) — $e(X)=P(T=1\mid X)$.
• Doubly robust estimation — combines outcome model and propensity model; consistent if either is correct.

Causal DAGs (Pearl 1995, 2000)

Directed acyclic graphs encode causal structure. The do-calculus identifies which observational quantities equal interventional ones; the back-door criterion characterises sufficient adjustment sets; the front-door handles certain unmeasured confounding.

Instrumental variables (IV)

When confounders are unobserved, an instrument $Z$ that affects $T$ but $Y$ only through $T$ identifies the effect:

$${\tau }_{\mathrm{IV}}=\frac{\mathrm{Cov}(Y,Z)}{\mathrm{Cov}(T,Z)}.$$

Wright (1928) for econometrics. Identifies the local average treatment effect (LATE) among compliers (Imbens + Angrist 1994).

Regression discontinuity (RDD)

Treatment is assigned based on whether a running variable crosses a threshold. Compare units just above vs just below the cutoff — they are exchangeable in the limit. Thistlethwaite + Campbell (1960); modern theory in Hahn, Todd, Van der Klaauw (2001).

Difference-in-differences (DiD)

Two groups (treated, control), two time periods (pre, post). Under parallel trends in counterfactual untreated outcomes, the treatment effect is

$${\tau }_{\mathrm{DiD}}=({\bar{Y}}_{\mathrm{treat},\mathrm{post}}-{\bar{Y}}_{\mathrm{treat},\mathrm{pre}})-({\bar{Y}}_{\mathrm{ctrl},\mathrm{post}}-{\bar{Y}}_{\mathrm{ctrl},\mathrm{pre}}).$$

Card + Krueger (1994) minimum wage study is the classic application. Recent literature (Goodman-Bacon 2021, Callaway + Sant’Anna 2021) re-examines staggered adoption settings.

Synthetic controls (Abadie + Gardeazabal 2003)

Construct a weighted combination of control units that matches the treated unit’s pre-treatment trajectory; treatment effect = post-treatment divergence. Useful for $N_{\mathrm{treated}}=1$ (single state / country / firm).

See causal-inference for the deeper treatment.

17. Decision theory

A unifying frame: pick action $a$ given data $X$ minimising risk $R(a,\theta )=E_{\theta }[L(a,\theta )]$ for loss $L$.

• Bayes risk $=\int R(a,\theta )\pi (\theta )d\theta$; minimiser is the Bayes estimator.
• Minimax risk $={\min }_a{\max }_{\theta }R(a,\theta )$; conservative.
• Admissibility — no other estimator dominates uniformly.

Stein’s paradox (1956). In dimensions $d\ge 3$, the sample mean $\bar{X}$ as an estimator of $\mu \in {\mathbb{R}}^d$ under squared loss is inadmissible — the James-Stein estimator dominates it. Foundational for shrinkage methods and empirical Bayes.

18. Information-theoretic statistics

• Entropy — $H(X)=-\sum p(x)\log p(x)$ (discrete) or $-\int p\log p$ (continuous, differential entropy).
• KL divergence — $\mathrm{KL}(p\Vert q)=\int p\log (p/q)$. Non-negative; zero iff $p=q$ a.s.; asymmetric. Maximum likelihood = minimising $\mathrm{KL}({\hat{p}}_{\mathrm{empirical}}\Vert p_{\theta })$.
• Mutual information — $I(X;Y)=\mathrm{KL}(p_{X,Y}\Vert p_X{p}_Y)$. Symmetric, non-negative.
• Cross-entropy — $H(p,q)=-\sum p\log q=H(p)+\mathrm{KL}(p\Vert q)$. The standard loss for classification.
• Fisher information — second-derivative curvature of the log-likelihood; quantifies how informative data is about $\theta$.

See information-theory for entropy, channel capacity, source coding.

19. Pitfalls and folklore

• “Statistically significant” ≠ practically important. $p<0.05$ from $n=10^6$ may correspond to a trivial effect size.
• Multiple testing inflates false discoveries. Pre-register, or correct (Bonferroni / BH).
• Correlation ≠ causation. Without randomisation or identification strategy, regression coefficients are not causal.
• Selection bias / survivorship. Sample design dictates what you can infer.
• Simpson’s paradox. Aggregated trend can reverse within subgroups; an omitted variable problem.
• Berkson’s paradox. Conditioning on a common effect induces spurious correlation between causes.
• The 80% power myth. Frequentist power depends on the alternative; “80% power” means little without specifying effect size.
• Confusing CI and credible interval. Frequentist CI is a property of the procedure; Bayesian credible interval is a property of $\theta$ given data.
• The base-rate fallacy. $P(D\mid H)$ is not $P(H\mid D)$; remember the prior. Medical test interpretation pitfall.
• Heavy tails break a lot. Sample mean is a bad estimator when variance is infinite; bootstrap can fail; CLT does not apply.
• Non-stationarity in time series. Most classical tests assume stationarity; check before applying.
• Validation $\ne$ test. With model selection, the validation set becomes part of the training procedure; reserve a true held-out set.
• MLE can be inconsistent or biased on small samples despite asymptotic optimality.
• Improper priors don’t always yield proper posteriors. Check before reporting.

20. Originators and key references

• Bernoulli (1713), De Moivre (1733), Laplace (1812) — early probability.
• Bayes (1763), Laplace (1774) — Bayes’ theorem.
• Gauss (1809), Legendre (1805) — least squares.
• Galton, Pearson (late 19th C) — correlation, regression.
• Fisher (1922, 1925, 1930s) — likelihood, MLE, ANOVA, design of experiments, p-values.
• Neyman + E. Pearson (1928, 1933) — hypothesis testing framework.
• Kolmogorov (1933) — measure-theoretic foundation.
• Wald (1939, 1947) — sequential analysis, decision theory.
• Stein (1956) — Stein’s paradox; foundation of shrinkage.
• Tukey (1962, 1977) — exploratory data analysis, multiple comparisons.
• Cox (1972) — Cox proportional hazards for survival data.
• Rubin (1974), Neyman (1923) — potential outcomes.
• Efron (1979) — bootstrap.
• Pearl (1995, 2000) — causal DAGs and do-calculus.
• Tibshirani (1996) — LASSO.
• Benjamini + Hochberg (1995) — false discovery rate.

21. Further reading

• Casella + Berger, Statistical Inference (2nd ed., 2002) — the standard graduate-level frequentist text.
• Wasserman, All of Statistics (2004), All of Nonparametric Statistics (2006) — compact and modern.
• Bishop, Pattern Recognition and Machine Learning (2006) — Bayesian + ML perspective.
• Gelman, Carlin, Stern, Dunson, Vehtari, Rubin, Bayesian Data Analysis (3rd ed., 2013) — the Bayesian bible.
• Hastie, Tibshirani, Friedman, Elements of Statistical Learning (2nd ed., 2009) — classic for predictive modelling.
• Imbens + Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences (2015) — potential outcomes treatise.
• Pearl, Causality (2nd ed., 2009) — DAG-based causal inference.
• Murphy, Probabilistic Machine Learning (2022, 2023) — modern ML-flavoured probabilistic treatment.
• Billingsley, Probability and Measure (3rd ed., 1995) — measure-theoretic probability reference.
• Durrett, Probability: Theory and Examples (5th ed., 2019) — graduate probability with concrete examples.

22. Companion notes in this library

• probability-fundamentals — axioms, CLT/LLN, inequalities in depth.
• probability-distributions — reference of named distributions.
• hypothesis-testing-mle — frequentist testing.
• bayesian-inference — Bayesian theory.
• mcmc-sampling — sampling algorithms.
• variational-inference — VI / ELBO.
• gaussian-processes — nonparametric Bayes for functions.
• markov-chains-and-hmm — state-space discrete probability.
• stochastic-calculus — continuous-time stochastic processes.
• causal-inference — DAGs, IV, RDD, DiD.
• information-theory — entropy, KL, MI.
• measure-theory-and-integration — analytic foundation.
• _compare_probability-frameworks — Bayesian vs frequentist vs likelihoodist side-by-side.