Part XIII — Regression and Statistical Models¶

Part XII reduced evidence to a single bit against a hypothesis named in advance, and closed on the observation that the arithmetic is almost never the problem — that the reference distribution is almost always correct for some question, and that nothing in any output records whether that question was the one being asked. This part stops asking whether a number differs from something and starts asking what it is a function of. What the change adds is a specification: a set of columns, a link, a penalty, a sample. What it discovers under that specification is the same finding one layer deeper, because now the offending assumption is not hidden in a table someone else derived — it is sitting in the analyst's own design matrix, and every quantity on the output page is defined relative to it. A slope, a coefficient, a standard error, a probability and a selected set are each conditional on something the report does not carry.

The dependencies run in file order and the first two pages are one argument told twice. Simple Linear Regression establishes the estimand — the best linear predictor under the sampled marginal of $x$, which is the conditional mean only when the conditional mean is affine — and Multiple Linear Regression is the same page with a projection, where Frisch–Waugh–Lovell replaces the covariance ratio with a partial effect and the coefficient stops describing its own predictor. Generalized Linear Models and Logistic Regression are a pair, the second being the binomial member of the first's class, and reading either alone misses that the variance function which fails on counts cannot fail on a two-valued response, so the failures swap rather than accumulate. Regularization is what the ill-conditioning of the second page forces, and it answers that problem completely while quietly answering a second one — which variables matter — badly. The last two are a pair in the other direction: Model Diagnostics asks the fit to criticize itself and Residual Analysis examines the object it criticizes with, which turns out to have been shaped by the fit before either page saw it. Where this part stops is worth stating too: no candidate specification is ranked against a non-nested rival, no complexity is penalized by a criterion, and no $\lambda$ is chosen by resampling, all of which are Part XIV; no family of specifications is corrected for the size of the search that found it, which is Part XV; no posterior over $\beta$ is computed and no model is given a probability, which is Part XVI; no optimizer is built for a non-concave surface and no sampler is run, which is Part XVII; no lagged dependent variable is fitted and no conditional variance is modelled, which is Time Series; no resampling scheme is constructed from first principles, which is Bootstrap Methods; and the correspondence between a Gaussian prior and ridge is not re-derived, because Maximum a Posteriori Estimation derived it and this part spends it.

One failure runs through the part and it has a single shape: the fit converges, the output is complete and internally consistent, and the quantity the reader takes away is defined by something the output does not print. One curved relationship sampled four ways gives slopes of $+0.5003$, $-0.3765$, $+1.3834$ and $+1.2948$, each matching its own population target, with the sign reversing between two halves of the same data. A predictor measured with noise returns a slope of $0.0997$ when the truth is $0.50$ while its $t$-statistic holds at $11.16$ and the nominal $5\%$ test rejects $1.0000$ of the time. Five designs with identical slopes and identical error variance report $R^{2}$ from $0.0055$ to $0.9552$. A coefficient reads $1.4716$ alone and $1.0475$ beside a predictor correlated at $0.90$, while an irrelevant column moves it to $1.4719$. Collinearity drives $\operatorname{sd}(\hat\beta_1)$ from $0.0713$ to $1.6019$ as $\operatorname{sd}(\hat\beta_1+\hat\beta_2)$ falls to $0.0713$ and out-of-sample error stays flat at $1.0157$ to $1.0162$. A Poisson fit prints the same standard error, $0.0185$, whether the true spread is $0.0183$ or $0.0516$, and declares a slope of exactly zero significant $47.50\%$ of the time. A residual deviance calibrated at $201.2$ against $198$ with thirty trials per row reads $265.7$ with one, and rejects a correct model $1.0000$ of the time. A perfectly separating predictor is fitted at $b_1=1194.08$ and reported with a Wald p-value of $0.9861$. Rebalanced training data displaces an intercept by $0.3227$, leaving AUC identical at $0.5744$ while moving the invested fraction from $50.27\%$ to $90.50\%$. A lasso at its prediction-optimal penalty admits a true-zero decoy $42.6\%$ of the time, and at a sparser setting retains that decoy $10.8\%$ of the time against a genuine predictor's $1.2\%$. An observation planted $8.0$ units off the line comes back with a residual of $2.4293$ while doubling the slope, and as its accomplices multiply the maximum Cook's distance falls from $8.9974$ to $0.0906$ — below the $0.1133$ a clean sample produces by chance — while the estimate degrades and the prescribed remedy makes it worse. And one Breusch–Pagan test on one set of residuals rejects $0.0375$ looking along the fitted values and $0.5345$ looking along $|x|$. In every case the algebra is right, the software is right, and the gap between what was computed and what was wanted appears in no field of the output.

Topics¶

Topic	Focus
Simple Linear Regression	Least squares as an exact solution to the projection problem rather than an approximate one to the conditional-mean problem, so its estimand is the best linear predictor under the sampled marginal of $x$; one curved relationship giving slopes $+0.5003$, $-0.3765$, $+1.3834$ and $+1.2948$ against population values $0.5000$, $-0.3783$, $1.3783$ and $1.2731$ with the sign reversing across two halves of one dataset; Gauss–Markov guaranteeing efficiency and guaranteeing nothing against a wrong mean function; classical measurement error multiplying the slope by $\sigma^{2}_{x^{}}/(\sigma^{2}_{x^{}}+\sigma^{2}_u)$ to give $0.0997$ against a truth of $0.50$ while the mean $t$ holds at $11.16$ and rejection sits at $1.0000$; and $R^{2}$ running $0.0055$ to $0.9552$ across five designs whose slopes are all $0.50$, pinned at $0.2002$ by construction when the predictor is a sign
Multiple Linear Regression	The normal equations as the stationarity condition of a convex quadratic whose residual is orthogonal to every column, so the fit determines a projection and the coefficients are only its coordinates in a chosen basis; Frisch–Waugh–Lovell verified to $4.44\times10^{-16}$, making $b_1$ read $1.4716$ alone, $1.0475$ beside a predictor correlated at $0.90$ and $1.4719$ beside an irrelevant one; a joint $F$ whose power depends on one noncentrality, delivering $0.2505$ and $0.2459$ under two alternatives of matched total signal where a targeted $t$ swings $0.4999$ to $0.1389$; and multicollinearity as ill-conditioning, taking $\operatorname{cond}(X^\top X)$ from $1.3$ to $2052.4$ and $\operatorname{sd}(\hat\beta_1)$ from $0.0713$ to $1.6019$ while the sum's spread falls to $0.0713$ and out-of-sample error holds at $1.0157$ to $1.0162$
Generalized Linear Models	An exponential family, a linear predictor and a link, with $b'(\theta)=\mu$ and $b''(\theta)=\operatorname{var}(y)/\phi$ making the variance a deterministic function of the mean once the family is named; the canonical link reducing the score to $X^\top(y-\mu)=0$ so one fourteen-line routine fits Gaussian, Poisson and binomial responses to score residuals of $1.84\times10^{-12}$, $1.75\times10^{-12}$ and $3.44\times10^{-12}$ in $2$, $9$ and $6$ iterations and reproduces `lstsq` to $9.99\times10^{-16}$; overdispersion freezing the nominal standard error at $0.0185$ while the true spread reaches $0.0516$, driving a nominal $5\%$ test of a zero slope to size $0.4750$ that the Pearson scale repairs to $0.0505$; and a residual deviance calibrated at $201.2$ against $198$ with thirty trials per row and destroyed at one, reading $265.7$ with standard deviation $6.4$ and rejecting a correct model $1.0000$ of the time
Logistic Regression	The logit link making each coefficient one additive log-odds update, so $e^{\beta_j}$ is an odds ratio rather than a probability change; strict concavity giving a unique optimum that is unattained exactly when the classes separate, where the standard error diverges faster than the coefficient and a perfect predictor is reported at $b_1=1194.08$ with $\widehat{\operatorname{se}}=68532.56$ and a Wald p-value of $0.9861$ that a ridge of $10^{-3}$ repairs to a finite $47.27$; maximum likelihood inflated at order $p/n$, averaging $1.485$ times the truth at $n=50$ and $p=10$ against $1.006$ at $n=1000$ and $p=2$ while the Wald size holds between $0.0453$ and $0.0584$; and a rebalanced training set displacing the intercept by a measured $0.3227$ against a theoretical $0.3228$, leaving AUC identical at $0.5744$ while the invested fraction moves from $50.27\%$ to $90.50\%$
Regularization	A penalty as a constraint set whose geometry decides the outcome, the $\ell_1$ kink giving a subdifferential interval that makes exact zeros an event of positive probability where the smooth $\ell_2$ penalty produces them almost never; ridge acting per principal direction with multiplier $d_i^{2}/(d_i^{2}+\lambda)$ measured from $0.9901$ to $0.0099$, taking the worst direction's coefficient spread from $10.004$ to $0.099$ and $\operatorname{cond}(X^\top X)$ from $10000.0$ to exactly $100.0$ for a total error of $3.0061$ against least squares' $118.8089$; an existence theorem always satisfied but worth $94.4\%$, $79.6\%$ or $61.9\%$ at optima of $2.51$, $1.00$ and $0.50$ depending on where $\beta$ points; and a lasso admitting a true-zero decoy $42.6\%$ of the time at its prediction-optimal $\lambda=14.9$ and retaining it $10.8\%$ against a genuine predictor's $1.2\%$ at $\lambda=150$, while out-of-sample error rises $1.0721$ to $3.3087$
Model Diagnostics	The hat matrix symmetric and idempotent to $2.78\times10^{-17}$ with $\operatorname{tr}H=2.0000$ and mean leverage $p/n=0.0333$, so leverage is a fixed budget the design distributes; $\operatorname{var}(e_i)=\sigma^{2}(1-h_{ii})$ making an influential point's residual small by construction, so an observation planted $8.0$ off the line returns $2.4293$ while moving the slope from $0.4239$ to $0.8629$; Sherman–Morrison making every deletion diagnostic free and inflating deleted prediction error by $1/(1-h_{ii})$; Cook's distance needing both factors, since leverage read $0.5739$ identically for points differing twelvefold in effect and the largest studentized residual $9.546$ belonged to one moving the slope $-0.013$ where $5.751$ moved it $+0.424$; and single-deletion reasoning reaching one observation, with Cook's $D$ falling $8.9974$ to $0.0906$ below a clean sample's $0.1133$ as the estimate degrades to $-0.1776$ and the prescribed remedy worsens it
Residual Analysis	A residual as an error with $p$ dimensions projected out, correlated and unequally scaled under a correct model, with measured standard deviations $0.9836$, $0.9827$ and $0.3430$ against $\sigma\sqrt{1-h_{ii}}$ of $0.9852$, $0.9800$ and $0.3432$, a pair correlation of $+0.1083$ against $+0.1084$, variances summing to $37.001$ against $\sigma^{2}(n-p)=37.000$ and $\lVert X^\top e\rVert_\infty=6.66\times10^{-15}$; only the externally studentized version carrying an exact $t_{n-p-1}$ law; one Breusch–Pagan test rejecting $0.0375$, $0.3855$ and $0.2865$ along the fitted values and $0.5345$, $1.0000$ and $1.0000$ along $