Part XIV — Model Selection

Part XIII fitted a specification and found that every quantity on the output page was conditional on it — a slope defined by the window of \(x\) that was sampled, a coefficient defined by which other columns stood beside it, a probability defined by a variance function nobody chose deliberately. It closed by naming what it had not done: rank a candidate against a non-nested rival, penalize a complexity, choose a penalty by resampling. This part does those things, and in doing them it moves the conditionality up one level. The specification is no longer given; it is selected, by a procedure, from a set, using the same data that will then be used to report on it. What that adds is a second estimator sitting on top of the first, and the second one has a sampling distribution, a bias, a variance and an error bar of its own — none of which appear anywhere in the output of the first.

The dependencies run in file order and the first page is the foundation for the four that follow. Bias–Variance Tradeoff establishes the three-term decomposition of prediction error, the target every later page is trying to minimise, and proves in the same breath that the split determining what to do about it is not identifiable from one sample. Cross-Validation and Information Criteria (AIC/BIC) are two estimates of that page's curve drawn without knowing the truth — the first by withholding data, the second by pricing the optimism analytically — and section 4 of the second shows they are asymptotically the same rule, so the choice between them is about assumptions and arithmetic rather than outlook. Feature Selection removes the assumption both of them were quietly making, that the candidate set arrived from somewhere else, and shows that a search corrupts every statistic computed afterwards. Model Averaging is the escape, since an average contains no maximum, and it is honest about what it exchanges: a selection problem for an estimation problem one level up. Where this part stops is worth stating too: no family-wise error rate is controlled and nothing is charged for the number of candidates a search examined, which is Part XV; no posterior over parameters is computed and no Bayes factor evaluated, which is Part XVI; no optimizer is built and no sampler run, which is Part XVII; the \(\ell_1\) geometry that makes a penalty sparse is not re-derived, because Regularization derived it and this part spends it; no resampling scheme is constructed from first principles, which is Bootstrap Methods; and no conditional variance is modelled, which is Time Series.

One failure runs through the part and it is not an error of arithmetic: every estimate below is correct, and each is correct about an object slightly different from the one being reported. A decomposition sums to its simulated total at \(1.0678\) against \(1.0681\) and says that \(1.0000\) of it belongs to nobody, while the complexity minimising it is degree \(3\) against a truth of degree \(5\) and does not reach the truth until \(n=20{,}000\). Two smoothers sharing \(\operatorname{tr}(S)=4.000\) and a variance of \(0.0665\) to every printed digit have squared biases of \(0.0012\) and \(0.1494\). Five-fold cross-validation is unbiased at \(1.3371\) against \(1.3407\) for a model trained on eighty rows, and the model shipped was trained on a hundred and has an error of \(1.2559\). The standard error printed beside that score is \(0.51\) of its true spread at \(K=2\). The same estimator on the same data returns \(-0.0469\) or \(+0.5023\) depending only on how the indices were partitioned. BIC recovers a sparse truth \(0.998\) of the time while AIC stalls at \(0.725\) forever, and every column reverses when no finite truth exists. A single change of response variable moves a criterion by \(4{,}598.6\) points and makes the naive verdict pick the log model \(1.0000\) of the time whichever model is true. Forward selection of five columns from a hundred pure-noise candidates returns \(R^{2}=0.2617\) and a median \(p\)-value of \(0.000029\), rejecting on every one of three thousand datasets. Screening before the folds reports \(0.4524\) where the honest procedure reports \(1.3228\). A nominal \(95\%\) interval covers its true zero \(0.58\%\) of the time while narrowing. And a combination whose weights are fitted on the members' own training rows puts \(1.0000\) of them on the single member that memorised the labels. In every case the estimator did what it was built to do, and the sentence describing the result names something else.

Topics

Topic	Focus
Bias–Variance Tradeoff	Prediction error as a noise floor plus a squared bias plus a variance, the first cross term vanishing only because the test observation was assumed independent of the training sample — an assumption about the split rather than about the model; exact totals of \(1.0678\), \(1.1000\) and \(1.2167\) matching simulation at \(1.0681\), \(1.1002\) and \(1.2166\), with the error-minimising degree \(3\) beating the true degree \(5\)'s \(1.1000\) and the minimiser climbing \(2\), \(3\), \(3\), \(3\), \(4\), \(5\) as \(n\) runs \(30\) to \(20{,}000\) while training error falls monotonically \(3.6925\) to \(0.7844\); complexity as \(\sigma^{2}\operatorname{tr}(SS^\top)/n\), matching measured variance at \(0.0665\) against \(0.0667\) while the conventionally reported \(\operatorname{tr}(S)\) overstates ridge by \(31\%\) at \(2.981\) against \(2.267\), and two smoothers at an identical \(4.000\) differing \(124\)-fold in squared bias; and a split unidentifiable from one sample, two models totalling \(1.2167\) and \(1.2161\) separated by \(0.0006\) against a per-dataset spread of \(0.2329\), with an estimated floor returning \(-0.2463\) for a squared quantity
Cross-Validation	\(K\)-fold exactly unbiased for the error of a model trained on \(n-n/K\) rows, matching at \(1.6883\) against \(1.6936\) and \(1.3371\) against \(1.3407\) while the shipped model's \(1.2559\) is overstated by \(34.4\%\), \(6.5\%\), \(2.9\%\) and \(-0.4\%\) — a pessimism that charges the richer candidate more; the deleted residual \(e_i/(1-h_{ii})\) reproducing brute-force leave-one-out to \(1.25\times10^{-14}\) for one fit against eighty and leaving its class silently for an adaptive smoother at \(1.562266\) against an honest \(1.571273\), then selecting \(\lambda=3.16\) at \(1.2150\) against least squares' \(1.2336\) with \(\operatorname{tr}(H)=4.633\) of \(6\); fold errors sharing training data so the reported standard error runs \(0.51\), \(0.73\), \(0.81\) and \(0.88\) of the truth with no rescaling available; and leakage as a function of seam length, shuffled folds manufacturing \(+0.5023\) from independent series while contiguous folds are safe at \(-0.0469\) with eight boundaries and contaminated at \(+0.1040\) with ninety-eight
Information Criteria (AIC/BIC)	AIC's \(2p\) as the measured optimism of a maximised log-likelihood, equal to \(\operatorname{tr}(J^{-1}K)\) and collapsing to \(p\) only under correct specification, against BIC's \(p\log n\) from a Laplace approximation to a marginal likelihood and therefore a different target entirely; the efficiency-versus-consistency theorem measured, BIC recovering a sparse truth at \(0.924\), \(0.976\), \(0.991\) and \(0.998\) and predicting twice as well at \(0.0557\) against \(0.0959\), with every column reversing under a tapering truth where AIC's \(0.0096\) beats BIC's \(0.0167\); AIC's refusal to converge shown to be a fixed tail area, taking one surplus parameter \(0.1520\) of the time against \(P(\chi^{2}_{1}>2)=0.1573\); AIC and leave-one-out selecting identically on \(0.804\), \(0.870\), \(0.954\) and \(0.993\) of datasets with out-of-sample errors agreeing to four decimals; and both criteria voided silently by a change of response variable worth \(4{,}598.6\) points and by serial dependence cutting BIC's hit rate from \(0.986\) to \(0.005\) against \(5.1\) effective rows of \(500\)
Feature Selection	\(2^{p}\) subsets making every practical procedure a heuristic, with forward selection provably unable to recover a jointly-perfect pair once a marginally-superior decoy is present; the search inflating every in-sample statistic computed as though it had not happened, five columns drawn from \(5\), \(10\), \(25\), \(50\) and \(100\) pure-noise candidates giving \(R^{2}\) from \(0.0500\) to \(0.2617\) and \(F\)-test rejection rates of \(0.0447\), \(0.2937\), \(0.8767\), \(0.9963\) and \(1.0000\) against a nominal \(0.05\) with a median \(p\)-value of \(0.000029\); cross-validation failing to detect a leak that preceded it, screening on the full sample reporting \(0.4524\) where the same folds with the screen moved inside report \(1.3228\) against a mean-prediction baseline of \(1.00\); and a selected coefficient's nominal \(95\%\) interval covering its true zero \(0.9508\), \(0.7734\), \(0.3402\) and \(0.0058\) of the time while narrowing from \(0.4481\) to \(0.4300\), with sample splitting restoring \(0.9436\) at a width of \(0.6530\)
Model Averaging	The ambiguity identity holding exactly and unconditionally, verified at \(0.583035-0.327325=0.255709\) with a gap of \(5.55\times10^{-17}\), and promising a win against the mean member rather than the best; bagging trading instability for bias, improving forward stepwise \(18.8\%\) as variance fell \(0.3205\) to \(0.1948\) while degrading least squares \(27.4\%\) because a linear estimator has nothing to cancel and pays the bootstrap regardless, with fifty members nominally keeping three columns touching \(17\) of \(20\) between them; weights fitted on the members' own rows placing \(1.0000\) on the one member that memorised the labels for a score of \(1.7728\), held-out folds cutting that to \(0.8336\), non-negativity to \(0.7654\) and no fitting at all winning at \(0.7471\) against a best single member's \(0.8028\); and optimal weights worth progressively more as the oracle improves \(0.7490\) to \(0.5024\) while their estimated version reverses and loses to \(1/N\) by \(18.1\%\)