Statistics and Sufficiency¶
The previous page fixed a set of candidate laws; this one asks what part of the data is needed to choose among them. The answer is often startlingly little, and the word for it is sufficiency: a statistic is sufficient when the rest of the sample, given its value, carries no information about the parameter at all — not approximately, not asymptotically, but exactly and at every sample size. That is a strong claim and it is provable, which makes it one of the few genuinely reassuring results in this part. It is also a claim relative to a model, and the reassurance evaporates precisely when the model does, in a way that is easy to state and easy to miss: the summaries a desk keeps because "these two numbers are all you need" are all you need under an assumption nobody re-checks after the code ships.
This page covers a statistic as a function of the data alone, sufficiency defined through a conditional law that does not depend on the parameter, the factorization theorem as the working test for it, minimal sufficiency and the sense in which the order statistics are always enough and usually too much, and the efficiency and risk numbers a reduction costs once the model it was derived under is wrong. It constructs no estimator and proves no optimality property, so Rao–Blackwell and the variance bounds are Properties of Estimators; it does not maximize a likelihood, which is Maximum Likelihood Estimation; it names the family where a fixed-dimension sufficient statistic exists but derives none of its properties, which is Exponential Families; it decomposes no estimator's error, which is Bias and Variance; it defines no model class, which is Statistical Models; it builds no test, which is Part XII; it computes no posterior, which is Part XVI; and it recommends no storage architecture.
The trading stake is a decision every desk makes and almost never revisits. Risk Measurement races estimators that consume different parts of the same day — a close-to-close estimator sees only closes, a Parkinson estimator sees the high and the low — and finds the ranking depends on which column is read, with the range estimator winning on error while "it runs 2.66 percentage points low". Underneath that comparison is a question this page makes precise: which functions of the data lose nothing, and under what assumption. The fifth section prices the answer on a risk number rather than a volatility estimate, and finds a nominal one-percent loss threshold computed from a running sum and sum of squares breached on \(1.454\%\) of days once the innovations stop being normal.
A Statistic Is Any Function of the Data Alone¶
A statistic is a measurable function \(T(X_1,\dots,X_n)\) of the sample. The whole content of the definition is in the last three words: the function may not involve the unknown parameter. The sample mean is a statistic; \(\bar X-\mu\) is not, because computing it requires knowing the answer. That constraint is what makes a statistic something a practitioner can actually evaluate, and it is why estimators, test statistics and summaries are all statistics by construction.
Statistics vary enormously in how much they compress. The identity map \(T(x)=x\) is a statistic and compresses nothing. The constant \(T(x)=0\) is a statistic and compresses everything, losing all information along the way. Between those extremes sits every summary in use, and the question this page answers is where the boundary lies between compression that costs nothing and compression that costs something.
That boundary is not a property of the statistic. It depends entirely on what the data is being used to learn. A summary that discards nothing relevant to a location parameter may discard everything relevant to a tail, which is why the question has to be asked against a model rather than in the abstract.
Sufficiency Is Losslessness Relative to a Model¶
Let \(\mathcal{P}=\{P_\theta:\theta\in\Theta\}\) be a model. A statistic \(T\) is sufficient for \(\theta\) if the conditional distribution of the sample given \(T\) does not depend on \(\theta\):
The reading is operational. If you know \(T\), then the remaining variation in the data could have been generated by a mechanism that never consulted \(\theta\) — so a colleague holding only \(T\) could simulate a dataset statistically indistinguishable from yours, without knowing the parameter. Nothing about \(\theta\) is left in what they would have to invent.
Proof that sufficiency implies a parameter-free reconstruction, so nothing about \(\theta\) survives outside \(T\)
Suppose \(T\) is sufficient and let \(t\) be an observed value. By definition the conditional law \(Q(\cdot\mid t)=\mathbf{P}_\theta(X\in\cdot\mid T=t)\) is free of \(\theta\). Construct a new sample \(\tilde X\) by drawing from \(Q(\cdot\mid T(X))\) using randomness independent of everything else. Then for any set \(A\),
so \(\tilde X\) has exactly the same distribution as \(X\), for every \(\theta\). The construction used \(T\) and a randomization that involved no parameter, so the original sample can be rebuilt in law from \(T\) alone.
The converse direction is what makes the definition useful in practice: if such a reconstruction exists then no procedure based on \(X\) can outperform, in distribution, the corresponding procedure based on \(T\) plus independent noise — which is the informal statement that \(T\) "contains all the information".
The load-bearing hypothesis is that the conditional law is free of \(\theta\) for every \(\theta\) in \(\Theta\), and \(\Theta\) is part of the model rather than part of the data. Enlarge the model — add a degrees-of-freedom parameter, allow the variance to move over time — and a statistic that was sufficient can stop being sufficient without a single observation changing. Sufficiency is a property of the pair (statistic, model) and is never a property of a statistic, which is why every claim of the form "these two numbers are all we need" is an assertion about the model with the model left unstated.
The Factorization Theorem Reads Sufficiency Off the Likelihood¶
Verifying the definition directly requires computing a conditional distribution, which is unpleasant. The factorization theorem replaces that with inspection.
Proof that sufficiency is equivalent to the likelihood factoring through the statistic, and what the continuous case needs beyond the discrete argument
Claim. \(T\) is sufficient for \(\theta\) if and only if the joint mass function factors as \(p_\theta(x)=g_\theta(T(x))\,h(x)\) for some functions \(g_\theta\) and \(h\), with \(h\) free of \(\theta\).
If. Suppose the factorization holds. Then for \(x\) with \(T(x)=t\),
and \(g_\theta(t)\) cancels because it is constant over the set being summed. The result is free of \(\theta\), so \(T\) is sufficient.
Only if. Suppose \(T\) is sufficient. Write \(p_\theta(x)=\mathbf{P}_\theta(X=x,\ T=T(x))=\mathbf{P}_\theta(T=T(x))\cdot\mathbf{P}_\theta(X=x\mid T=T(x))\). The first factor is a function of \(T(x)\) and \(\theta\); the second is free of \(\theta\) by sufficiency. Setting \(g_\theta=\mathbf{P}_\theta(T=\cdot)\) and \(h=\mathbf{P}(X=\cdot\mid T=\cdot)\) gives the factorization.
The continuous case is the same statement about densities and the same algebra, but the cancellation step conditions on an event of probability zero, so "the conditional distribution given \(T=t\)" has to be constructed rather than divided out. The rigorous version is the Halmos–Savage theorem, which recasts the factorization as a statement about densities with respect to a dominating measure and holds almost everywhere rather than pointwise.
The load-bearing step is that \(g_\theta\) depends on \(x\) only through \(T(x)\), which is what lets it cancel from a ratio over the level set of \(T\). The theorem converts an unverifiable claim about conditional distributions into a syntactic check on the likelihood — write it down, and whatever the parameter touches is a sufficient statistic — and that is the entire reason sufficiency is usable rather than merely definable.
For an iid normal sample the joint density is
in which the data appears only through \(\sum_i x_i\) and \(\sum_i x_i^2\). So \(T=(\sum_i x_i,\sum_i x_i^2)\) is sufficient, equivalently \((\bar x,s^2)\), and the reduction is from \(n\) numbers to two at every \(n\).
import numpy as np
rng = np.random.default_rng(10051)
n = 12 # two samples, one sufficient value
x = 0.4 + 1.7 * rng.standard_normal(n)
q = rng.standard_normal(n)
q = q - q.mean()
y = x.mean() + q * (x.std(ddof=0) / q.std(ddof=0)) # same sum and sum of squares
print(f" sample A sum {x.sum():.10f} sum of squares {(x ** 2).sum():.10f}")
print(f" sample B sum {y.sum():.10f} sum of squares {(y ** 2).sum():.10f}")
print(f" max |A - B| elementwise {np.abs(np.sort(x) - np.sort(y)).max():.4f}")
def loglik(mu, sig, d):
return (-d.size * np.log(sig) - ((d - mu) ** 2).sum() / (2 * sig ** 2)).item()
print(" mu sigma loglik(A) loglik(B) gap")
for mu, sg in ((0.0, 1.0), (0.4, 1.7), (-1.0, 3.0), (2.5, 0.8)):
a, b = loglik(mu, sg, x), loglik(mu, sg, y)
print(f" {mu:8.2f} {sg:9.2f} {a:12.6f} {b:12.6f} {abs(a - b):11.2e}")
# => sample A sum 11.7097396034 sum of squares 45.5835983056
# sample B sum 11.7097396034 sum of squares 45.5835983056
# max |A - B| elementwise 1.3158
# mu sigma loglik(A) loglik(B) gap
# 0.00 1.00 -22.791799 -22.791799 3.55e-15
# 0.40 1.70 -12.965429 -12.965429 0.00e+00
# -1.00 3.00 -17.683518 -17.683518 0.00e+00
# 2.50 0.80 -45.787043 -45.787043 0.00e+00
The first three lines build the demonstration. Sample A is twelve draws from a normal; sample B is a different set of twelve numbers constructed to have the identical sum and identical sum of squares. They agree on those two quantities to ten decimal places and they are genuinely different data — sorted against each other, the largest elementwise discrepancy is \(1.3158\), which on a series with standard deviation \(1.7\) is most of a standard deviation.
The table is the factorization theorem with the abstraction removed. At four widely separated parameter values the two samples produce log-likelihoods agreeing to \(3.55\times10^{-15}\) or to exactly zero — floating-point identity, not approximate agreement. The likelihood surface over \((\mu,\sigma)\) is not merely similar for the two datasets; it is the same surface.
The consequence is worth stating plainly, because it is the strongest positive result in this part. Every inference that depends on the data only through the likelihood — maximum likelihood, likelihood ratios, Bayes with any prior — returns bit-identical answers on sample A and sample B. Under the normal model, storing twelve numbers rather than two is storing ten numbers that no likelihood-based procedure can read, and that is a theorem rather than an approximation.
Minimal Sufficiency Is the Most the Model Permits, and the Order Statistics Are Always Enough¶
Sufficiency alone is a weak requirement, because the whole sample is trivially sufficient. The useful question is how far compression can go.
Proof that the order statistics are sufficient for any iid model, and that minimal sufficiency is a statement about level sets of the likelihood ratio
For an iid sample the joint density is \(\prod_i f_\theta(x_i)\), and a product does not depend on the order of its factors. So \(p_\theta(x)=p_\theta(x_{\pi})\) for every permutation \(\pi\), and taking \(T(x)\) to be the vector of order statistics \((x_{(1)},\dots,x_{(n)})\) gives the factorization \(p_\theta(x)=g_\theta(T(x))\cdot 1\). The order statistics are sufficient for every iid model, with no assumption on \(f_\theta\) whatever.
That reduction discards only the labels, which is why it is free and why it is nearly worthless: the dimension of \(T\) still grows with \(n\). A statistic \(T\) is minimal sufficient if it is a function of every other sufficient statistic, and the working characterization is due to Lehmann and Scheffé: \(T\) is minimal sufficient when
The level sets of a minimal sufficient statistic are exactly the sets of samples the model cannot tell apart. For the normal with both parameters unknown that partition is indexed by \((\sum x_i,\sum x_i^2)\); for a Cauchy location family it is the full set of order statistics, and no further reduction exists at any \(n\).
The load-bearing distinction is between a reduction that is available and one that is useful. Sufficiency guarantees losslessness and says nothing about dimension, and the order statistics show the two can come apart completely. A model earns a fixed-dimension summary or it does not, and which models earn one is a sharp question with a sharp answer — that answer is Exponential Families, and the Cauchy is the standard reminder that the answer is sometimes no.
The order statistics being sufficient for every iid model is why rank and quantile methods lose so little, and why they are the right default when the family is in doubt
Discarding the time order of an iid sample costs nothing at all, which sounds like a technicality and is the licence under which a large part of nonparametric statistics operates. A method that reads only ranks or only quantiles is using a function of a statistic that is sufficient for every iid model simultaneously, so it is never throwing away information about a parameter of any such model — it is only declining to exploit parametric structure that may or may not be there. That is the precise sense in which the median is not a cruder mean: it is a functional of an always-sufficient reduction, whereas the sample mean's optimality is a claim that holds under a family and fails outside it, as the next section measures. The caveat is equally precise and is the one people skip: the order statistics are sufficient for an iid model, and a return series is the case where that hypothesis is the one under suspicion, which is why the same argument gives no comfort at all about serial dependence.
Reduction Is Free Only If the Model Is Right¶
Everything above holds exactly under a stated model. What follows is what it costs when the model is a little wrong, and the answer is that the reduction stops being lossless in a way nothing in the pipeline reports.
import numpy as np
from scipy.stats import t as tdist
rng = np.random.default_rng(10053)
reps, n = 40_000, 200 # the mean is optimal for exactly one
print(" law mean var median var 10% trimmed var best / mean")
for name, nu in (("normal", None), ("t(5)", 5.0), ("t(3)", 3.0)):
if nu is None:
d = rng.standard_normal((reps, n))
else:
d = tdist.rvs(nu, size=(reps, n), random_state=rng) / np.sqrt(nu / (nu - 2))
ds = np.sort(d, axis=1)
k = int(0.1 * n)
v = [d.mean(axis=1).var(), np.median(d, axis=1).var(), ds[:, k:n - k].mean(axis=1).var()]
print(f" {name:<12} {v[0]:11.6f} {v[1]:13.6f} {v[2]:18.6f} {min(v) / v[0]:14.3f}")
# => law mean var median var 10% trimmed var best / mean
# normal 0.004990 0.007738 0.005276 1.000
# t(5) 0.004979 0.005152 0.004023 0.808
# t(3) 0.005010 0.003077 0.002701 0.539
Under the normal the pair \((\bar x,s^2)\) is sufficient and the sample mean is the efficient estimator of location, and the table confirms it: the mean's sampling variance is \(0.004990\) against the median's \(0.007738\) and the trimmed mean's \(0.005276\), so the ratio of the best available to the mean is \(1.000\). Nothing beats it, which is exactly what sufficiency plus the normal model predicts.
Move to a \(t(5)\) — a tail milder than the one the course fits to SPY — and the ordering inverts. The trimmed mean's variance is \(0.004023\) against the mean's \(0.004979\), a ratio of \(0.808\). At \(t(3)\) the trimmed mean reaches \(0.002701\) against the mean's \(0.005010\), a ratio of \(0.539\). The sample mean is discarding forty-six percent of the achievable precision, and the information it is discarding lives in exactly the observations the sufficient pair summarized into a single sum of squares.
import numpy as np
from scipy.stats import t as tdist
rng = np.random.default_rng(10057)
reps, n = 20_000, 2_000 # eight years of daily returns
print(" law true 1% VaR from the pair from the sample breach rate, pair")
for name, nu in (("normal", None), ("t(5)", 5.0), ("t(3)", 3.0)):
if nu is None:
d = rng.standard_normal((reps, n))
truth = -2.3263
else:
d = tdist.rvs(nu, size=(reps, n), random_state=rng) / np.sqrt(nu / (nu - 2))
truth = tdist.ppf(0.01, nu) / np.sqrt(nu / (nu - 2))
pair = d.mean(axis=1) - 2.3263 * d.std(axis=1, ddof=1) # all a running sum can give you
emp = np.quantile(d, 0.01, axis=1)
br = np.mean(d < pair[:, None])
print(f" {name:<12} {truth:11.4f} {pair.mean():16.4f} {emp.mean():18.4f} {100 * br:19.3f}%")
# => law true 1% VaR from the pair from the sample breach rate, pair
# normal -2.3263 -2.3262 -2.3171 0.999%
# t(5) -2.6065 -2.3254 -2.5934 1.505%
# t(3) -2.6216 -2.3039 -2.6041 1.454%
The normal row is the system working. A one-percent loss threshold computed from nothing but a running sum and a running sum of squares lands at \(-2.3262\) against a truth of \(-2.3263\), and the realized breach rate is \(0.999\%\) against a budgeted \(1\%\). Two accumulators, constant memory, exactly right.
The other two rows are the honest failure and they are worse than the efficiency table. Under a \(t(5)\) the true one-percent quantile is \(-2.6065\) and the pair returns \(-2.3254\); under a \(t(3)\) the truth is \(-2.6216\) and the pair returns \(-2.3039\). The realized breach rates are \(1.505\%\) and \(1.454\%\) — a risk limit sized for one exceedance in a hundred days is taking closer to one and a half, which over a trading year is roughly three unbudgeted breaches. The final column of the previous block already said why: the pair is not sufficient for this model, so the tail shape it cannot see is precisely the quantity being asked for.
The last comparison is the one that makes it actionable. The empirical quantile computed from the retained sample returns \(-2.5934\) and \(-2.6041\) against truths of \(-2.6065\) and \(-2.6216\) — close, at every row, under every law, with no model assumption at all. The information needed to get the answer right was in the data the whole time and had been thrown away by a summary chosen under an assumption that was never revisited.
A summary that is sufficient under the model you fitted is lossy under the model that generated the data, and the loss concentrates exactly where you were not looking
The pattern is a system design decision rather than a statistical one, which is why it survives code review. Running sums and sums of squares are the natural accumulators for a streaming risk system: they are \(O(1)\) in memory, exact, mergeable across shards, and provably lossless — under a normal model. Nothing in the implementation records that last clause, and nothing downstream can recover what the accumulator discarded, so the failure surfaces years later as a breach count that runs high with no bug to find. Two cheap defences exist and neither requires abandoning the design. Keep a bounded reservoir or a \(t\)-digest alongside the moments so an empirical quantile remains computable, and then backtest the threshold by counting realized breaches against the nominal rate rather than by checking that the volatility estimate looks reasonable. The breach count is the diagnostic that distinguishes the rows above; the volatility estimate is not, because it is correct in all of them. The general form is worth carrying beyond risk systems: whenever a pipeline reduces before it stores, the reduction has fixed the set of questions that can ever be asked of that history, and it was chosen by whoever wrote the accumulator.
What You Are Allowed to Throw Away¶
Three things were established and they run in opposite directions. Sufficiency is real and exact: two datasets sharing a sufficient statistic produce likelihood surfaces identical to machine precision, so under a stated model the discarded observations are unreadable by any likelihood-based procedure at any sample size. The reduction is available for free in one form for every iid model, since the order statistics discard only labels — and that form does not compress, which is why the useful question is whether a fixed-dimension summary exists. And when the model is wrong the guarantee does not degrade gracefully; it is simply absent, costing forty-six percent of achievable precision on a location estimate and roughly half again too many breaches on a tail threshold.
The symmetry with the previous page is exact and worth naming. Statistical Models showed that a misspecified model remains consistent for a pseudo-true parameter, so the estimator keeps working and answers a different question. Sufficiency is the same phenomenon one level down: a reduction derived under a model keeps working and preserves a different quantity, and in both cases the output is formatted identically to the correct one. The two failures compound, because a summary chosen under a model is usually stored, and storage makes the assumption permanent in a way a fit does not — a misspecified model can be refitted tomorrow, and a discarded observation cannot be recovered.
What remains is the sharp question the fourth section raised and declined to answer: which models admit a sufficient statistic whose dimension does not grow with the sample. The answer is a single family, it is essentially the only one, and nearly everything that makes routine statistical practice convenient — conjugate priors, closed-form updates, concave likelihoods, constant-memory accumulators — turns out to be the same fact about that family wearing different clothes. That is Exponential Families.
Every summary is a decision about which questions the future is allowed to ask, and it is made under a model, once, by whoever wrote the code that stores it.