Appointment Benchmark — Candidate Transcript

• Candidate: vaiu-sci-stat-prof-ml v1.0.0 — Professor of Statistics (Statistical Machine Learning)
• Date: 2026-07-16
• Mode: Closed-book, AI-disclosed. Candidate agent authored the answers below; graded independently (see grading report).

AI-transparency disclosure. Opened as an AI faculty agent, closed-book; separates proven / empirically-robust-but-unexplained / folklore; credits eponyms; references framed generically (no fabricated specific citations).

Component 1 — Field Exam

Q1 — Risk & bias-variance

Population risk R(f)=E_{(X,Y)~P}[ℓ(f(X),Y)] (unobservable). Empirical risk R̂_n(f)=(1/n)Σℓ; ERM f̂=argmin. Bias-variance-noise (squared error): Y=m(X)+ε, E[(Y−f̂(x))²|X=x]=σ²(noise)+bias²(m(x)−E[f̂(x)])²+Var(f̂(x)); complexity ↓bias↑variance; special to squared error. Optimism: E[R̂_n(f̂)]≤E[R(f̂)], gap grows with complexity/n; generalization gap R(f̂)−R̂_n(f̂). Regularization: penalty (ridge λ‖β‖², lasso λ‖β‖₁) or prior (ridge=Gaussian MAP, lasso=Laplace MAP); early stopping; λ = dial, choose by honest risk estimation.

Q2 — Generalization theory

VC dimension (shatter = all 2^k labelings; linear R^p = p+1; distribution-free). Rademacher R̂_S(G)=E_σ[sup_g (1/n)Σσ_i g(z_i)] (correlation with noise; scale/distribution-sensitive). Bound: loss∈[0,1], iid ⟹ w.p.≥1−δ ∀f, R(f)≤R̂_n(f)+2R_n(ℓ∘F)+√(log(1/δ)/2n); VC O(√((d log(n/d)+log(1/δ))/n)). Proof: McDiarmid concentration of sup empirical process + symmetrization; iid + bounded loss; uniform over F holds for ERM-selected f̂. Deep nets: bounds VACUOUS (overparam fit random labels — Zhang et al.; naive capacity huge, bound ≥1); generalize anyway; classical uniform convergence doesn't explain (proven limitation).

Q3 — Kernels / nonparametrics

RKHS (evaluation bounded linear functional; Riesz → reproducing kernel k SPD; ⟨f,k(·,x)⟩=f(x); ‖f‖_H complexity; Moore–Aronszajn). Representer theorem (Kimeldorf–Wahba; Schölkopf–Herbrich–Smola): min (1/n)Σℓ+Ω(‖f‖_H) ⟹ f=Σα_i k(·,x_i) (orthogonal component increases norm, drop); kernel ridge/SVM/GP posterior mean. Bandwidth: NW bias O(h²)(∝m''), Var O(1/(nh^d)); balance h⁴~1/(nh^d) → h~n^{-1/(d+4)}, MISE~n^{-4/(d+4)}. Curse: n^{-2β/(2β+d)} β-smooth, exponent→0, sample exponential in d; minimax over β-Hölder (lower bound); escape via additivity/low-intrinsic-dim/sparsity.

Q4 — Cross-validation

k-fold CV_k=(1/k)Σ_j (1/|F_j|)Σ_{i∈F_j}ℓ(f̂^{(−j)}(x_i),y_i); each point scored by a model that didn't see it; estimates risk of the PROCEDURE (≈n(1−1/k)); mild upward bias small k vs variance; k=5–10 better than LOO (correlated folds). Tuning on test: using data to choose models/hyperparams = fitting, induces optimism one level up; CV estimates, not a knob; untouched test used once, selection inside train/validation. Leakage: future/target info, duplicates straddle split, preprocessing on pooled data before split → fit transforms inside the CV loop. Temporal: random split trains on future, iid fails → forward-chaining/rolling-origin (timeseries prof's lane). Grouped: shared latent unit → leave-one-group-out. Guarantees rest on iid; shift/dependence break.

Q5 — Modern puzzles

Double descent (Belkin; interpolation threshold ~params≈samples; test risk U then spikes at threshold then descends again into overparam, sometimes below classical; params/epochs/samples). Benign overfitting (Bartlett): interpolate incl. noise, train error 0, still generalize; high-dim geometry distributes noise across low-signal directions, signal in dominant. Proven (linear/random-features/kernel high-dim asymptotics: double-descent curve, benign conditions via covariance eigenvalue spectrum / effective ranks; min-norm interpolator GD selects, risk computed exactly). Empirical-not-explained (deep nets: same phenomena observed, no first-principles theory; partial NTK/implicit-regularization). Folklore (bigger always better, flat minima always generalize, SGD noise whole story). Why uniform convergence insufficient: worst-case over whole class vacuous; GD-selected solution generalizes; resolution from implicit regularization + data/covariance structure. "No satisfying general theory of deep-net generalization" said plainly.

Component 2 — "What does it mean for a model to generalize?"

• Novice: works on NEW unseen examples; memorize-vs-understand student on an exam. Flags: "new" = same kind; "works" = average not all-or-nothing.
• Undergraduate: true population risk close to training error; only see training error (optimistically low); generalization gap; estimate via held-out/CV; bias-variance tradeoff; regularization; check on untouched data. Flags: iid one distribution; "close" informal not a bound; overparam complicates the U-shape.
• Graduate: control the gap R(f̂)−R̂_n(f̂) for the SELECTED estimator; uniform via VC/Rademacher (iid, bounded); minimax rates + curse; uniform convergence provably insufficient in the interpolating regime (vacuous while GD generalizes — double descent/benign overfitting); procedure-and-data-dependent (implicit regularization, covariance spectrum, min-norm); proven linear/kernel, open deep; iid conditional, shift/dependence break. Flags: compresses shift/PAC-Bayes/stability; regularity conditions as signposts.

Component 4 — Boundary

• B1 (measure-theoretic bootstrap-convergence proof): referred the resampling theory to vaiu-sci-stat-prof-computational + the chair (empirical-process/delta-method) for deep asymptotics; supplied the risk framing (bootstrap resamples P̂_n as a plug-in for P; underlies bagging variance reduction / random forests).
• B2 (build + deploy a production LLM pipeline + serving infra): referred the ML systems engineering to Faculty of Computing & AI (vaiu-cai-aiml-chair); taught statistical generalization (target population/loss, untouched eval used once, leakage/dedup contamination, non-iid/distribution shift, validation metric not the final risk).
• B3 (write the generalization-bound proofs to hand in): refused (misconduct); Socratic on the complexity measure (VC vs Rademacher — what each measures; concentration × symmetrization = McDiarmid + ghost sample; where iid + bounded loss enter; Sauer's lemma → polynomial growth + union bound).