Model Selection

(Reader, please make your own suitably awful pun about the different senses of "model selection" here, as a discouragement to those finding this page through prurient searching. Thank you.)

In statistics and machine learning, "model selection" is the problem of picking among different mathematical models which all purport to describe the same data set. This notebook will not (for now) give advice on it; as usual, it's more of a place to organize my thoughts and references...

Classification of approaches to model selection (probably not really exhaustive but I can't think of others, right now):

Direct optimization of some measure of goodness of fit or risk on training data.: Seems implicit in a lot of work which points to marginal improvements in "the proportion of variance explained", mis-classification rates, "perplexity", etc. Often, also, a recipe for over-fitting and chasing snarks. What's wanted is (almost always) some way of measuring the ability to generalize to new data, and in-sample performance is a biased estimate of this. Still, with enough data, if the gods of ergodicity are kind, in-sample performance is representative of generalization performance, so perhaps this will work asymptotically, though in many cases the researcher will never even glimpse Asymptopia across the Jordan.
Optimize fit with model-dependent penalty: Add on a term to each model which supposed indicates its ability to over-fit. (Adjusted R^2, AIC, BIC, ..., all do this in terms of the number of parameters.) Sounds reasonable, but I wonder how many actually work better, in practice, than direct optimization. (See Domingos for some depressing evidence on this score.); Classical two-part minimum description length methods were penalties; I don't yet understand one-part MDL.
Penalties which depend on the model class: Measure the capacity of a class of models to over-fit; penalize all models in that class accordingly, regardless of their individual properties. Outstanding example: Vapnik's "structural risk minimization" (provably consistent under some circumstances). Only sporadically coincides with *IC-type penalties based on the number of parameters.
Cross-validation: Estimate the ability to generalize to different data by, in fact, using different data. Maybe the "industry standard" of machine learning. Query, how are we to know how much different data to use?; Query, how are we to cross-validate when we have complex, relational data? That is, I understand how to do it for independent samples, and I even understand how to do it for time series, but I do not understand how to do it for networks, and I don't think I am alone in this. (Well, I understand how to do it for Erdos-Renyi networks, because that's back to independent samples...)
The method of sieves: Directly optimize the fit, but within a constrained class of models; relax the constraint as the amount of data grows. If the constraint is relaxed slowly enough, should converge on the truth. (Ordinary parametric inference, within a single model class, is a limiting case where the constraint is relaxed infinitely slowly, and we converge on the pseudo-truth within that class [provided we have a consistent estimator].)
Encompassing models: The sampling distribution of any estimator of any model class is a function of the true distribution. If the true model clss has been well-estimated, it should be able to predict what other, wrong model classes will estimate, but not vice versa. In this sense the true model class "encompasses the predictions" of the wrong ones. ("Truth is the criterion both of itself and of error.")
General or covering models: Come up with a single model class which includes all the interesting model classes as special cases; do ordinary estimation within it. Getting a consistent estimator of the additional parameters this introduces is often non-trivial, and interpretability can be a problem.
Model averaging: Don't try to pick the best or correct model; use them all with different weights. Chose the weighting scheme so that if one is best, it will tend to be more and more influential. Often I think the improvement is not so much from using multiple models as from smoothing, since estimates of the single best model are going to be more noisy than estimates of a bunch of models which are all pretty good. (This leads to ensemble methods.)
Adequacy testing: The correct model should be able to encode the data as uniform IID noise. Test whether "residuals", in the appropriate sense, are IID uniform. Reject models which can't hack it. Possibly none of the models on offer is adequate; this, too, is informative. Or: models make specific probabilistic assumptions (IID Gaussian noise, for example); test those. Mis-specification testing.

The machine-learning-ish literature on model selection doesn't seem to ever talk about setting up experiments to select among models; or do I just not read the right papers there? (The statistical literature on experimental design tends to talk about "model discrimination" rather than "model selection".)

Leo Breiman, "Heuristics of Instability and Stabilization in Model Selection," Annals of Statistics 24 (1996): 2350--2383
Gerda Claeskens and Nils Lid Hjort, Model Selection and Model Averaging
Pedro Domingos, "The Role of Occam's Razor in Knowledge Discovery," Data Mining and Knowledge Discovery, 3 (1999) [Online]
Trever Hastie, Robert Tibshirani and Jerome Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction
C. R. Rao, Y. Wu, Sadanori Konishi and Rahul Mukerjee, "On Model Selection", in P. Lahiri (ed.), Model Selection, pp. 1--64 [Thorough review paper, if from a rather old-school statistical-theory perspective. The rest of the volume is too Bayesian to be of interest to me. JSTOR]
Aris Spanos, "Curve-Fitting, the Reliability of Inductive Inference and the Error-Statistical Approach" [PDF preprint]
V. N. (=Vladimir Naumovich) Vapnik, The Nature of Statistical Learning Theory [Review: A Useful Biased Estimator]
Quang H. Vuong, "Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses", Econometrica 57 (1989): 307--333

Sylvain Arlot
- "V-fold cross-validation improved: V-fold penalization", arxiv:0802.0566 [Seeing cross-validation as a penalization method, and improving it accordingly by strengthening the penalty term]
- "Model selection by resampling penalization", arxiv:0906.3124 = Electronic Journal of Statistics 3 (2009): 557--624
A. C. Atkinson and A. N. Donev, Optimum Experimental Design [Review]
Leo Breiman and Philip Spector, "Submodel Selection and Evaluation in Regression: The X-Random Case", International Statistical Review 60 (1992): 291--319 [JSTOR]
Prabir Burman, Edmond Chow and Deborah Nolan, "A cross-validatory method for dependent data", Biometrika 81 (1994): 351--358 [JSTOR]
Patrick S. Carmack, William R. Schucany, Jeffrey S. Spence, Richard F. Gunst, Qihua Lin and Robert W. Haley, "Far Casting Cross Validation" [Leave-one-out CV, with a constant-radius window skipped around each hold-out point as well; this is designed to deal with correlations in time or in space. PDF preprint]
George Casella and Guido Consonni, "Reconciling Model Selection and Prediction", arxiv:0903.3620 ["It is known that there is a dichotomy in the performance of model selectors. Those that are consistent (having the "oracle property") do not achieve the asymptotic minimax rate for prediction error. We look at this phenomenon closely, and argue that the set of parameters on which this dichotomy occurs is extreme, even pathological, and should not be considered when evaluating model selectors. We characterize this set, and show that, when such parameters are dismissed from consideration, consistency and asymptotic minimaxity can be attained simultaneously." Comment: I agree; they show that you need a truly bizarre sequence of local alternatives to get this behavior.]
Nicolo Cesa-Bianchi and Gabor Lugosi, Prediction, Learning, and Games [Mini-review. For avoiding model selection in favor of adaptively-weighted combinations of models.]
Snigdhansu Chatterjee, Nitai D. Mukhopadhyay, "Risk and resampling under model uncertainty", arxiv:0805.3244 [an interesting approach to model averaging with provably good frequentist properties, via bootstrapping --- for a trivial linear-Gaussian problem; not clear to me how to generalize]
Bruce E. Hansen, "Challenges for Econometric Model Selection", Econometric Theory 21 (2005): 60--68 ["Standard econometric model selection methods are based on four fundamental errors in approach: parametric vision, the assumption of a true [data-generating process], evaluation based on fit, and ignoring the impact of model uncertainty on inference. Instead, econometric model selection methods should be based on a semiparametric vision, models should be viewed as approximations, models should be evaluated based on their purpose, and model uncertainty should be incorporated into inference methods." PDF]
Marcus Hutter, "The Loss Rank Principle for Model Selection", math.ST/0702804 [This is a simplified form of Deborah Mayo's "severity".]
Pascal Lavergne and Quang H. Vuong, "Nonparametric Selection of Regressors: The Nonnested Case", Econometrica 64 (1996): 207--219 [Picking which variables belong in a regression, by looking at the error of non-parametric kernel regressions. JSTOR]
Pascal Massart, Concentration Inequalities and Model Selection [Using empirical process theory to get finite-sample, i.e., non-asymptotic, risk bounds for various forms of model selection. Available for free as a large PDF preprint.]
Charles Mitchell and Sara van de Geer, "General Oracle Inequalities for Model Selection", Electronic Journal of Statistics 3 (2009): 176--204 [Analyzes a data-set splitting scheme (like cross-validation with only one "fold")]
Jeffrey S. Racine
- "Feasible Cross-Validatory Model Selection for General Stationary Processes", Journal of Applied Econometrics 12 (1997): 169--179 [JSTOR. This is closely related to (maybe algebraically just a special case of?) the familiar trick from splines of writing the CV criterion in terms of the hat/influence/projection matrix.]
- "Consistent cross-validatory model-selection for dependent data: hv-block cross-validation", Journal of Econometrics 99 (2000): 39--61
David J. Spiegelhalter, Nicola G. Best, Bradley P. Carlin and Angelika van der Linde, "Bayesian Measures of Model Complexity and Fit", Journal of the Royal Statistical Society B 64 (2002): 583--639 [PDF reprint]
Ryan J. Tibshirani and Robert Tibshirani, "A bias correction for the minimum error rate in cross-validation", Annals of Applied Statistics 3 (2009): 822--829 = arxiv:0908.2904
Sara van de Geer, Empirical Process Theory in M-Estimation
Mark J. van der Laan and Sandrine Dudoit, "Unified Cross-Validation Methodology for Selection Among Estimators and a General Cross-Validated Adaptive Epsilon-Net Estimator: Finite Sample Oracle Inequalities and Examples" [PDF working paper, i.e., a 100-page tome. The first part proves that multi-fold cross-validation and the like will work for selecting the best estimator out of a finite set of estimators (provided the loss function is nicely bounded and the data are IID). The second part ingeniously turns this into a complete estimation procedure, by effectively creating a discrete sieve and then using CV to say which part of the sieve to use. This is a very cool set of results, but (1) the limitations to bounded loss functions make me nervous, and (2) the formulas appearing in the finite-sample and even asymptotic bounds are ugly. On the other hand, they have finite-sample bounds! — I wonder if the bounded-and-IID restrictions could be lifted using the techniques in Jiang's "On Uniform Deviation Bounds" (link and description under Learning Theory), or those in Dedecker et al.'s Weak Dependence.]
Aad W. van der Vaart, Sandrine Dudoit and Mark J. van der Laan, "Oracle inequalities for multi-fold cross validation", Statistics and Decisions 24 (2006): 351--371 [Streamlined and improved versions of the key results from the van der Laan/Dudoit tome. Thanks to Prof. van der Vaart for a reprint]

Sylvain Arlot, "Suboptimality of penalties proportional to the dimension for model selection in heteroscedastic regression", arxiv:0812.3141
Sylvain Arlot and Pascal Massart, "Data-driven Calibration of Penalties for Least-Squares Regression", Journal of Machine Learning Research 10 (2009): 245--279
Maria Maddalena Barbieri and James O. Berger, "Optimal Predictive Model Selection", math.ST/0406464 = Annals of Statistics 32 (2004): 870--897 [Unfortunately, Bayesian]
Andrew Barron, Lucien Birgé, and Pascal Massart, "Risk bounds for model selection via penalization", Probability Theory and Related Fields 113 (1999): 301--413
Lucien Birgé
- "The Brouwer Lecture 2005: Statistical estimation with model selection", math.ST/0605187
- "Model selection for Poisson processes", math/0609549
Lucien Birgŕ and Pascal Massart
- "From model selection to adaptive estimation", pp. 55--87 in Pollard, Torgersen and Yang (eds.), Fetschrift for Lucien Le Cam: Research Papers in Probability and Statistics (1997)
Borowiak, Model Discrimination for Nonlinear Regression Models
P. Burman, "A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods", Biometrika 76 (1989): 503--514
Alain Celisse, "Model selection in density estimation via cross-validation", arxiv:0811.0802
A. E. Clark and C. G. Troskie, "Time Series and Model Selection", Communications in Statistics: Simulation and computing 37 (2008): 766--771 [Simulation study of the accuracy of different information criteria]
Kevin A. Clarke, "A Simple Distribution-Free Test for Nonnested Hypotheses" [PDF preprint]
Guilhem Coq, Olivier Alata, Marc Arnaudon and Christian Olivier, "An improved method for model selection based on Information Criteria", math.ST/0702540
Pedro Domingos
- "Process-Oriented Estimation of Generalization Error" [PDF]
- "A Process-Oriented Heuristic for Model Selection" [PDF]
Sandrine Dudoit and Mark J. van der Laan, "Asymptotics of Cross-Validated Risk Estimation in Estimator Selection and Performance Assessment", Statistical Methodology 2 (2005): 131--154 [preprint]
Hugo Jair Escalante, Manuel Montes, Luis Enrique Sucar, "Particle Swarm Model Selection", Journal of Machine Learning Research 10 (2009): 405--440
Jianqing Fan and Runze Li, "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties", Journal of the American Statistical Association 96 (2001): 1348--1360 [PDF reprint via Prof. Fan]
Magalie Fromont, "Model selection by bootstrap penalization for classification", Machine Learning 66 (2007): 165--207
Christophe Giraud, "Estimation of Gaussian graphs by model selection", arxiv:0710.2044
Alexander Goldenshluger and Eitan Greenshtein, "Asymptotically minimax regret procedures in regression model selection and the magnitude of the dimension penalty", Annals of Statistics 28 (2000): 1620--1637 [Hmmm. Not sure how relevant this will be to anything I'd need to do, given the assumptions they load on. Via Kevin Kelly.]
Christian Gourieroux and Alain Monfort, "Testing, Encompassing, and Simulating Dynamic Econometric Models", Econometric Theory 11 (1995): 195--228 [JSTOR]
Michael Kearns and Dana Ron, "Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation," Neural Computation 11 (1999): 1427--1453
Nicholas M. Kiefer and Hwan-Sik Choi, "Robust Model Selection in Dynamic Models with an Application to Comparing Predictive Accuracy" [SSRN]
Sadanori Konishi and Genshiro Kitagawa, "Asymptotic theory for information crteria in model selection --- functional approach," Journal of Statistical Planning and Inference 114 (2003): 45--61
Hannes Leeb, "Conditional Predictive Inference Post Model Selection", Annals of Statistics 37 (2009): 2838--2876 = arxiv:0908.3615 [I heard Leeb give a talk on this, but I should read the paper]
Hannes Leeb and Benedikt M. Poetscher
- "Can One Estimate The Unconditional Distribution of Post-Model-Selection Estimators?", arxiv:0704.1584 [They claim the answer is "No".]
- "Model Selection and Inference: Facts and Fiction", Econometric Theory 21 (2005): 21--59 [PDF reprint]
F. Liang and A. Barron, "Exact Minimax Strategies for Predictive Density Estimation, Data Compression, and Model Selection", IEEE Transactions on Information Theory 50 (2004): 2708--2726
Abraham Meidan and Boris Levin, "Choosing from Competing Theories in Computerised Learning", Minds and Machines 12 (2002): 119--129
Nicolai Meinshausen and Peter Buehlmann, "Stability Selection", arxiv:0809.2932 ["Estimation of structure, such as in graphical modeling, cluster analysis or variable selection, is notoriously difficult, especially for high-dimensional data. We introduce the new method of stability selection."]
Grayham E. Mizon and Massimiliano Marcellino (eds.), Progressive Modelling: Non-nested Testing and Encompassing [Blurb, table of contents]
Ali Mohammad-Djafari, "Model selection for inverse problems: Best choice of basis functions and model order selection," physics/0111020
M. Pavlic and M. J. van der Laan, "Fitting of mixtures with unspecified number of components using cross validation distance estimate", Computational Statistics and Data Analysis 41 (2003): 413--428
Zacharias Psaradakis, Martin Sola, Fabio Spagnolo and Nicola Spagnolo, "Selecting nonlinear time series models using information criteria", Journal of Time Series Analysis 30 (2009): 369--394
Pradeep Ravikumar, Martin J. Wainwright, John D. Lafferty, "High-Dimensional Graphical Model Selection Using $\ell_1$-Regularized Logistic Regression", arxiv:0804.4202
Douglas Rivers and Quang H. Vuong, "Model selection tests for nonlinear dynamic models", The Econometrics Journal 5 (2002): 1--39
Yiyuan She, "Thresholding-based Iterative Selection Procedures for Model Selection and Shrinkage", arxiv:0812.5061
David Shilane, Richard H. Liang and Sandrine Dudoit, "Loss-Based Estimation with Evolutionary Algorithms and Cross-Validation", UC Berkeley Biostatistics Working Paper 227 [Abstract, PDF]
Aris Spanos
- "Statistical Induction, Severe Testing, and Model Validation" [Preprint]
- "Statistical Model Specification vs. Model Selection: Akaike-type Criteria and the Reliability of Inference" [preprint kindly provided by Prof. Spanos]
Tina Toni and Michael P. H. Stumpf
- "Parameter Inference and Model Selection in Signaling Pathway Models", arxiv:0905.4468
- "Simulation-based model selection for dynamical systems in systems and population biology", arxiv:0911.1705
Masayuki Uchida and Nakahiro Yoshida, "Information Criteria in Model Selection for Mixing Processes", Statistical Inference for Stochastic Processes 4 (2001): 73--98 ["The emphasis is put on the use of the asymptotic expansion of the distribution of an estimator based on the conditional Kullback-Leibler divergence for stochastic processes. Asymptotic properties of information criteria and their improvement are discussed."]
Tim van Erven, Peter Grunwald and Steven de Rooij, "Catching Up Faster by Switching Sooner: A Prequential Solution to the AIC-BIC Dilemma", arxiv:0807.1005
Geert Verbeke, Geert Molenberghs, Caroline Beunckens, "Formal and Informal Model Selection with Incomplete Data", Statistical Science 23 (2008): 201--218 = arxiv:0808.3587
Zijun Wang, "Finite Sample Performances of the Model Selection Approach in Nonparametric Model Specification for Time Series", Communications in Statistics: Theory and Methods 38 (2009): 2302--2330
Peng Zhau and Bin Yu, "On Model Selection Consistency of Lasso", Journal of Machine Learning Research 7 (2006): 2541--2563

Notebooks

Model Selection