Empirical Process Theory

04 Jan 2010 09:53

(I first used the next few paragraphs as part of a review of Pollard's book of lecture notes. I have no shame about self-plagiarism.)

The simplest sort of empirical process arises when trying to estimate a probability distribution from sample data. The difference between the empirical distribution function F_n(x) and the true distribution function F(x) converges to zero everywhere (by the law of large numbers), and — this is non-trivial — the maximum difference between the empirical and true distribution functions converges to zero, too (by the Glivenko-Cantelli theorem, a uniform law of large numbers). The "empirical process" E_n(x) is the re-scaled difference, n^1/2[F_n(x) - F(x)], and it converges to a Gaussian stochastic process that only depends on the true distribution (by the functional central limit theorem). Empirical process theory is concerned with generalizing this sort of material to other stochastic processes determined by random samples, and indexed by infinite classes (like the real line, or the class of all Borel sets on the line, or some space parameterizing a regression model). The typical objects of concern are proving uniform limit theorems, and with establishing distributional limits. (For instance, one might one want to prove that the errors of all possible regression models in some class will come close to their expected errors, so that maximum-likelihood or least-squares estimation is consistent. [For more on that line of thought, see Sara van de Geer's book.]) This endeavor is closely linked to Vapnik-Chervonenkis-style learning theory, and in fact one can see VC theory as an application of empirical process theory.

As usual, I am most interested in results for dependent data.

Bruce E. Hansen, "The Likelihood Ratio Test Under Nonstandard Conditions: Testing the Markov Switching Model of GNP", Journal of Applied Econometrics 7 (1992): S61--S82 [I very much like the approach of treating the likelihood ratio as an empirical process; why haven't I seen it before? (Also, the state-of-the-art in simulating Gaussian processes must be much better now than what Hansen had in '92, which would make this even more practical.) PDF reprint.]
Pascal Massart, Concentration Inequalities and Model Selection [Using empirical process theory to get finite-sample, i.e., non-asymptotic, risk bounds for various forms of model selection. Available for free as a large PDF preprint.]
David Pollard
- "Asymptotics via Empirical Processes", Statistical Science 4 (1989): 341--354
- Convergence of Stochastic Processes
- Empirical Processes: Theory and Applications [Mini-review]
Sara van de Geer, Empirical Processes in M-Estimation
Mathukumalli Vidyasagar, A Theory of Learning and Generalization: With Applications to Neural Networks and Control Systems [Mini-review]

Radoslaw Adamczak, "A tail inequality for suprema of unbounded empirical processes with applications to Markov chains", arxiv:0709.3110
Herold Dehling (ed.), Empirical Process Techniques for Dependent Data [blurb]
Michael R. Kosorok, Introduction to Empirical Processes and Semiparametric Inference [PDF preprint; blurb]
D. Marinucci, "The Empirical Process for Bivariate Sequences with Long Memory", Statistical Inference for Stochastic Processes 8 (2005): 205--224
Richard Samworth and Oliver Johnson, "The empirical process in Mallows distance, with application to goodness-of-fit tests", math.ST/0504424
Aad W. van der Vaart, Jon A. Wellner
- "Empirical processes indexed by estimated functions", arxiv:0709.1013 ["We consider the convergence of empirical processes indexed by functions that depend on an estimated parameter $\eta$ and give several alternative conditions under which the ``estimated parameter'' $\eta_n$ can be replaced by its natural limit $\eta_0$ uniformly in some other indexing set $\Theta$"]
- Weak Convergence and Empirical Processes: With Applications to Statistics
Bin Yu, "Rates of Convergence for Empirical Processes of Stationary Mixing Sequences," Annals of Probability 22 (1994): 94--116