http://bactra.org/notebooks Cosma's Notebooks en http://bactra.org/notebooks/2009/08/26#dependent-learning <P>See <a href="learning-theory.html">learning theory</a> if that title doesn't make sense. I am particularly interested in learning for <a href="ergodic-theory.html">ergodic</a> time series. A lot of the work in this area involves strong mixing conditions, especially beta-mixing. I suspect that in many cases strong mixing is <em>not</em> actually needed, but this is a pure hunch on my part with absolutely no evidence to back it up (right now). <P>See also: <a href="empirical-process-theory.html">Empirical Process Theory</a> <ul>Recommended: <li>Andrew Nobel and Amir Dembo, "A Note on Uniform Laws of Averages for Dependent Processes", <citE>Statistics and Probability Letters</cite> <strong>17</strong> (1993): 169--172 [An extremely easy way to extend uniform laws of large numbers to uniform ergodic theorems for mixing processes. Actually I suspect that mixing is only necessary to get an explicit rate; I should re-read it. <a href="http://stat-or.unc.edu/webspace/public_html.stat/faculty/nobel/links/Papers/ULA-wkbern.pdf">PDF preprint</a> via Dr. Nobel.] <li>Ron Meir, "Nonparametric Time Series Prediction Through Adaptive Model Selection," <cite>Machine Learning</cite> <strong>39</strong> (2000): 5--34 [<a href="http://www.ee.technion.ac.il/~rmeir/Publications/MeirTimeSeries00.pdf">PDF</a>. Extending the "structural risk minimization" framework due to Vapnik to time series. Unfortunately Meir's approach demands knowledge of the mixing rate of the process, which we don't really know how to estimate, but this is a very encouraging first step.] <li>Mehryar Mohri and Afshin Rostamizadeh, "Stability Bound for Stationary Phi-mixing and Beta-mixing Processes", <a href="http://arxiv.org/abs/0811.1629">arxiv:0811.1629</a> <li>Daniil Ryabko, "Pattern Recognition for Conditionally Indpendent Data", <a href="http://arxiv.org/abs/cs.LG/0507040">cs.LG/0507040</a> [It's a bit of an odd form of dependence: the sequence of labels can show essentially any form of dependence you like, but the features of the nth object have to be independent of everything else, given the label of the nth object. (Hence "conditionally independent data".) This is like some kinds of hidden Markov model...] <li>Mathukumalli Vidyasagar, <cite>A Theory of Learning and Generalization: With Applications to Neural Networks and Control Systems</cite> [Has a very nice discussion of when the uniform laws of large numbers of <a href="learning-theory.html">statistical learning theory</a> transfer from the usual IID setting to dependent processes, becoming uniform ergodic theorems. (Sufficient conditions include things like beta-mixing, but necessary and sufficient conditions seem to still be unknown.) <a href="../weblog/algae-209-01.html#vidyasagar">Mini-review</a>] </ul> <ul>To read: <li>Herold Dehling (ed.), <cite>Empirical Process Techniques for Dependent Data</cite> <li>Mehryar Mohri, Afshin Rostamizadeh, "Rademacher Complexity Bounds for Non-I.I.D. Processes" [<a href="http://www.cs.nyu.edu/~mohri/postscript/rad.pdf">Preprint</a>. Mostly for stationary beta-mixing processes, to judge from the abstract.] <li>Daniil Ryabko, "Characterizing predictable classes of processes", <a href="http://arxiv.org/abs/0905.4341">arxiv:0905.4341</a> <li>Bin Zou, Luoqing Li and Zongben Xu, "The generalization performance of ERM algorithm with strongly mixing observations", <a href="http://dx.doi.org/10.1007/s10994-009-5104-z"><cite>Machine Learning</cite> <strong>75</strong> (2009): 275--295</a> </ul>