http://bactra.org/notebooks Cosma's Notebooks en http://bactra.org/notebooks/2012/04/18#bootstrap <P>Bootstrapping is a way of figuring out the properties of statistical estimators (and other procedures, like hypothesis tests) by simulation. What we would really like to know his how different our answers could have been, if we re-ran our experiment. We can't actually do this, but we can fit a model to our data and simulate from it, and see what answer we'd get from the simulations. We can even do this from exceedingly general non-parametric estimates, like re-sampling the original data. This is a brilliant idea, and my default way of handling the uncertainty of estimation in complex models or with complex systems. But <a href="../weblog/652.html">having just written 3500 words on this for a magazine</a>, I feel absolutely no inclination to explain myself further. <P>I most interested in resampling techniques for dependent data, and would be ecstatic if I could figure out a non-parametric bootstrap for <a href="network-data-analysis.html">networks</a>. &mdash; Presumably <a href="universal-prediction.html">universal prediction algorithms</a> could be used for this purpose? <P>See also: <a href="cross-validation.html">Cross-Validation</a>; <a href="statistics.html">Statistics</a> <ul>Recommended, big picture: <li>A. C. Davison and D. V. Hinkley, <cite>Bootstrap Methods and their Applications</cite> <li>Bradley Efron <uL> <li>"Bootstrap Methods: Another Look at the Jackknife", <a href="http://projecteuclid.org/euclid.aos/1176344552"><cite>Annals of Statistics</cite> <strong>7</strong> (1979): 1--26</a> [The original paper; staggeringly understandable] <li><cite>The Bootstrap, the Jackknife, and Other Resampling Plans</cite> [1982 notes volume] </ul> </ul> <ul>Recommended, close-ups: <li>Peter B&uuml;hlmann, "Sieve Bootstrap with Variable Length Markov Chains for Stationary Categorical Time Series", <cite>Journal of the American Statistical Association</cite> <strong>97</strong> (2002): 443--456 [<a href="http://e-collection.ethbib.ethz.ch/view/eth:24072">PDF preprint</a>] <li>Paul Doukhan, Silika Prohl, and Christian Y. Robert, "Subsampling weakly dependent time series and application to extremes", <a href="http://arxiv.org/abs/1009.0805">arxiv:1009.0805</a> [Thanks to Dr. Prohl for a pre-pre-print] <li>A. C. Field and A. H. Welsh, "Bootstrapping clustered data", <a href="http://dx.doi.org/10.1111/j.1467-9868.2007.00593.x"><cite>Journal of the Royal Statistical Society</cite> B <strong>69</strong> (2007): 369--390</a> <li>Silvia Goncalves and Halbert White, "Maximum likelihood and the bootstrap for nonlinear dynamic models", <a href="http://dx.doi.org/10.1016/S0304-4076(03)00204-5"><cite>Journal of Econometrics</cite> <strong>119</strong> (2004): 199--219</a> <li>Hans R. K&uuml;nsch, "The Jackknife and the Bootstrap for General Stationary Observations", <a href="http://projecteuclid.org/euclid.aos/1176347265"><cite>Annals of Statistics</cite> <strong>17</strong> (1989): 1217--1241</a> <li>S. N. Lahiri, <cite>Resampling Methods for Dependent Data</cite> [<a href="../weblog/algae-2010-05.html#lahiri">Mini-review</a>] <li>Elizaveta Levina and Peter J. Bickel, "Texture synthesis and nonparametric resampling of random fields", <a href="http://projecteuclid.org/euclid.aos/1162567632"><cite>Annals of Statistics</cite> <strong>34</strong> (2006): 1751--1773</a> <li>Peter Mccullagh, "Resampling and exchangeable arrays", <a href="http://projecteuclid.org/euclid.bj/1081788029"><cite>Bernoulli</cite> <strong>6</strong> (2000): 285--301</a> [Well, half-recommended. Everything he says here is right, but I think one could construct an exactly parallel argument to show that resampling could not get correct standard errors for the mean of a stationary sequence; and of course it can't if you insist on resampling dependent data as though it were independent. See the papers by Owen and by Owen and Eckles.] <li>Art B. Owen, "The pigeonhole bootstrap", <a href="http://projecteuclid.org/euclid.aoas/1196438024"><cite>Annals of Applied Statistics</cite> <strong>1</strong> (2007): 386--411</a> <li>Art B. Owen and Dean G. Eckles, "Bootstrapping data arrays of arbitrary order", <a href="http://arxiv.org/abs/1106.2125">arxiv:1106.2125</a> </ul> <ul>Modesty forbids me to recommend: <li>CRS, "The Bootstrap", <a href="http://www.americanscientist.org/issues/pub/2010/3/the-bootstrap/1"><cite>American Scientist</cite> <strong>98</strong> (2010): 186--190</a> [<a href="../weblog/652.html">Self-commentary</a>] </ul> <ul>To read: <li>Sylvain Arlot, Gilles Blanchard, and Etienne Roquain, "Some nonasymptotic results on resampling in high dimension, I: Confidence regions", <a href="http://projecteuclid.org/euclid.aos/1262271609"><cite>Annals of Statistics</cite> <strong>38</strong> (2010): 51--82</a> <li>Peter B&uuml;hlmann, "Bootstraps for Time Series", <a href="http://projecteuclid.org/euclid.ss/1023798998"><cite>Statistical Science</cite> <strong>17</strong> (2002): 52--72</a> <li>Snigdhansu Chatterjee and Arup Bose, "Generalized bootstrap for estimating equations", <a href="http://arxiv.org/abs/math.ST/0504515">math.ST/0504515</a> = <a href="http://dx.doi.org/10%2E1214/009053604000000904"><cite>Annals of Statistics</cite> <strong>33</strong> (2005): 414--436</a> <li>Guang Cheng and Jianhua Z. Huang, "Bootstrap consistency for general semiparametric M-estimation", <a href="http://projecteuclid.org/euclid.aos/1279638543"><cite>Annals of Statistics</cite> <strong>38</strong> (2010): 2884--2915</a> <li>Michael R. Chernick, <cite>Bootstrap Methods: A Practitioner's Guide</cite> <li>Andreas Christmann, Matias Salibian-Barrera, Stefan Van Aelst, "On the stability of bootstrap estimators", <a href="http://arxiv.org/abs/1111.1876">arxiv:1111.1876</a> <li>Herold Dehling, Martin Wendler, "Central Limit Theorem and the Bootstrap for U-Statistics of Strongly Mixing Data", <a href="http://arxiv.org/abs/0811.1888">arxiv:0811.1888</a> <li>Mathias Drton and Benjamin Williams, "Quantifying the failure of bootstrap likelihood ratio tests", <a href="http://dx.doi.org/10.1093/biomet/asr033 "><cite>Biometrika</cite> <strong>98</strong> (2011): 919--934</a> <li>Efron and Tibshirani, <cite>An Introduction to the Bootstrap</cite> <li>Yanqin Fan, Qi Li and Insik Min, "A Nonparametric Bootstrap Test of Conditional Distributions", <a href="http://dx.doi.org/10.1017/S0266466606060294"><cite>Econometric Theoy</cite> <strong>22</strong> (2006): 587--613</a> <li>Cheng-Der Fuh and Inchi Hu, "Estimation in hidden Markov models via efficient importance sampling", <cite>Bernoulli</cite> <strong>13</strong> (2007): 492--513, <a href="http://arxiv.org/abs/0708.4152">arxiv:0708.4152</a> <li>Jurgen Franke, Jens-Peter Kreiss and Enno Mammen, "Bootstrap of Kernel Smoothing in Nonlinear Time Series", <a href="http://projecteuclid.org/euclid.bj/1078951087"><cite>Bernoulli</cite> <strong>8</strong> (2002): 1--37</a> <li>Axel Gandy, Patrick Rubin-Delanchy, "An algorithm to compute the power of Monte Carlo tests with guaranteed precision", <a href="http://arxiv.org/abs/1110.1248">arxiv:1110.1248</a> <li>Philip Good <ul> <li><cite>Permutation, Parametric, and Bootstrap Tests of Hypotheses</cite> <li><citE>Resampling Methods: A Practical Guide to Data Analysis</cite> </ul> <li>Peter G. Hall, <cite>The Bootstrap and Edgeworth Expansion</cite>, <li>Peter Hall and Hugh Miller, "Bootstrap confidence intervals and hypothesis tests for extrema of parameters", <a href="http://dx.doi.org/10.1093/biomet/asq045"><cite>Biometrika</cite> <Strong>97</strong> (2010): 881--892</a> [E.g., looking at the largest (or smallest) of a bunch of regression coefficients] <li>Eunju Hwang, Dong Wan Shin, "Stationary bootstrapping for non-parametric estimator of nonlinear autoregressive model", <a href="http://dx.doi.org/10.1111/j.1467-9892.2010.00699.x"><cite>Journal of Time Series Analysis</cite> forthcoming (2011)</a> <li>Arnold Janssen and Thorsten Pauls, "How Do Bootstrap and Permutation Tests Work?", <cite>The Annals of Statistics</cite> <strong>31</strong> (2003): 768--806 <li>Ariel Kleiner, Ameet Talwalkar, Purnamrita Sarkar, Michael I. Jordan, "A Scalable Bootstrap for Massive Data", <a href="http://arxiv.org/abs/1112.5016">arxiv:1112.5016</a> <li>Jens-Peter Kreiss, Efstathios Paparoditis, Dimitris N. Politis, "On the range of validity of the autoregressive sieve bootstrap", <cite>Annals of Statistics</cite> <strong>39</strong> (2011): 2103--2130, <a href="http://arxiv.org/abs/12016211">arxiv:12016211</a> <li>S. N. Lahiri, "Edgeworth expansions for studentized statistics under weak dependence", <a href="http://projecteuclid.org/euclid.aos/1262271619"><cite>Annals of Statistics</cite> <strong>38</strong> (2010): 388--434</a> <li>Stephen M. S. Lee and P. Y. Lai, "Improving coverage accuracy of block bootstrap confidence intervals", <a href="http://arxiv.org/abs/0804.4361">arxiv:0804.4361</a> <li>Anne Leucht, "Degenerate U- and V-statistics under weak dependence: Asymptotic theory and bootstrap consistency", <a href="http://dx.doi.org/10.3150/11-BEJ354"><cite>Bernoulli</cite> <strong>18</strong> (2012): 552--585</a> <li>W. S. Lok and Stephen M. S. Lee, "Robustness Diagnosis for Bootstrap Inference", <a href="http://dx.doi.org/"><cite>Journal of Computational and Graphical Statistics</cite> <strong>20</strong> (2011): 448--460</a> <li>Michael H. Neumann, Efstathios Paparoditis, "Goodness-of-fit tests for Markovian time series models: Central limit theory and bootstrap approximations", <cite>Bernoulli</citE> <strong>14</strong> (2008): 14--46, <a href="http://arxiv.org/abs/0803.0835">arxiv:0803.0835</a> <li>Daniel J. Nordman, "A note on the stationary bootstrap's variance", <cite>Annals of Statistics</cite> <strong>37</strong> (2009): 359--370, <a href="http://arxiv.org/abs/0903.0474">arxiv:0903.0474</a> <li>Dimitris N. Politis, "The Impact of Bootstrap Methods on Time Series Analysis", <a href="http://projecteuclid.org/euclid.ss/1063994977"><cite>Statistical Science</cite> <Strong>18</strong> (2003): 219--230</a> <li>Dimitris N. Politis, Joseph P. Romano and Michael Wolf, <cite>Subsampling</cite> <li>Zacharias Psaradakis <ul> <li>"A sieve bootstrap test for stationarity," <a href="http://dx.doi.org/10.1016/S0167-7152(03)00012-9"><cite>Statistics and Probability Letters</cite> <strong>62</strong> (2003): 263--274</a> <li>"Blockwise bootstrap testing for stationarity", <a href="http://dx.doi.org/10.1016/j.spl.2005.09.001"><cite>Statistics and Probability Letters</cite> <strong>76</strong> (2006): 562--570</a> </ul> <li>Joseph P. Romano, Azeem M. Shaikh, "On the Uniform Asymptotic Validity of Subsampling and the Bootstrap", <a href="http://arxiv.org/abs/1204.2762">arxiv:1204.2762</a> <li>Matias Salibian-Barrera, Stefan van Aelst and Gert Willems, "Fast and robust bootstrap", <a href="http://dx.doi.org/10.1007/s10260-007-0048-6"><cite>Statistical Methods and Applications</cite> <strong>17</strong> (2009): 41--71</a> <li>Jun Shao and Dongsheng Tu, <citE>The Jackknife and the Bootstrap</cite> <li>Xiaofeng Shao, "The Dependent Wild Bootstrap", <a href="http://dx.doi.org/10.1198/jasa.2009.tm08744"><cite>Journal of the American Statistical Association</cite> <strong>105</strong> (2010): 218--235</a> <li>Olimjon Sh. Sharipov and Martin Wendler, "Bootstrap for the Sample Mean and for U-Statistics of Stationary Processes", <a href="http://arxiv.org/abs/0911.3083">arxiv:0911.3083</a> <li>Jan Sprenger, "Science without (parametric) models: the case of bootstrap resampling", <a href="http://dx.doi.org/10.1007/s11229-009-9567-z"><cite>Synthese</cite> <strong>180</strong> (2011): 65--76</a> <li>Jos&eacute; Trashorras, Olivier Wintenberger, "Large deviations for bootstrapped empirical measures", <a href="http://arxiv.org/abs/1110.4620">arxiv:1110.4620</a> <li>Lionel Truquet, "On a nonparametric resampling scheme for Markov random fields", <a href="http://dx.doi.org/10.1214/11-EJS644"><cite>Electronic Journal of Statistics</cite> <strong>5</strong> (2011): 1503--1536</a> <li>Herwig Wendt, Patrice Abry and Stephane Jaffard, "Bootstrap for Empirical Multifractal Analysis", <citE>IEEE Signal Processing Magazine</cite> July 2007, pp. 38--48 [+ technical papers by these authors] <li>Abdelhak M. Zoubir and D. Robert Iskander, <citE>Bootstrap Techniques for Signal Processing</cite> </ul>