Forecasting Non-Stationary Processes

23 Jul 2010 09:19

---

Some non-stationary processes are in fact easy to forecast: periodic ones, for example, are strictly speaking not stationary. An ergodic Markov chain started far from its invariant distribution is also non-stationary, but easy to predict (it will approach the stationary distribution). Both of these cases are conditionally stationary, which I think is all that's really needed.

What's more interesting is the problem of so to speak really non-stationary processes. It's hard to imagine that there is any way to truly predict an arbitrary non-stationary process. (Basically: as soon as you think you have established a trend-line, the Adversary can always reverse the trend, without creating any problems of consistency with earlier data.) If you can constrain the class of allowable non-stationary processes, however, then something might be possible. Alternately, one might lower expectations, not to actually predicting well, but to predicting with low regret.

I actually have an Idea about using model averaging here, but need to find the time to work on it.

See also: Ensemble Methods in Machine Learning; Time Series; Universal Prediction

Recommended (very misc):
• S. Caires and J. A. Ferreira, "On the Non-parametric Prediction of Conditionally Stationary Sequences", Statistical Inference for Stochastic Processes 8 (2005): 151--184
• R. Dahlhaus, "Fitting Time Series Models to Nonstationary Processes", Annals of Statistics 25 (1997): 1--37
• Mark Herbster and Manfred K. Warmuth, "Tracking the Best Expert", Machine Learning 32 (1998): 151--178 [PS version via Dr. Herbster]
• Elad Hazan and Satyen Kale, "Extracting certainty from uncertainty: regret bounded by variation in costs", Machine Learning 80 (2010): 165--188
• Jeremy Zico Kolter and Marcus A. Maloof
  • "Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts", Journal of Machine Learning Research 8 (2007): 2755--2790
  • "Using Additive Expert Ensembles to Cope with Concept Drift", ICML 2005 [PDF reprint via Kolter]
• Maxim Raginsky, Roummel F. Marcia, Jorge Silva and Rebecca M. Willett
  • "Sequential Probability Assignment via Online Convex Programming Using Exponential Families" [ISIT 2009; PDF]
  • "Sequential anomaly detection in the presence of noise and limited feedback", arxiv:0911.2904
• Kyupil Yeon, Moon Sup Song, Yongdai Kim, Hosik Choi, Cheolwoo Park, "Model averaging via penalized regression for tracking concept drift", Journal of Computational and Graphical Statistics online before print (2010)

To read:
• István Berkes, Lajos Horváth and Shiqing Ling, "Estimation in nonstationary random coefficient autoregressive models", Journal of Time Series Analysis 30 (2009): 395--416 ["the unit root problem does not exist in the RCA model"!]
• Alexey Chernov, Vladimir Vovk, "Prediction with Advice of Unknown Number of Experts", arxiv:1006.0475
• Michael P. Clements and David F. Hendry, Forecasting Non-Stationary Economic Time Series
• Rainer Dahlhaus and Wolfgang Polonik, "Empirical spectral processes for locally stationary time series", Bernoulli 15 (2009): 1--39, arxiv:902.1448
• Ching-Kang Ing, Jin-Lung Lin, Shu-Hui Yu, "Toward optimal multistep forecasts in non-stationary autoregressions", Bernoulli 15 (2009): 402--437 = arxiv:0906.2266 ["Optimal" assuming that you know you are facing a linear AR model.]
• C. T. Jose, B. Ismail, S. Jayasekhar, "Trend, Growth Rate, and Change Point Analysis: A Data Driven Approach", Communications in Statistics: Simulation and Computation 37 (2008): 498--506
• Yan Karklin and Michael S. Lewicki, "A Hierarchical Bayesian Model for Learning Nonlinear Statistical Regularities in Nonstationary Natural Signals", Neural Computation 17 (2005): 397--423
• Zudi Lu, Dag Johan Steinskog, Dag Tjostheim and Qiwei Yao, "Adaptively Varying-Coefficient Spatiotemporal Models", Journal of the Royal Statistical Society B 71 (2009): 859--880 [PDF preprint]
• Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer and Neil D. Lawrence (eds.), Dataset Shift in Machine Learning
• P. F. Verdes, P. M. Granitto and H. A. Ceccatto, "Overembedding Method for Modeling Nonstationary Systems", Physical Review Letters 96 (2006): 118701

To write:
• CRS + friends, "Adapting to non-stationarity with growing predictor ensembles"