Forecasting Non-Stationary Processes

Some non-stationary processes are in fact easy to forecast: periodic ones, for example, are strictly speaking not stationary. An ergodic Markov chain started far from its invariant distribution is also non-stationary, but easy to predict (it will approach the stationary distribution). Both of these cases are conditionally stationary, which I think is all that's really needed.

What's more interesting is the problem of so to speak really non-stationary processes. It's hard to imagine that there is any way to truly predict an arbitrary non-stationary process. (Basically: as soon as you think you have established a trend-line, the Adversary can always reverse the trend, without creating any problems of consistency with earlier data.) If you can constrain the class of allowable non-stationary processes, however, then something might be possible. Alternately, one might lower expectations, not to actually predicting well, but to predicting with low regret.

I actually have an Idea about using model averaging here, but need to find the time to work on it.

S. Caires and J. A. Ferreira, "On the Non-parametric Prediction of Conditionally Stationary Sequences", Statistical Inference for Stochastic Processes 8 (2005): 151--184
R. Dahlhaus, "Fitting Time Series Models to Nonstationary Processes", Annals of Statistics 25 (1997): 1--37
Mark Herbster and Manfred K. Warmuth, "Tracking the Best Expert", Machine Learning 32 (1998): 151--178 [PS version via Dr. Herbster]
Elad Hazan and Satyen Kale, "Extracting certainty from uncertainty: regret bounded by variation in costs", Machine Learning 80 (2010): 165--188
Jeremy Zico Kolter and Marcus A. Maloof
- "Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts", Journal of Machine Learning Research 8 (2007): 2755--2790
- "Using Additive Expert Ensembles to Cope with Concept Drift", ICML 2005 [PDF reprint via Kolter]
Claire Monteleoni and Tommi S. Jaakkola, "Online Learning of Non-stationary Sequences", pp. 1093--1100 in NIPS 2003 (vol. 16) [Figuring out at what rate to switch between experts]
Maxim Raginsky, Roummel F. Marcia, Jorge Silva and Rebecca M. Willett
- "Sequential Probability Assignment via Online Convex Programming Using Exponential Families" [ISIT 2009; PDF]
- "Sequential anomaly detection in the presence of noise and limited feedback", arxiv:0911.2904
Kyupil Yeon, Moon Sup Song, Yongdai Kim, Hosik Choi, Cheolwoo Park, "Model averaging via penalized regression for tracking concept drift", Journal of Computational and Graphical Statistics online before print (2010)

CRS, Abigail Z. Jacobs, Kristina Lisa Klinkner and Aaron Clauset, "Adapting to Non-stationarity with Growing Expert Ensembles", arxiv:1103.0949

István Berkes, Lajos Horváth and Shiqing Ling, "Estimation in nonstationary random coefficient autoregressive models", Journal of Time Series Analysis 30 (2009): 395--416 ["the unit root problem does not exist in the RCA model"!]
Satish T. S. Bukkapatnam and Changqing Cheng, "Forecasting the evolution of nonlinear and nonstationary systems using recurrence-based local Gaussian process models", Physical Review E 82 (2010): 056206
Alexey Chernov, Vladimir Vovk, "Prediction with Advice of Unknown Number of Experts", arxiv:1006.0475
Michael P. Clements and David F. Hendry, Forecasting Non-Stationary Economic Time Series
Rainer Dahlhaus and Wolfgang Polonik, "Empirical spectral processes for locally stationary time series", Bernoulli 15 (2009): 1--39, arxiv:902.1448
Ching-Kang Ing, Jin-Lung Lin, Shu-Hui Yu, "Toward optimal multistep forecasts in non-stationary autoregressions", Bernoulli 15 (2009): 402--437 = arxiv:0906.2266 ["Optimal" assuming that you know you are facing a linear AR model.]
C. T. Jose, B. Ismail, S. Jayasekhar, "Trend, Growth Rate, and Change Point Analysis: A Data Driven Approach", Communications in Statistics: Simulation and Computation 37 (2008): 498--506
Yan Karklin and Michael S. Lewicki, "A Hierarchical Bayesian Model for Learning Nonlinear Statistical Regularities in Nonstationary Natural Signals", Neural Computation 17 (2005): 397--423
Zudi Lu, Dag Johan Steinskog, Dag Tjostheim and Qiwei Yao, "Adaptively Varying-Coefficient Spatiotemporal Models", Journal of the Royal Statistical Society B 71 (2009): 859--880 [PDF preprint]
Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer and Neil D. Lawrence (eds.), Dataset Shift in Machine Learning
Joshua W. Robinson, Alexander J. Hartemink, "Learning Non-Stationary Dynamic Bayesian Networks", Journal of Machine Learning Research 11 (2010): 3647--3680
P. F. Verdes, P. M. Granitto and H. A. Ceccatto, "Overembedding Method for Modeling Nonstationary Systems", Physical Review Letters 96 (2006): 118701
Ou Zhao, Michael Woodroofe, "Estimating a monotone trend", arxiv:0812.3188

CRS + co-conspirators to be named later, "This Time Is Different"

Notebooks

Forecasting Non-Stationary Processes