Sparse Deep Gaussian Process Approximation and Application of Dynamic System Identification

This thesis introduces several new Gaussian process approximations based on the sparse spectrum approximation and the sparse Nyström approximation. We use several new variational Bayesian distribution techniques to extend the sparse spectrum approximation and combine these new extensions with the other sparse Nyström approach in a non-obvious way. We further show an application in dimensionality reduction for these newly derived approximations. We show connections of noisy greedy methods and the Support vector regression in the least square setting to Gaussian processes. Moreover, the main contribution of this work is the derivation of new sparse deep Gaussian process models modeling time-series data for the dynamic system-identification task. We show in our experiments comparable performance to state of the art methods. Our experiments in this system identification task compare a lot of different approaches, including different Gaussian processes approximations, LSTMs, recurrent neural networks, Support vector regression, greedy selection methods etc. . These methods are derived by different approximation techniques regarding the model definition and also highlight a lot of different aspects regarding different optimization techniques. The theoretical analysis in the PAC-Bayesian framework gives theoretical insights in the statistical learning theory of our sparse deep Gaussian process models. We show a consistency result, compare the raw model against the variational distribution approximation techniques, extend these results to some specific parameter cases and validate some of the results in the experimental section.