Выходит 6 номеров в год
ISSN Печать: 2152-5080
ISSN Онлайн: 2152-5099
Indexed in
STABLE LIKELIHOOD COMPUTATION FOR MACHINE LEARNING OF LINEAR DIFFERENTIAL OPERATORS WITH GAUSSIAN PROCESSES
Краткое описание
In many applied sciences, the main aim is to learn the parameters in the operational equations which best fit the observed data. A framework for solving such problems is to employ Gaussian process (GP) emulators which are well-known as nonparametric Bayesian machine learning techniques. GPs are among a class of methods known as kernel machines which can be used to approximate rather complex problems by tuning their hyperparameters. The maximum likelihood estimation (MLE) has widely been used to estimate the parameters of the operators and kernels. However, the MLE-based and Bayesian inference in the standard form are usually involved in setting up a covariance matrix which is generally ill-conditioned. As a result, constructing and inverting the covariance matrix using the standard form will become unstable to learn the parameters in the operational equations. In this paper, we propose a novel approach to tackle these computational complexities and also resolve the ill-conditioning problem by forming the covariance matrix using alternative bases via the Hilbert−Schmidt SVD (HS-SVD) approach. Applying this approach yields a novel matrix factorization of the block-structured covariance matrix which can be implemented stably by isolating the main source of the ill-conditioning. In contrast to standard matrix decompositions which start with a matrix and produce the resulting factors, the HS-SVD is constructed from the Hilbert−Schmidt eigenvalues and eigenvectors without the need to ever form the potentially ill-conditioned matrix. We also provide stable MLE and Bayesian inference to adaptively estimate hyperparameters, and the corresponding operators can then be efficiently predicted at some new points using the proposed HS-SVD bases. The efficiency and stability of the proposed HS-SVD method will be compared with the existing methods by several illustrations of the parametric linear equations, such as ordinary and partial differential equations, and integro-differential and fractional order operators.
-
Bender, E.A., An Introduction to Mathematical Modeling, Mineola, NY: Dover Publications Inc., 1978.
-
de Vries, K., Nikishova, A., Czaja, B., Zavodszky, G., and Hoekstra, A.G., Inverse Uncertanity Quantification of a Cell Model Using a Gaussian Process Metamodel, Int. J. Uncertainty Quantif., 10(4):333-349,2020.
-
Narayan, A., Yan, L., and Zhou, T., Optimal Design for Kernel Interpolation: Applications to Uncertainty Quantification, J. Comput. Phys, 430:1-20, 2021.
-
Qin, T., Chen, Z., Jakeman, J.D., and Xiu, D., Deep Learning of Parameterized Equations with Applications to Uncertanity Quantification, Int. J. Uncertainty Quantif., 11(2):63-82, 2021.
-
Raissi, M., Perdikaris, P., and Em Karniadakis, G., Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations, SIAMJ. Sci. Comput., 40(1):172-198,2018.
-
Raissi, M., Perdikaris, P., and Em Karniadakis, G., Machine Learning of Linear Differential Equations Using Gaussian Processes, J. Comput. Phys., 348(1):683-693, 2017.
-
Mueller, J.L. and Siltanen, S., Linear and Nonlinear Inverse Problems with Practical Applications, Computational Science and Engineering, Vol. 10, Philadelphia: SIAM, 2012.
-
Neto,F.D.M. andNeto, A.J.S., An Introduction to Inverse Problems with Applications, Berlin: Springer-Verlag, 2013.
-
Williams, C.K. and Rasmussen, C.E., Gaussian Processes for Machine Learning, Cambridge, MA: The MIT Press, 2006.
-
Murphy, K.P., Machine Learning: A Probabilistic Perspective, Cambridge, MA: The MIT Press, 2012.
-
Vapnik, V., The Nature of Statistical Learning Theory, Berlin: Springer Science & Business Media, 2013.
-
Guo, M. and Hesthaven, J.S., Reduced Order Modeling for Nonlinear Structural Analysis Using Gaussian Process Regression, Comput. Methods Appl. Mech. Eng., 341(1):807-826,2018.
-
Tikhonov, A., Solution of Incorrectly Formulated Problems and the Regularization Method, Sov. Math. Dokl., 5:1035-1038, 1963.
-
Poggio, T. and Girosi, F., Networks for Approximation and Learning, Proc. IEEE, 78:1481-1497,1990.
-
Fasshauer, G.E., Meshfree Approximation Methods with Matlab, Interdisciplinary Mathematical Sciences, Vol. 6, Singapore: World Scientific Publishing, 2007.
-
Fasshauer, G.E. and McCourt, M., Kernel-Based Approximation Methods Using MATLAB, Interdisciplinary Mathematical Sciences, Singapore: World Scientific Publishing, 2015.
-
McCourt, M. and Fasshauer, G.E., Stable Likelihood Computation for Gaussian Random Fields, Recent Applications of Harmonic Analysis to Function Spaces, Differential Equations, and Data Science. Applied and Numerical Harmonic Analysis, Cham, Swizerland: Birkher, 2015.
-
Kohn, R., Ansley, C.F., and Tharm, D., The Performance of Cross-Validation and Maximum Likelihood Estimators of Spline Smoothing Parameters, J. Am. Stat. Assoc, 86(416):1042-1050,1991.
-
Fornberg, B., Larsson, E., and Flyer, N., Stable Computations with Gaussian Radial Basis Functions, SIAM J. Sci. Comput:., 33(2):869-892,2011.
-
Pazouki, M. and Schaback, R., Bases for Kernel-Based Spaces, J. Comput. Appl. Math., 236(4):575-588, 2011.
-
Quang, M.H., Niyogi, P., and Yao, Y., Mercer's Theorem, Feature Maps, and Smoothing, in Proc. of Int. Conf. on Computational Learning Theory (COLT), 2007.
-
Fasshauer, G.E. and McCourt, M., Stable Evaluation of Gaussian RBF Interpolants, SIAMJ. Sci. Comput., 34(2):737-762, 2012.
-
Gulian, M., Raissi, M., Perdikaris, P., and Em Karniadakis, G., Machine Learning of Space-Fractional Differential Equations, SIAMJ. Sci. Comput., 41(4):2485-2509, 2019.
-
Esmaeilbeigi, M., Chatrabgoun, O., and Cheraghi, M., The Role of Hilbert-Schmidt SVD Basis in Hermite-Birkhoff Interpolation in Fractional Sense, Comput. Appl. Math., 38(82):1-20,2019.
-
Esmaeilbeigi, M., Chatrabgoun, O., and Cheraghi, M., Fractional Hermite Interpolation Using RBFs in High Dimensions over Irregular Domains with Application, J. Comput. Phys., 375:1091-1120, 2018.
-
Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H.P., Design and Analysis of Computer Experiments, Stat. Sci., 4(4):409-435, 1989.
-
Rasmussen, C.E. and Ghahramani, Z., Occam's Razor, in Adv. Neural. Inform. Process. Syst., pp. 294-300, 2001.
-
Aronszajn, N., Theory of Reproducing Kernels, Trans. Am. Math. Soc., 68:337-404,1950.
-
Saitoh, S., Theory of Reproducing Kernels and Its Application, Harlow, UK: Longman, 1988.
-
Berlinet, A. and Thomas-Agnan, C., Reproducing Kernel Hilbert Spaces in Probability and Statistics, Berlin: Springer Science & Business Media, 2011.
-
Datta, B.N., Numerical Linear Algebra and Applications, 2nd ed., Philadelphia: SIAM, 2010.
-
Podlubny, I., Fractional Differential Equations: An Introduction to Fractional Derivatives, Fractional Differential Equations, to Methods of Their Solution and Some of Their Applications, New York: Academic Press, 1998.
-
Esmaeilbeigi Mohsen, Chatrabgoun Omid, Daneshkhah Alireza, Shafa Maryam, On the impact of prior distributions on efficiency of sparse Gaussian process regression, Engineering with Computers, 2022. Crossref