图书馆订阅: Guest
国际不确定性的量化期刊

每年出版 6 

ISSN 打印: 2152-5080

ISSN 在线: 2152-5099

The Impact Factor measures the average number of citations received in a particular year by papers published in the journal during the two preceding years. 2017 Journal Citation Reports (Clarivate Analytics, 2018) IF: 1.7 To calculate the five year Impact Factor, citations are counted in 2017 to the previous five years and divided by the source items published in the previous five years. 2017 Journal Citation Reports (Clarivate Analytics, 2018) 5-Year IF: 1.9 The Immediacy Index is the average number of times an article is cited in the year it is published. The journal Immediacy Index indicates how quickly articles in a journal are cited. Immediacy Index: 0.5 The Eigenfactor score, developed by Jevin West and Carl Bergstrom at the University of Washington, is a rating of the total importance of a scientific journal. Journals are rated according to the number of incoming citations, with citations from highly ranked journals weighted to make a larger contribution to the eigenfactor than those from poorly ranked journals. Eigenfactor: 0.0007 The Journal Citation Indicator (JCI) is a single measurement of the field-normalized citation impact of journals in the Web of Science Core Collection across disciplines. The key words here are that the metric is normalized and cross-disciplinary. JCI: 0.5 SJR: 0.584 SNIP: 0.676 CiteScore™:: 3 H-Index: 25

Indexed in

ON TRANSFER LEARNING OF NEURAL NETWORKS USING BI-FIDELITY DATA FOR UNCERTAINTY PROPAGATION

卷 10, 册 6, 2020, pp. 543-573
DOI: 10.1615/Int.J.UncertaintyQuantification.2020033267
Get accessGet access

摘要

Due to their high degree of expressiveness, neural networks have recently been used as surrogate models for mapping inputs of an engineering system to outputs of interest. Once trained, neural networks are computationally inexpensive to evaluate and remove the need for repeated evaluations of computationally expensive models in uncertainty quantification applications. However, given the highly parameterized construction of neural networks, especially deep neural networks, accurate training often requires large amounts of simulation data that may not be available in the case of computationally expensive systems. In this paper, to alleviate this issue for uncertainty propagation, we explore the application of transfer learning techniques using training data generated from both high- and low-fidelity models. We explore two strategies for coupling these two datasets during the training procedure, namely, the standard transfer learning and the bi-fidelity-weighted learning. In the former approach, a neural network model mapping the inputs to the outputs of interest is trained based on the low-fidelity data. The high-fidelity data are then used to adapt the parameters of the upper layer(s) of the low-fidelity network, or train a simpler neural network to map the output of the low-fidelity network to that of the high-fidelity model. In the latter approach, the entire low-fidelity network parameters are updated using data generated via a Gaussian process model trained with a small high-fidelity dataset. The parameter updates are performed via a variant of stochastic gradient descent with learning rates given by the Gaussian process model. Using three numerical examples, we illustrate the utility of these bi-fidelity transfer learning methods where we focus on accuracy improvement achieved by transfer learning over standard training approaches.

参考文献
  1. Gurney, K., An Introduction to Neural Networks, Boca Raton, FL: CRC Press, 2014.

  2. Goodfellow, I., Bengio, Y., and Courville, A., Deep Learning, Cambridge, MA: MIT Press, 2016.

  3. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., and Zheng, X., Tensorflow: A System for Large-Scale Machine Learning, in Proc. of 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI16), Savannah, GA, pp. 265-283, 2016.

  4. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y., Theano: A CPU and GPU Math Expression Compiler, in Proc. of the Python for Scientific Computing Conf. (SciPy), Austin, TX, Vol. 4,2010.

  5. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T., Caffe: Convolutional Architecture for Fast Feature Embedding, Comput. Vision Pattern Recog., arXiv:1408.5093,2014. (a).

  6. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A., Automatic Differentiation in PyTorch, in Proc. NIPS 2017 Autodiff Workshop: The Future of Gradient-Based Machine Learning Software and Techniques, Long Beach, CA, 2017.

  7. U.S. Department of Energy, U.S. Department of Energy and Intel to Build First Exascale Supercomputer, from https://www.energy.gov/articles/us-department-energy-and-intel-build-first-exascale-supercomputer, accessed Aug. 20, 2019.

  8. U.S. Department of Energy, U.S. Department of Energy and Cray to Deliver Record-Setting Frontier Supercomputer at ORNL, from https://www.energy.gov/articles/us-department-energy-and-cray-deliver-record-setting-frontier-supercomputer-ornl, accessed Aug. 29, 2019.

  9. TOP500.org, TOP 500 LIST, from https://www.top500.org/lists/2019/06/, accessed Aug. 20,2019.

  10. Cybenko, G., Approximation by Superpositions of a Sigmoidal Function, Math. Control, Signals Syst., 2(4):303-314, 1989.

  11. Hornik, K., Stinchcombe, M., and White, H., Multilayer Feedforward Networks are Universal Approximators, Neural Networks, 2(5):359-366, 1989.

  12. Hornik, K., Stinchcombe, M., and White, H., Universal Approximation of an Unknown Mapping and Its Derivatives Using Multilayer Feedforward Networks, Neural Networks, 3(5):551-560, 1990.

  13. Opschoor, J.A., Petersen, P., and Schwab, C., Deep ReLU Networks and High-Order Finite Element Methods, SAM, ETH Zurich, Switzerland, Tech. Rep. 2019-07, 2019.

  14. Baker, N., Alexander, F., Bremer, T., Hagberg, A., Kevrekidis, Y., Najm, H., Parashar, M., Patra, A., Sethian, J., and Wild, S., Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence, Tech. Rep., USDOE Office of Science (SC), Washington, DC (US), 2019.

  15. Fernandez-Godino, M.G., Park, C., Kim, N.H., and Haftka, R.T., Review of Multi-Fidelity Models, Stat. Appl., arXiv:1609.07196, 2016.

  16. Peherstorfer, B., Willcox, K., and Gunzburger, M., Survey of Multifidelity Methods in Uncertainty Propagation, Inference, and Optimization, SIAMRev, 60(3):550-591,2018.

  17. Sirovich, L., Turbulence and the Dynamics of Coherent Structures Part I: Coherent Structures, Q. Appl. Math, 45(3):561-571, 1987.

  18. Berkooz, G., Holmes, P., and Lumley, J.L., The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows, Annu. Rev. FluidMech, 25(1):539-575, 1993.

  19. Benner, P., Gugercin, S., and Willcox, K., A Survey of Projection-Based Model Reduction Methods for Parametric Dynamical Systems, SIAM Rev., 57(4):483-531, 2015.

  20. Rasmussen, C.E., Gaussian Processes in Machine Learning, in Advanced Lectures on Machine Learning, New York: Springer, pp. 63-71,2004.

  21. Martin, J.D. and Simpson, T.W.,Use ofKriging Models to Approximate Deterministic Computer Models, AIAAJ., 43(4):853- 863, 2005.

  22. Forrester, A.I. and Keane, A.J., Recent Advances in Surrogate-Based Optimization, Prog. Aerosp. Sci., 45(1-3):50-79, 2009.

  23. Koutsourelakis, P.S., Accurate Uncertainty Quantification Using Inaccurate Computational Models, SIAM J. Sci. Comput., 31(5):3274-3300, 2009.

  24. Li, J. and Xiu, D., Evaluation of Failure Probability via Surrogate Models, J. Comput. Phys, 229(23):8966-8980, 2010.

  25. Li, J., Li, J., and Xiu, D., An Efficient Surrogate-Based Method for Computing Rare Failure Probability, J. Comput. Phys, 230(24):8683-8697,2011.

  26. Peherstorfer, B., Kramer, B., and Willcox, K., Combining Multiple Surrogate Models to Accelerate Failure Probability Estimation with Expensive High-Fidelity Models, J. Comput. Phys., 341:61-75, 2017.

  27. Kennedy, M. and O'Hagan, A., Predicting the Output from a Complex Computer Code when Fast Approximations are Available, Biometrika, 87(1):1-13, 2000.

  28. Qian, Z., Seepersad, C., Joseph, V., Allen, J.K., and Wu, C., Building Surrogate Models based on Detailed and Approximate Simulations, J. Mech. Des., 128(4):668-677, 2006.

  29. Kleiber, W., Sain, S.R., Heaton, M.J., Wiltberger, M., Reese, C.S., and Bingham, D., Parameter Tuning for a Multi-Fidelity Dynamical Model of the Magnetosphere, Ann. Appl. Stat., 7:1286-1310, 2013.

  30. Le Gratiet, L. and Garnier, J., Recursive Co-Kriging Model for Design of Computer Experiments with Multiple Levels of Fidelity, Int. J. Uncertainty Quantif., 4:365-386, 2014.

  31. Eldred, M., Recent Advances in Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Analysis and Design, in Proc. of 50th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conf. 17th AIAA/ASME/AHS Adaptive Structures Conf., p. 2274,2009.

  32. Ng, L.W.T. and Eldred, M., Multifidelity Uncertainty Quantification Using Non-Intrusive Polynomial Chaos and Stochastic Collocation, in Proc. of 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conf. 20th AIAA/ASME/AHS Adaptive Structures Conf. 14th AIAA, p. 1852, 2012.

  33. Padron, A.S., Alonso, J.J., and Eldred, M.S., Multi-Fidelity Methods in Aerodynamic Robust Optimization, in Proc. of 18th AIAA Non-Deterministic Approaches Conf., p. 0680, 2016.

  34. Doostan, A., Ghanem, R., and Red-Horse, J., Stochastic Model Reduction for Chaos Representations, Comput. Meth. Appl. Mech. Eng., 196(37-40):3951-3966, 2007.

  35. Ghanem, R., Saad, G., and Doostan, A., Efficient Solution of Stochastic Systems: Application to the Embankment Dam Problem, Struct. Saf., 29(3):238-251, 2007.

  36. Narayan, A., Gittelson, C., and Xiu, D., A Stochastic Collocation Algorithm with Multifidelity Models, SIAMJ. Sci. Comput., 36(2):A495-A521, 2014.

  37. Zhu, X., Narayan, A., and Xiu, D., Computational Aspects of Stochastic Collocation with Multifidelity Models, SIAM/ASA J. Uncertainty Quantif, 2(1):444-463,2014.

  38. Doostan, A., Geraci, G., and Iaccarino, G., A Bi-Fidelity Approach for Uncertainty Quantification of Heat Transfer in a Rectangular Ribbed Channel, in Proc. ASME Turbo Expo 2016: Turbomachinery Technical Conference and Exposition, p. V02CT45A031, 2016.

  39. Hampton, J., Fairbanks, H.R., Narayan, A., and Doostan, A., Practical Error Bounds for a Non-Intrusive Bi-Fidelity Approach to Parametric/Stochastic Model Reduction, J. Comput. Phys, 368:315-332, 2018.

  40. Fairbanks, H.R., Jofre, L., Geraci, G., Iaccarino, G., and Doostan, A., Bi-Fidelity Approximation for Uncertainty Quantification and Sensitivity Analysis of Irradiated Particle-Laden Turbulence, Comput. Phys, arXiv:1808.05742,2018.

  41. Skinner, R.W., Doostan, A., Peters, E.L., Evans, J.A., and Jansen, K.E., Reduced-Basis Multifidelity Approach for Efficient Parametric Study of NACA Airfoils, AIAA J, 57(4):1481-1491,2019.

  42. Pan, S.J. and Yang, Q., A Survey on Transfer Learning, IEEE Trans. Knowl. Data Eng., 22(10):1345-1359, 2010.

  43. Weiss, K., Khoshgoftaar, T.M., and Wang, D., A Survey of Transfer Learning, J. Big Data, 3(1):9, 2016.

  44. Li, F., Pan, S.J., Jin, O., Yang, Q., and Zhu, X., Cross-Domain Co-Extraction of Sentiment and Topic Lexicons, in Proc. of 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pp. 410-419, 2012.

  45. Zhu, Y., Chen, Y., Lu, Z., Pan, S.J., Xue, G.R., Yu, Y., and Yang, Q., Heterogeneous Transfer Learning for Image Classification, in Proc. 25th AIAA Conf. on Artificial Intelligence, 2011.

  46. Collobert, R. and Weston, J., A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning, in Proc. of the 25th International Conf. on Machine Learning, pp. 160-167, 2008.

  47. Kan, M., Wu, J., Shan, S., and Chen, X., Domain Adaptation for Face Recognition: Targetize Source Domain Bridged by Common Subspace, Int. J. Comput. Vision, 109(1-2):94-109,2014.

  48. Ma, Y., Gong, W., and Mao, F., Transfer Learning Used to Analyze the Dynamic Evolution of the Dust Aerosol, J. Quantitative Spectrosc. Radiat. Transf, 153:119-130, 2015.

  49. Ogoe, H.A., Visweswaran, S., Lu, X., and Gopalakrishnan, V., Knowledge Transfer via Classification Rules Using Functional Mapping for Integrative Modeling of Gene Expression Data, BMC Bioinf., 16(1):226, 2015.

  50. Widmer, C. and Ratsch, G., Multitask Learning in Computational Biology, in Proc. of ICML Workshop on Unsupervised and Transfer Learning, pp. 207-216,2012.

  51. Dehghani, M., Zamani, H., Severyn, A., Kamps, J., and Croft, W.B., Neural Ranking Models with Weak Supervision, in Proc. of 40th International ACM SIGIR Conf. on Research and Development in Information Retrieval, pp. 65-74, 2017.

  52. Dehghani, M., Mehrjou, A., Gouws, S., Kamps, J., and Scholkopf, B., Fidelity-Weighted Learning, in Proc. of Int. Conf. on Learning Representations (ICLR 2018), 2018.

  53. Hinton, G.E., Osindero, S., and Teh, Y.W., A Fast Learning Algorithm for Deep Belief Nets, Neural Comput., 18(7):1527-1554,2006.

  54. Vapnik, V. and Vashist, A., A New Learning Paradigm: Learning Using Privileged Information, Neural Networks, 22(5-6):544-557,2009.

  55. Vapnik, V. and Izmailov, R., Learning Using Privileged Information: Similarity Control and Knowledge Transfer, J. Machine Learn. Res., 16(1):2023-2049, 2015.

  56. Torrey, L. and Shavlik, J., Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, Hershey, PA: IGI Global, pp. 242-264, 2010.

  57. Rosenstein, M.T., Marx, Z., Kaelbling, L.P., and Dietterich, T.G., To Transfer or not to Transfer, in Proc. of NIPS 2005 Workshop on Transfer Learning, Vol. 898, p. 3, 2005.

  58. Ge, L., Gao, J., Ngo, H., Li, K., and Zhang, A., On Handling Negative Transfer and Imbalanced Distributions in Multiple Source Transfer Learning, Stat. Anal. Data Mining: The ASA Data Sci. J, 7(4):254-271, 2014.

  59. Wang, Z., Dai, Z., Poczos, B., and Carbonell, J., Characterizing and Avoiding Negative Transfer, in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11293-11302,2019.

  60. He, K., Zhang, X., Ren, S., and Sun, J., Deep Residual Learning for Image Recognition, in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.

  61. Clevert, D.A., Unterthiner, T., and Hochreiter, S., Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), Comput. Sci. Machine Learning, arXiv:1511.07289,2015.

  62. Higham, C.F. and Higham, D.J., Deep Learning: An Introduction for Applied Mathematicians, Math. Hist. Over, arXiv:1801.05894, 2018.

  63. Kingma, D. and Ba, J., Adam: A Method for Stochastic Optimization, Comput. Sci. Mach. Learn., arXiv:1412.6980, 2014.

  64. De, S., Hampton, J., Maute, K., and Doostan, A., Topology Optimization under Uncertainty Using a Stochastic Gradient-Based Approach, Math. Opt. Control, arXiv:1902.04562, 2019.

  65. Bengio, Y., Practical Recommendations for Gradient-Based Training of Deep Architectures, New York: Springer, pp. 437-478,2012.

  66. He, K., Zhang, X., Ren, S., and Sun, J., Delving Deep into Rectifiers: Surpassing Human-Level Performance on Imagenet Classification, in Proc. of the IEEE Int. Conf. on Computer Vision, pp. 1026-1034, 2015.

  67. Hanin, B., Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations, Stat. Mach. Learn., arXiv:1708.02691,2017.

  68. Meng, X. and Karniadakis, G.E., A Composite Neural Network That Learns from Multi-Fidelity Data: Application to Function Approximation and Inverse PDE Problems, J. Comput. Phys., 401:109020, 2020.

  69. Lu, L., Dao, M., Kumar, P., Ramamurty, U., Karniadakis, G.E., and Suresh, S., Extraction of Mechanical Properties of Materials through Deep Learning from Instrumented Indentation, Proc. Nat. Acad. Sci., 117(13):201922210, 2020.

  70. Motamed, M., A Multi-Fidelity Neural Network Surrogate Sampling Method for Uncertainty Quantification, Math. Numer. Anal, arXiv:1909.01859, 2019.

  71. Mitchell, T.J. and Beauchamp, J.J., Bayesian Variable Selection in Linear Regression, J. Am. Stat. Assoc., 83(404):1023-1032, 1988.

  72. Johnson, R. and Zhang, T., Accelerating Stochastic Gradient Descent Using Predictive Variance Reduction, in Proc. of Advances in Neural Information Processing Systems, pp. 315-323,2013.

  73. De, S., Maute, K., and Doostan, A., Bi-Fidelity Stochastic Gradient Descent for Structural Optimization under Uncertainty, Comp. Mech, 66:745-771, 2020.

  74. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., and Dubourg, V., Scikit-Learn: Machine Learning in Python, J. Mach. Learn. Res., 12:2825-2830, 2011.

  75. Logg, A., Mardal, K.A., and Wells, G.N., Automated Solution of Differential Equations by the Finite Element Method, New York: Springer, 2012.

  76. Hadigol, M., Maute, K., and Doostan, A., On Uncertainty Quantification of Lithium-Ion Batteries: Application to an LiC6/LiCoO2 Cell, J. Power Sources, 300:507-524,2015.

  77. Newman, J. and Tiedemann, W., Porous-Electrode Theory with Battery Applications, AIChE J, 21(1):25-41,1975.

  78. Diskin, B., Thomas, J., Rumsey, C.L., and Schwoppe, A., Grid Convergence for Turbulent Flows, in Proc. of 53rd AIAA Aerospace Sciences Meeting, p. 1746, 2015.

  79. Whiting, C.H. and Jansen, K.E., A Stabilized Finite Element Method for the Incompressible Navier-Stokes Equations Using a Hierarchical Basis, Int. J. Numer. Methods Fluids, 35(1):93-116, 2001.

  80. Spalart, P. and Allmaras, S., A One-Equation Turbulence Model for Aerodynamic Flows, in Proc. of 30th Aerospace Sciences Meeting and Exhibit, p. 439, 1992.

  81. Pope, S.B., Turbulent Flows, Cambridge: Cambridge University Press, 2001.

  82. Duvenaud, D., Automatic Model Construction with Gaussian Processes, PhD, University of Cambridge, 2014.

对本文的引用
  1. Pandita Piyush, Ghosh Sayan, Gupta Vipul K., Meshkov Andrey, Wang Liping, Application of Deep Transfer Learning and Uncertainty Quantification for Process Identification in Powder Bed Fusion, ASCE-ASME J Risk and Uncert in Engrg Sys Part B Mech Engrg, 8, 1, 2022. Crossref

  2. Keshavarzzadeh Vahid, Kirby Robert M., Narayan Akil, Robust topology optimization with low rank approximation using artificial neural networks, Computational Mechanics, 68, 6, 2021. Crossref

  3. Rosafalco Luca, Torzoni Matteo, Manzoni Andrea, Mariani Stefano, Corigliano Alberto, A Self-adaptive Hybrid Model/data-Driven Approach to SHM Based on Model Order Reduction and Deep Learning, in Structural Health Monitoring Based on Data Science Techniques, 21, 2022. Crossref

  4. De Subhayan, Doostan Alireza, Neural network training using ℓ1-regularization and bi-fidelity data, Journal of Computational Physics, 458, 2022. Crossref

  5. De Subhayan, Uncertainty Quantification of Locally Nonlinear Dynamical Systems Using Neural Networks, Journal of Computing in Civil Engineering, 35, 4, 2021. Crossref

  6. Partin Lauren, Rushdi Ahmad A., Schiavazzi Daniele E., Multifidelity data fusion in convolutional encoder/decoder assembly networks for computational fluid dynamics, AIAA SCITECH 2022 Forum, 2022. Crossref

  7. Trinh Van Hai, Guilleminot Johann, Perrot Camille, Vu Viet Dung, Learning acoustic responses from experiments: A multiscale-informed transfer learning approach, The Journal of the Acoustical Society of America, 151, 4, 2022. Crossref

  8. De Subhayan, Ebna Hai Bhuiyan Shameem Mahmood, Doostan Alireza, Bause Markus, Prediction of Ultrasonic Guided Wave Propagation in Fluid–Structure and Their Interface under Uncertainty Using Machine Learning, Journal of Engineering Mechanics, 148, 3, 2022. Crossref

  9. Harada Kohei, Rajaram Dushhyanth, Mavris Dimitri N., Application of Multi-Fidelity Physics-Informed Neural Network on Transonic Airfoil using Wind Tunnel Measurements, AIAA SCITECH 2022 Forum, 2022. Crossref

  10. Newberry Felix, Hampton Jerrad, Jansen Kenneth, Doostan Alireza, Bi-fidelity reduced polynomial chaos expansion for uncertainty quantification, Computational Mechanics, 69, 2, 2022. Crossref

  11. Huang Xufeng, Xie Tingli, Wang Zhuo, Chen Lei, Zhou Qi, Hu Zhen, A Transfer Learning-Based Multi-Fidelity Point-Cloud Neural Network Approach for Melt Pool Modeling in Additive Manufacturing, ASCE-ASME J Risk and Uncert in Engrg Sys Part B Mech Engrg, 8, 1, 2022. Crossref

  12. Kou Jiaqing, Ning Chenjia, Zhang Weiwei, Transfer Learning for Flow Reconstruction Based on Multifidelity Data, AIAA Journal, 60, 10, 2022. Crossref

  13. Jiang Su, Durlofsky Louis J., Use of multifidelity training data and transfer learning for efficient construction of subsurface flow surrogate models, Journal of Computational Physics, 474, 2023. Crossref

Begell Digital Portal Begell 数字图书馆 电子图书 期刊 参考文献及会议录 研究收集 订购及政策 Begell House 联系我们 Language English 中文 Русский Português German French Spain