RT Journal Article ID 5d9ead70797784fb A1 Shin, Yeonjong A1 Karniadakis, George Em T1 TRAINABILITY OF ReLU NETWORKS AND DATA-DEPENDENT INITIALIZATION JF Journal of Machine Learning for Modeling and Computing JO JMLMC YR 2020 FD 2020-06-19 VO 1 IS 1 SP 39 OP 74 K1 ReLU networks K1 trainability K1 dying ReLU K1 overparameterization K1 over-specification K1 data-dependent initialization AB In this paper we study the trainability of rectified linear unit (ReLU) networks at initialization. A ReLU neuron is said to be dead if it only outputs a constant for any input. Two death states of neurons are introduced−tentative and permanent death. A network is then said to be trainable if the number of permanently dead neurons is sufficiently small for a learning task. We refer to the probability of a randomly initialized network being trainable as trainability. We show that a network being trainable is a necessary condition for successful training, and the trainability serves as an upper bound of training success rates. In order to quantify the trainability, we study the probability distribution of the number of active neurons at initialization. In many applications, overspecified or overparameterized neural networks are successfully employed and shown to be trained effectively. With the notion of trainability, we show that overparameterization is both a necessary and a sufficient condition for achieving a zero training loss. Furthermore, we propose a data-dependent initialization method in an overparameterized setting. Numerical examples are provided to demonstrate the effectiveness of the method and our theoretical findings. PB Begell House LK https://www.dl.begellhouse.com/journals/558048804a15188a,583c4e56625ba94e,5d9ead70797784fb.html