Hybrid Method of Conventional Neural Network Training
- 作者: Golubinskiy A.N1, Tolstykh A.A2
-
隶属关系:
- JSC “Concern “Sozvezdie”
- Moscow University of the Ministry of Internal Affairs of Russia
- 期: 卷 20, 编号 2 (2021)
- 页面: 463-490
- 栏目: Artificial intelligence, knowledge and data engineering
- URL: https://journal-vniispk.ru/2713-3192/article/view/266310
- DOI: https://doi.org/10.15622/ia.2021.20.2.8
- ID: 266310
如何引用文章
全文:
详细
The paper proposes a hybrid method for training convolutional neural networks. The method consists of combining second and first-order methods for different elements of the architecture of a convolutional neural network. The hybrid convolution neural network training method allows to achieve significantly better convergence compared to Adam; however, it requires fewer computational operations to implement. Using the proposed method, it is possible to train networks on which learning paralysis occurs when using first-order methods. Moreover, the proposed method could adjust its computational complexity to the hardware on which the computation is performed; at the same time, the hybrid method allows using the mini-packet learning approach. The analysis of the ratio of computations between convolutional neural networks and fully connected artificial neural networks is presented. The mathematical apparatus of error optimization of artificial neural networks is considered, including the method of backpropagation of the error, the Levenberg-Marquardt algorithm. The main limitations of these methods that arise when training a convolutional neural network are analyzed. The analysis of the stability of the proposed method when the initialization parameters are changed. The results of the applicability of the method in various problems are presented.
作者简介
A. Golubinskiy
JSC “Concern “Sozvezdie”
Email: annikgol@mail.ru
Moskovsky Av. 92
A. Tolstykh
Moscow University of the Ministry of Internal Affairs of Russia
Email: tolstykh.aa@yandex.ru
Koptevskaya St. 63
参考
- Гудфеллоу Я., Бенджио И., Курвилль А. Глубокое обучение. Москва: «ДМК Пресс», 2017. 652 с.
- Хайкин С. Нейронные сети. Полный курс. 2-е изд. Москва: Издательский дом "Вильямс", 2006. 1104 с.
- Kingma D.P., Ba J. Adam: A Method for Stochastic Optimization // CoRR, Т. abs/1412.6980, 2014.
- Bo Y.H., WeiL., I-Chen W. Stochastic Gradient Descent with Hyperbolic-Tangent Decay // CoRR, Т. abs/1806.01593, 2018. С. 1-10.
- Wilamowski B.M., Irwin D.J. Intelligent systems. 2-е-е изд. Бока-Ратон: CRC Press, 2011. 568 с.
- S. Smith J., Wu B., M. Wilamowski B. Neural Network Training With Leven-berg-Marquardt and Adaptable Weight Compression // IEEE Transactions on Neural Networks and Learning Systems, Т. PP, Jul 2018. С. 1-8.
- Szegedy C., Ioffe S., Vanhoucke V., Alemi A.A. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning // International Conference on Learning Representations (ICLR) Workshop, 2016. С. 375-387.
- He K., Zhang X., Ren S., Sun J. [1502.01852] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification // arXiv.org e-Print archive. 2015. URL: https://arxiv.org/abs/1502.01852 (дата обращения: 12.11.2020).
- Szegedy С., Liu W., Jia Y., Sermanet P., Reed S., Angueloy D., Ethan D., Vanhoucke V., Rabinovich A. Going deeper with convolutions // IEEE Confer-ence on Computer Vision and Pattern Recognition. 2014. С. 1-9.
- Zaccone G., Karim R., Menshawy A. Deep Learning with TensorFlow: Explore neural networks with Python. Бирмингем: Packt Publishing, 2017. 320 с.
- Вьюгин В.В. Математические основы теории машинного обучения и про-гнозирования. Москва: МЦНМО, 2013. 387 с.
- Shepherd A.J. Second-Order Methods for Neural Networks: Fast and Reliable Training Methods for Multi-Layer Perceptrons. New York: Springer, 1997. 160 с.
- Nocedal J., Wright S.J. Numerical Optimization. 2nd ed. New York: Springer, 2006. 664 с.
- Голубинский А.Н. О построении архитектур и оценке параметров искус-ственных нейронных сетей // Теория и техника радиосвязи, № 1, 2020. С. 72-87.
- Максимушкин В.В., Арзамасцев А.А. Сравнительная оценка вычислитель-ной сложности обучения искусственной нейронной сети с жестким ядром и сети с классической структурой // Вестник российских университетов. Математика., № 2, 2006. С. 190-197.
- Абрамов С.А. Лекции о сложности алгоритмов. Москва: МЦНМО, 2012. 248 с.
- Yann LeCun's Home Page [Электронный ресурс] // MNIST handwritten digit database, Yann LeCun, Corinna Cortes and Chris Burges: [сайт]. [2012]. URL: http://yann.lecun.com/exdb/mnist/ (дата обращения: 23.05.2020).
- Smith L.N. No More Pesky Learning Rate Guessing Games // CoRR, Vol. abs/1506.01186, 2015. С. 1-10.
- Salimans T., Kingma D.P. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks // In: Advances in Neural In-formation Processing Systems 29. Curran Associates, Inc., 2016. С. 901–909.
- Matuszyk P., Castillo R.T., Kottke D., Spiliopoulou M. A Comparative Study on Hyperparameter Optimization for Recommender Systems // Workshop on Recommender Systems and Big Data Analytics (RS-BDA'16) @ iKNOW 2016. 2016.
- Hu G., Peng X., Yang Y., Hospedales T.M., Verbeek J. Frankenstein: Learning Deep Face Representations using Small Data // CoRR, Т. abs/1603.06470, 2016.
- Ioffe S., Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift // CoRR, Т. abs/1502.03167, 2015.
- Lewkowycz A., Bahri Y., Dyer E., Sohl-Dickstein J., Gur-Ari G. The large learning rate phase of deep learning: the catapult mechanism. 2020.
- Liao Q., Kawaguchi K., Poggio T.A. Streaming Normalization: Towards Sim-pler and More Biologically-plausible Normalizations for Online and Recurrent Learning // CoRR, Т. abs/1610.06160, 2016.
- Mahajan D., Girshick R.B., Ramanathan V., He K., Paluri M., Li Y., Bharambe A., van der Maaten L. Exploring the Limits of Weakly Supervised Pretraining // CoRR, Т. abs/1805.00932, 2018.
- Petroski Such F., Madhavan V., Conti E., Lehman J., Stanley K.O., Clune J. Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning // ArXiv e-prints, 2017.
- Su H., Zhu X., Gong S. Deep Learning Logo Detection with Data Expansion by Synthesising Context // CoRR, Т. abs/1612.09322, 2016.
- Touvron H., Vedaldi A., Douze M., Jégou H. Fixing the train-test resolution discrepancy // CoRR, Т. abs/1906.06423, 2019.
- Xu C., Qin T., Wang G., Liu T.Y. An Actor-Critic Algorithm For Learning Rate 2017. С. 1–12.
- Xu C., Qin T., Wang G., Liu T.Y. Reinforcement Learning for Learning Rate Control // CoRR, Т. abs/1705.11159, 2017.
补充文件
