A Review of Evolutionary Deep Neural Architecture Search with Performance Predictor

QIAOZenglin; QIAO Zenglin; ZHAOXinchao; ZHAO Xinchao; WULingyu; WU Lingyu

doi:10.48014/bcam.20230822002

A Review of Evolutionary Deep Neural Architecture Search with Performance Predictor

中国应用数学通报 2023年第1卷第1期页码[1-9] 下载全文[2MB] [HTML]

摘要

Automated design of deep neural networks using performance predictor has become a hot topic in current research. Neural architecture search (NAS) methods can be used to enable automatic design of neural network structures by defining different search spaces, search strategies, or optimization strategies. Evolutionary computation by many researchers as the search strategy for NAS, which is called evolutionary NAS (ENAS) . However, ENAS is time-consuming in evaluating the performance of network structures, which hinders the development of ENAS. Therefore, predicting network architecture performance using performance predictor can improve ENAS search speed and save computational resources. This paper summarizes several ENAS methods that utilize performance predictor, and discusses an outlook on search space, search strategies, and the future directions of ENAS assisted by performance predictor.

DOI

10.48014/bcam.20230822002

文章类型

研究性论文

收稿日期

2023-08-22

接收日期

2023-09-15

出版日期

2023-12-28

关键词

Neural architecture search, evolutionary algorithm, deep learning, convolutional neural network

作者

QIAO Zenglin^1,2, ZHAO Xinchao^1,2,*, WU Lingyu^1,2

Author

QIAO Zenglin^1,2, ZHAO Xinchao^1,2,*, WU Lingyu^1,2

所在单位

1. School of Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
2. Key Laboratory of Mathematics and Information Networks (Beijing University of Posts and Telecommunications) , Ministry of Education, Beijing 100876, China.

浏览量

526

下载量

295

参考文献

[1] D. W. Otter, J. R. Medina, and J. K. Kalita. A survey of the usages of deep learning for natural language processing[J]. IEEE transactions on neural networks and learning systems, 2020, 32(2): 604-624.
doi:10.1109/TNNLS.2020.2979670
[2] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition[J]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 770-778.
[3] G. Nguyen et al. Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey[J]. Artificial Intelligence Review, 2019, 52: 77-124.
https://doi.org/10.1007/s10462-018-09679-z
[4] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
doi:10.1145/3065386
[5] C. Liang-Chieh, G. Papandreou, I. Kokkinos, et al. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs[C]. International Conference on Learning Representations, 2015.
https://doi.org/10.48550/arXiv.1412.7062
[6] W. L. Christian Szegedy, Yangqing Jia, Pierre Sermanet, et al. Going Deeper With Convolutions[J]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2015: 1-9.
doi:10.1109/CVPR.2015.7298594
[7] L. Xie and A. Yuille. Genetic cnn[J]. Proceedings of the IEEE international conference on computer vision, 2017: 1379-1388.
doi:10.1109/ICCV.2017.154
[8] H. Liu, K. Simonyan, O. Vinyals, et al. Hierarchical Representations for Efficient Architecture Search[C]. International Conference on Learning Representations, 2018.
https://doi.org/10.48550/arXiv.1711.00436
[9] H. Cai, T. Chen, W. Zhang, et al. Efficient architecture search by network transformation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32, no. 1.
https://doi.org/10.1609/aaai.v32i1.11709
[10] B. Zoph, V. Vasudevan, J. Shlens, et al. Learning transferable architectures for scalable image recognition[J]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8697-8710.
https://doi.org/10.48550/arXiv.1707.07012
[11] E. Real et al. Large-scale evolution of image classifiers [C]. International Conference on Machine Learning, 2017: PMLR, pp. 2902-2911.
[12] M. Suganuma, S. Shirakawa, and T. Nagao. A genetic programming approach to designing convolutional neural network architectures[C]. Proceedings of the genetic and evolutionary computation conference, 2017, pp. 497-504.
doi:10.1145/3071178.3071229
[13] B. Baker, O. Gupta, N. Naik, et al. Designing Neural Network Architectures using Reinforcement Learning[C]. International Conference on Learning Representations, 2016.
doi:10.48550/arXiv.1611.02167
[14] Y. Sun, B. Xue, M. Zhang, et al. Evolving deep convolutional neural networks for image classification[J]. IEEE Transactions on Evolutionary Computation, 2019, 24(2): 394-407.
doi:10.1109/TEVC.2019.2916183
[15] T. Back. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms[M]. Oxford university press, 1996.
https://doi.org/10.1108/k.1998.27.8.979.4
[16] W. Banzhaf, P. Nordin, R. E. Keller, et al. Francone, Genetic programming: an introduction: on the automatic evolution of computer programs and its applications[M]. Morgan Kaufmann Publishers Inc. , 1998.
doi:dl.acm.org/doi/abs/10.5555/323917
[17] H. -G. Beyer and H. -P. Schwefel. Evolution strategies-a comprehensive introduction[J]. Natural computing, 2002, 1: 3-52.
https://doi.org/10.1023/A:1015059928466
[18] L. M. Schmitt. Theory of genetic algorithms[J]. Theoretical Computer Science, 2001, 259(1-2): 1-61.
https://doi.org/10.1016/S0304-3975(00)00406-0
[19] K. Deb, A. Pratap, S. Agarwal, et al. A fast and elitist multiobjective genetic algorithm: NSGA-II[J]. IEEE transactions on evolutionary computation, 2002, 6(2): 182-197.
doi:10.1109/4235.996017
[20] M. Jiang, Z. Huang, L. Qiu, et al. Transfer learningbased dynamic multiobjective optimization algorithms[J]. IEEE Transactions on Evolutionary Computation, 2017, 22(4): 501-514.
doi:10.1109/TEVC.2017.2771451
[21] Z. Qiao et al. Gaussian bare-bones gradient-based optimization: Towards mitigating the performance concerns[J]. International Journal of Intelligent Systems, 2022, 37(6): 3193-3254.
https://doi.org/10.1002/int.22658
[22] C. Yinnan, Y. Lingjuan, L. Rui, et al. A Multi-period Constrained Multi-objective Evolutionary Algorithm with Orthogonal Learning for Solving the Complex Carbon Neutral Stock Portfolio Optimization Model[J]. Journal of Systems Science and Complexity, p. 0.
https://doi.org/10.1007/s11424-023-2406-3
[23] Y. Sun, B. Xue, M. Zhang, et al. Automatically designing CNN architectures using the genetic algorithm for image classification[J]. IEEE transactions on cybernetics, 2020, 50(9): 3840-3854.
doi:10.1109/TCYB.2020.2983860
[24] B. Zoph, and Q. Le. Neural Architecture Search with Reinforcement Learning[C]. International Conference on Learning Representations, 2016.
https://doi.org/10.48550/arXiv.1611.01578
[25] L. Wen, L. Gao, X. Li, et al. A new genetic algorithm based evolutionary neural architecture search for image classification[J]. Swarm and Evolutionary Computation, 2022, 75: 101191.
https://doi.org/10.1016/j.swevo.2022.101191
[26] C. He, H. Tan, S. Huang, et al. Efficient evolutionary neural architecture search by modular inheritable crossover[J]. Swarm and Evolutionary Computation, 2021, 64: 100894.
https://doi.org/10.1016/j.swevo.2021.100894
[27] B. Wang, Y. Sun, B. Xue, et al. Evolving deep convolutional neural networks by variable-length particle swarm optimization for image classification[C]. 2018 IEEE Congress on Evolutionary Computation(CEC), 2018: IEEE, pp. 1-8.
doi:10.1109/CEC.2018.8477735
[28] Y. Sun, H. Wang, B. Xue, et al. Surrogate-assisted evolutionary deep learning using an end-to-end random forest- based performance predictor[J]. IEEE Transactions on Evolutionary Computation, 2019, 24(2): 350-364.
doi:10.1109/TEVC.2019.2924461
[29] Z. Lu, K. Deb, E. Goodman, et al. Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search[C]. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, 2020: Springer, pp. 35-51.
https://doi.org/10.1007/978-3-030-58452-8_3
[30] B. Baker, O. Gupta, R. Raskar, et al. Accelerating Neural Architecture Search using Performance Prediction, 2018.
https://doi.org/10.48550/arXiv.1705.10823
[31] C. Liu et al. Progressive neural architecture search[C]. Proceedings of the European conference on computer vision(ECCV), 2018, pp. 19-34.
https://doi.org/10.1007/978-3-030-01246-5_2
[32] X. Dai et al. Chamnet: Towards efficient network design through platform-aware model adaptation[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11398-11407.
doi:10.1109/CVPR.2019.01166
[33] M. B. Calisto, and S. K. Lai-Yuen. EMONAS-Net: Efficient multiobjective neural architecture search using surrogate-assisted evolutionary algorithm for 3D medical image segmentation[J]. Artificial Intelligence in Medicine, 2021, 119: 102154.
https://doi.org/10.1016/j.artmed.2021.102154
[34] C. Wei, C. Niu, Y. Tang, et al. Npenas: Neural predictor guided evolution for neural architecture search[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022.
doi:10.1109/TNNLS.2022.3151160
[35] Y. Peng, A. Song, V. Ciesielski, et al. PRE-NAS: Evolutionary Neural Architecture Search with Predictor[J]. IEEE Transactions on Evolutionary Computation, 2022.
doi:10.1109/TEVC.2022.3227562

引用本文

QIAO Zenglin, ZHAO Xinchao, WU Lingyu. A review of evolutionary deep neural architecture search with performance predictor[J]. Bulletin of Chinese Applied Mathematics, 2023, 1(1): 1-9.