Initial results of neural architecture search (NAS) in natural language processing (NLP) have been achieved, but the search space of most NAS methods is based on the simplest recurrent cell and thus does not consider the modeling of long sequences. The remote information tends to disappear gradually when the input sequence is long, resulting in poor model performance. In this paper, we present an approach based on dual cells to search for a better-performing network architecture. We construct a search space that is more compatible with language modeling tasks by adding an information storage cell inside the search cell, so that we can make better use of the remote information of the sequence and improve the performance of the model. The language model searched by our method achieves better results than those of the baseline method on the Penn Treebank data set and WikiText-2 data set.
HintonG.E.S., et al., Reducing the Dimensionality of Data with Neural Networks. [J], Science, (2006).
2.
HeK., ZhangX., RenS., et al., Deep residual learning for image recognition[C]//, Proceedings of the IEEE conference on computer vision and pattern recognition2016, 770–778.
3.
TodericiG., O’MalleyS.M., HwangS.J., et al., Recurrent Neural Network Regularization.
4.
VaswaniA., ShazeerN., ParmarN., UszkoreitJ., JonesL., GomezA.N., KaiserŁ. and PolosukhinI., Attention is all you need, In Advances in Neural Information Processing Systems (2017), 5998–6008.
5.
SutskeverI., VinyalsO. and LeQ.V., Sequence to Sequence Learning with Neural Networks[C]// NIPS. MIT Press (2014).
6.
BakerB., GuptaO., NaikN. and RaskarR., Designing neural network architectures using reinforcement learning, ICLR (2017).
7.
RealE., MooreS., SelleA., SaxenaS., SuematsuY.L., TanJ., LeQ.V. and KurakinA., Large-scale evolution of image classifiers, ICML (2017).
8.
LuoR., TianF., QinT. and LiuT.-Y., Neural Architecture Optimization, NIPS (2018).
9.
LiuH., SimonyanK. and YangY., DARTS: Differentiable architecture search, ICLR (2019).
10.
ZophB. and LeQ.V., Neural architecture search with reinforcement learning, ICLR (2017).
11.
PhamH., GuanM.Y., ZophB. and LeQ.V., Jeff Dean, Efficient Neural Architecture Search via Parameter Sharing.
12.
ZophB., VasudevanV., ShlensJ. and LeQ.V., Learning transferable architectures for scalable image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, (2018), 8697–8710.
RealE., AggarwalA., HuangY. and LeQ.V., Regularized evolution for image classifier architecture search, AAAI (2019).
15.
WeiT., WangC., RuiY. and ChenC.W., Networkmorphism, in: International Conference on Machine Learning (2016), 564–572.
16.
ShinR., PackerC. and SongD., Differentiable neural network architecture search.
17.
ChenX., XieL., WuJ. and TianQ., Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation, In ICCV, (2019a).
18.
ZelaA., ElskenT., SaikiaT., MarrakchiY., BroxT. and HutterF., Understanding and robustifying differentiable architecture search, In ICLR, (2020).
19.
ChuX., ZhouT., ZhangB. and LiJ., Fair darts: Eliminating unfair advantages in differentiable architecture search, ECCV (2020).
20.
ChuX., ZhangB. and LiX., Noisy differentiable architecture search, ICLR (2021).
21.
DongX. and YangY., Searching for a robust neural architecture in four gpu hours, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2019).
22.
SoD.R., LiangC. and LeQ.V., The evolved transformer, In Proceedings of the 36th International Conference on Machine Learning (ICML). (2019).
23.
WangC., LiM. and SmolaA.J., Language models with transformers, CoRR, abs/1904.09408. (2019).
24.
XiaY., TanX., TianF., GaoF., ChenW., FanY., GongL., LengY., LuoR., WangY., WuL., ZhuJ., QinT. and LiuT.-Y., Microsoft Research Asia’s Systems for WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy, Association for Computational Linguistics (2019).
25.
JiangY., HuC., XiaoT., ZhangC. and ZhuJ., Improved differentiable architecture search for language modeling and named entity recognition, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 3583–3588, Hong Kong, China. Association for Computational Linguistics. (2019).
