This review provides a timely exploration of several novel neural network (NN) architectures and learning methods, following a concise overview of the fundamentals of NNs and some important associated challenges. There are many benefits to using NNs, including deep learning models, in scientific research and, by understanding novel techniques better suited to certain applications, this benefit can be maximised. Finally, a few developed and emerging alternative learning paradigms are surveyed for their potential benefit to future research. The reviewed literature and accompanying discussion are of generic value well beyond steel metallurgy, and there is much to be gained from assessing methods used in other areas of materials science and further afield in order to apply them to steel metallurgy.
BhadeshiaHKDH.Neural networks in materials science. ISIJ Int. 1999; 39(10):966–979.
4.
BhadeshiaHKDH.Neural Networks and Information in Materials Science. Stat Anal Data Min. 2009 Apr;1(5):296–305. Available from: http://doi.wiley.com/10.1002/sam.10018
5.
KrizhevskyASutskeverIHintonGE.ImageNet classification with deep convolutional neural networks. Commun ACM. 2017 May;60(6):84–90. Available from: https://doi.org/10.1145/3065386
6.
SimonyanKZissermanA. Very deep convolutional networks for large-scale image recognition. arXiv:14091556 [cs]. 2015 Apr. Available from: http://arxiv.org/abs/1409.1556
7.
ZeHSeniorASchusterM. Statistical parametric speech synthesis using deep neural networks. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing; 2013. p. 7962–7966. ISSN: 2379-190X.
8.
FanYQianYXieFL, et al. TTS synthesis with bidirectional LSTM based recurrent neural networks. INTERSPEECH; 2014.
9.
ZenHSakH. Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2015. p. 4470–4474. ISSN: 2379-190X.
OdejobiOAUmoruLE. Applications of soft computing techniques in materials engineering: a review. p. 28.
12.
LaghariRALiJLaghariAA, et al. A review on application of soft computing techniques in machining of particle reinforcement metal matrix composites. Arch Comput Meth Eng. 2020; 27:1363–1377. Available from: http://link.springer.com/10.1007/s11831-019-09340-0
Faghih-RoohiSHajizadehSNúñezA, et al. Deep convolutional neural networks for detection of rail surface defects. 2016 International Joint Conference on Neural Networks (IJCNN); 2016. p. 2584–2589. ISSN: 2161–4407.
15.
MasciJMeierUCiresanD, et al. Steel defect classification with Max-Pooling Convolutional Neural Networks. The 2012 International Joint Conference on Neural Networks (IJCNN); 2012. p. 1–6. ISSN: 2161–4393.
16.
DimidukDMHolmEANiezgodaSR.Perspectives on the impact of machine learning, deep learning, and artificial intelligence on materials, processes, and structures engineering. Integr Mater Manuf Innov. 2018 Sep;7(3):157–172. Available from: https://doi.org/10.1007/s40192-018-0117-8
17.
KuscheCReclikTFreundM, et al. Large-area, high-resolution characterisation and classification of damage mechanisms in dual-phase steel using deep learning. PLoS One. 2019 May;14(5):e0216493. Available from: http://dx.plos.org/10.1371/journal.pone.0216493
YescasMA.Prediction of the Vickers hardness in austempered ductile irons using neural networks. Int J Cast Metals Res. 2003 May;15(5):513–521. Available from: https://doi.org/10.1080/13640461.2003.11819537
GrangeRAHribalCRPorterLF.Hardness of tempered martensite in carbon and low-alloy steels. Met Trans A. 1977 Nov;8(11):1775–1785. Available from: https://doi.org/10.1007/BF02646882
22.
HäggbladH OldenburgM.Modelling and simulation of metal powder die pressing with use of explicit time integration. Model Simul Mater Sci Eng. 1994 Jul;2(4):893–911. Available from: https://doi.org/10.10882/F0965-03932F22F42F006
23.
KafkasFKarataşÇSozenA, et al. Determination of residual stresses based on heat treatment conditions and densities on a hybrid (FLN2-4405) powder metallurgy steel using artificial neural network. Mater Des. 2007 Jan;28(9):2431–2442. Available from: http://www.sciencedirect.com/science/article/pii/S0261306906002494
RahamanMMuWOdqvistJ, et al. Machine learning to predict the martensite start temperature in steels. Met Mater Trans A. 2019 May;50(5):2081–2091. Available from: https://doi.org/10.1007/s11661-019-05170-8
BlundellCCornebiseJKavukcuogluK, et al. Weight uncertainty in neural networks. arXiv:150505424 [cs, stat]. 2015 May. Available from: http://arxiv.org/abs/1505.05424
29.
PearceTLeibfriedFBrintrupA, et al. Uncertainty in neural networks: approximately Bayesian ensembling. arXiv:181005546 [cs, stat]. 2019 Oct. Available from: http://arxiv.org/abs/1810.05546
MorrisKCSchlenoffCSrinivasanV.Guest editorial: A remarkable resurgence of artificial intelligence and its impact on automation and autonomy. IEEE Trans Autom Sci Eng. 2017 Apr;14(2):407–409. IEEE Transactions on Automation Science and Engineering.
32.
MarcusG. Deep learning: a critical appraisal. arXiv:180100631 [cs, stat]. 2018 Jan. Available from: http://arxiv.org/abs/1801.00631
MooreSWFoxPJMarshSJT, et al. Bluehive – a field-programable custom computing machine for extreme-scale real-time neural network simulation. 2012 IEEE 20th International Symposium on Field-Programmable Custom Computing Machines; 2012. p. 133–140.
37.
JungSKimSs.Hardware implementation of a real-time neural network controller with a DSP and an FPGA for nonlinear systems. IEEE Trans Indust Electron. 2007 Feb;54(1):265–271. IEEE Transactions on Industrial Electronics.
38.
HawkinsDM.The Problem of Overfitting. J Chem Inf Comput Sci. 2004 Jan;44(1):1–12. American Chemical Society. Available from: https://doi.org/10.1021/ci0342472
39.
SrivastavaNHintonGKrizhevskyA, et al. Dropout: a simple way to prevent neural networks from overfitting. J Machine Learn Res. 2014 Jan;15(1):1929–1958.
40.
BianchiniMScarselliF.On the complexity of neural network classifiers: a comparison between shallow and deep architectures. IEEE Trans Neural Netw Learn Syst. 2014 Aug;25(8):1553–1565. IEEE Transactions on Neural Networks and Learning Systems.
41.
BornscheinJVisinFOsinderoS. Small data, big decisions: model selection in the small-data regime. p. 10.
42.
PanSJYangQ.A survey on transfer learning. IEEE Trans Knowl Data Eng. 2010 Oct;22(10):1345–1359. IEEE Transactions on Knowledge and Data Engineering.
43.
ShinHCRothHRGaoM, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer Learning. IEEE Trans Med Imaging. 2016 May;35(5):1285–1298. Available from: http://ieeexplore.ieee.org/document/7404017/
44.
ShortenCKhoshgoftaarTM.A survey on image data augmentation for deep learning. J Big Data. 2019 Jul;6(1):60. Available from: https://doi.org/10.1186/s40537-019-0197-0
45.
WongSCGattAStamatescuV, et al. Understanding data augmentation for classification: When to Warp? 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA); 2016. p. 1–6.
46.
HubelDHWieselTN.Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. (Lond.). 1962 Jan;160(1):106–154. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1359523/
47.
LecunYBottouLBengioY, et al. Gradient-based learning applied to document recognition. Proc IEEE. 1998 Nov;86(11):2278–2324.
48.
CangRLiHYaoH, et al. Improving direct physical properties prediction of heterogeneous materials from imaging data via convolutional neural network and a morphology-aware generative model. Comput Mater Sci. 2018 Jul;150:212–221. Available from: http://www.sciencedirect.com/science/article/pii/S0927025618302337
49.
LiXDanYDongR, et al. Computational screening of new perovskite materials using transfer learning and deep learning. Appl Sci. 2019 Jan;9(24):5510. Number: 24 Publisher: Multidisciplinary Digital Publishing Institute. Available from: https://www.mdpi.com/2076-3417/9/24/5510
50.
JordanMI. Attractor dynamics and parallelism in a connectionist sequential machine. Artificial neural networks: concept learning. IEEE Press; 1990. p. 112–127.
51.
ElmanJL.Finding structure in time. Cogn Sci. 1990; 14:179–211.
52.
PearlmutterX. Learning state space trajectories in recurrent neural networks. International 1989 Joint Conference on Neural Networks; Vol. 2, 1989. p. 365–372 ISSN: null.
ChoKvan MerriënboerBGulcehreC, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics; 2014. p. 1724–1734. Available from: https://www.aclweb.org/anthology/D14-1179
VeeramaniASCrewsJHBucknerGD.Hysteretic recurrent neural networks: a tool for modeling hysteretic materials and systems. Smart Mater Struct. 2009 Jun;18(7):075004. Available from: https://doi.org/10.1088/2F0964-17262F182F72F075004
MozaffarMPaulAAl-BahraniR, et al. Data-driven prediction of the high-dimensional thermal history in directed energy deposition processes via recurrent neural networks. Manuf Lett. 2018 Oct;18:35–39. Available from: http://www.sciencedirect.com/science/article/pii/S2213846318300804
62.
VenkateshVRackHJ.A neural network approach to elevated temperature creep-fatigue life prediction. Int J Fatigue. 1999; 21:225–234.
63.
PleuneTTChopraOK.Using artificial neural networks to predict the fatigue life of carbon and low-alloy steels. Nuclear Eng Design. 2000; 197:1–12.
64.
LowdDMeekC. Adversarial learning. Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining. KDD ‘05. Chicago (IL): Association for Computing Machinery; 2005. p. 641–647. Available from: https://doi.org/10.1145/1081870.1081950
65.
HuangLJosephADNelsonB, et al. Adversarial machine learning. Proceedings of the 4th ACM workshop on Security and artificial intelligence - AISec ‘11. Chicago (IL): ACM Press; 2011. p. 43. Available from: http://dl.acm.org/citation.cfm?doid=2046684.2046692
66.
DalviNDomingosPVermaD. Adversarial classification. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. KDD ‘04. Seattle (WA): Association for Computing Machinery; 2004. p. 99–108. Available from: https://doi.org/10.1145/1014052.1014066
67.
GoodfellowIJPouget-AbadieJMirzaM, et al. Generative Adversarial Networks. arXiv:14062661 [cs, stat]. 2014 Jun. version: 1. Available from: http://arxiv.org/abs/1406.2661
68.
KarrasTLaineSAilaT. A style-based generator architecture for generative adversarial networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2019. p. 4396–4405. ISSN: 1063–6919.
ShrivastavaAPfisterTTuzelO, et al. Learning from simulated and unsupervised images through adversarial training. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017. p. 2242–2251. ISSN: 1063–6919.
LiuZZhuDRodriguesSP, et al. Generative model for the inverse design of metasurfaces. Nano Lett. 2018 Oct;18(10):6570–6576. Available from: https://doi.org/10.1021/acs.nanolett.8b03171
73.
XieYFranzEChuM, et al. tempoGAN: a temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Trans Graphics (TOG).. 2018; 37(4):95–00.
74.
MiikkulainenBFaR. Using marker-based genetic encoding of neural networks to evolve finite-state behaviour. 1991. Available from: http://nn.cs.utexas.edu/?fullmer:ecal91
StanleyKOMiikkulainenR.Evolving neural networks through augmenting topologies. Evol Comput. 2002 Jun;10(2):99–127. MIT Press. Available from: https://doi.org/10.1162/106365602320169811
83.
RoySChakrabortiN. Development of an evolutionary deep neural net for materials research. TMS 2020 149th Annual Meeting & Exhibition Supplemental Proceedings. The Minerals, Metals & Materials Series. Cham: Springer International Publishing; 2020. p. 817–828.
84.
DebK.Multi-objective optimization using evolutionary algorithms. Hoboken: Wiley; 2001.
PetterssonFSuhCSaxénH, et al. Analyzing sparse data for nitride spinels using data mining, neural networks, and multiobjective genetic algorithms. Mater Manuf Process. 2008 Dec;24(1):2–9. Available from: https://www.tandfonline.com/doi/full/10.1080/10426910802539762
92.
PetterssonFBiswasASenPK, et al. Analyzing leaching data for low-grade manganese ore using neural nets and multiobjective genetic algorithms. Mater Manuf Process. 2009 Feb;24(3):320–330. Available from: http://www.tandfonline.com/doi/abs/10.1080/10426910802679386
MnihVKavukcuogluKSilverD, et al. Playing atari with deep reinforcement learning. arXiv:13125602 [cs]. 2013 Dec. Available from: http://arxiv.org/abs/1312.5602
96.
SilverDHubertTSchrittwieserJ, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Sci. 2018 Dec;362(6419):1140–1144. American Association for the Advancement of Science Section: Report. Available from: https://science.sciencemag.org/content/362/6419/1140
97.
SilverDHuangAMaddisonCJ, et al. Mastering the game of Go with deep neural networks and tree search. Nature. 2016 Jan;529(7587):484–489. Number: 7587 Publisher: Nature Publishing Group; Available from: https://www.nature.com/articles/nature16961
98.
VinyalsOBabuschkinICzarneckiWM, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature. 2019 Nov;575(7782):350–354. Number: 7782 Publisher: Nature Publishing Group; Available from: https://www.nature.com/articles/s41586-019-1724-z
DaleMMillerJFStepneyS. Reservoir computing as a model for in-materio computing. In: Adamatzky A, editor, Advances in unconventional computing. Cham: Springer International Publishing; 2017. p. 533–571. Series Title: (Emergence, Complexity and Computation; 22). Available from: http://link.springer.com/10.1007/978-3-319-33924-5-22
DuCCaiFZidanMA, et al. Reservoir computing using dynamic memristors for temporal information processing. Nat Commun. 2017 Dec;8(1):2204. Number: 1 Publisher: Nature Publishing Group; Available from: https://www.nature.com/articles/s41467-017-02337-y
104.
MoonJMaWShinJH, et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nat Electron. 2019 Oct;2(10):480–487. Number: 10 Publisher: Nature Publishing Group; Available from: https://www.nature.com/articles/s41928-019-0313-3
