Reinforcement learning-based methods are gaining popularity in energy management strategy (EMS) for hybrid electric vehicles (HEVs). Most of the literature in this area focuses on pure simulation with limited scenarios, while real-time control through embedded implementation is still limited. In this paper, based on the basic DQN algorithm, design of mode transition control strategy, parameterized functions, and directed Wasserstein distance, the embedded scenario-adaptive D4QN-EMS to control series-parallel multi-speed (SPMS) HEVs is established. MIL and HIL tests on the strategy using standard driving cycles and nine driving cycles representing characteristics of various Chinese cities are conducted. The results show that the proposed EMS can adaptively update the strategy according to the characteristic of actual driving scenarios without frequent mode switching, demonstrating excellent performance in both transient performance and fuel economy of the SPMS HEV. Across all test scenarios, the average total fuel consumption of the A-D4QN-EMS (4.52 L/100 km) improved by 9.73%, 4.21%, and 2.43% compared to the RB-EL (4.96 L/100 km), RB-PL (4.71 L/100 km), and D4QN (4.63 L/100 km) strategies, respectively.
SinghSKulshresthaMJRaniN, et al. An overview of vehicular emission standards. Mapan2023; 38(1): 241–263.
2.
AnselmaPGBelingardiG.Next generation HEV powertrain design tools: roadmap and challenges. SAE Technical Paper. Report no.: 0148-7191, 2019. SAE International.
3.
ZhangFHuXLangariR, et al. Energy management strategies of connected HEVs and PHEVs: recent progress and outlook. Prog Energy Combust Sci2019; 73: 235–256.
4.
MalikopoulosAA.Supervisory power management control algorithms for hybrid electric vehicles: a survey. IEEE Trans Intell Transp Syst2014; 15(5): 1869–1885.
5.
TranD-DVafaeipourMEl BaghdadiM, et al. Thorough state-of-the-art analysis of electric and hybrid vehicle powertrains: topologies and integrated energy management strategies. Renew Sustain Energy Rev2020; 119: 109596.
6.
WuYKLianRZWangY, et al. (eds.). Benchmarking deep reinforcement learning based energy management systems for hybrid electric vehicles. In: 2nd CAAI international conference on artificial intelligence (CICAI), 27–28 August 2022. Revised Selected Papers. Lecture Notes in Computer Science, Lecture Notes in Artificial Intelligence (13605), pp. 613–625. Beijing: Chinese Assoc Artificial Intelligence.
7.
LinXWangYBogdanP, et al. (eds.). Reinforcement learning based power management for hybrid electric vehicles. In: 2014 IEEE/ACM international conference on computer-aided design (ICCAD), 2014. New York, NY: IEEE.
8.
XuBRathodDZhangD, et al. Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle. Appl Energy2020; 259: 114200.
9.
François-LavetVHendersonPIslamR, et al. An introduction to deep reinforcement learning. Found Trends Mach Learn2018; 11(3–4): 219–354.
10.
WuJHeHPengJ, et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Appl Energy2018; 222: 799–811.
11.
DuGZouYZhangX, et al. Deep reinforcement learning based energy management for a hybrid electric vehicle. Energy2020; 201: 117591.
12.
QiXWuGBoriboonsomsinK, et al. Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles. Transp Res Rec2016; 2572(1): 1–8.
13.
LiuCMurpheyYL (eds.). Power management for plug-in hybrid electric vehicles using reinforcement learning with trip information. In: 2014 IEEE transportation electrification conference and expo (ITEC), 2014. New York, NY: IEEE.
14.
LiuTHuXLiSE, et al. Reinforcement learning optimized look-ahead energy management of a parallel hybrid electric vehicle. IEEE/ASME Trans Mechatron2017; 22(4): 1497–1507.
15.
WuJZouYZhangX, et al. An online correction predictive EMS for a hybrid electric tracked vehicle based on dynamic programming and reinforcement learning. IEEE Access2019; 7: 98252–98266.
16.
TangXLChenJXQinYC, et al. Reinforcement learning-based energy management for hybrid power systems: state-of-the-art survey, review, and perspectives. Chin J Mech Eng2024; 37(1): 16–35.
17.
HuYXuHJiangZ, et al. Supplementary learning control for energy management strategy of hybrid electric vehicles at scale. IEEE Trans Veh Technol2023; 72(6): 7290–7303.
18.
HuDXieHSongK, et al. An apprenticeship-reinforcement learning scheme based on expert demonstrations for energy management strategy of hybrid electric vehicles. Appl Energy2023; 342: 121227.
19.
QiCSongCXiaoF, et al. Generalization ability of hybrid electric vehicle energy management strategy based on reinforcement learning method. Energy2022; 250: 123826.
20.
BoLHanLXiangC, et al. A real-time energy management strategy for off-road hybrid electric vehicles based on the expected SARSA. Proc Inst Mech Eng D J Automob Eng2023; 237(2–3): 362–380.
21.
QiXLuoYWuG, et al. Deep reinforcement learning enabled self-learning control for energy efficient driving. Transp Res C Emerg Technol2019; 99: 67–81.
22.
ZhouJXueSXueY, et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning. Energy2021; 224: 120118.
23.
TanHZhangHPengJ, et al. Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space. Energy Convers Manag2019; 195: 548–560.
24.
SathreJ.Architectural support for secure and survivable embedded software. Ames, IA: Iowa State University, 2008.
25.
FengJZhongHAoG, et al. (eds.). Principles and application of the real-time hardware-in-the-loop simulation platform based on multi-thread and CAN. In: 2008 IEEE international symposium on industrial electronics, 2008. New York, NY: IEEE.
26.
BertsekasD.Reinforcement learning and optimal control. Nashua, NH: Athena Scientific, 2019.
27.
LiHWeiTRenA, et al. (eds.). Deep reinforcement learning: framework, applications, and embedded implementations. In: 2017 IEEE/ACM international conference on computer-aided design (ICCAD), 2017. New York, NY: IEEE.
28.
LuoFHuangZ (eds.). Embedded C code generation and embedded target development based on RTW-EC. In: 2010 third international conference on computer science and information technology, 2010. New York, NY: IEEE.
29.
GaneshAHXuB.A review of reinforcement learning based energy management systems for electrified powertrains: progress, challenge, and potential solution. Renew Sustain Energy Rev2022; 154: 111833.
30.
YeYXuBZhangJ, et al. (eds.). Reinforcement learning-based energy management system enhancement using digital twin for electric vehicles. In: 2022 IEEE vehicle power and propulsion conference (VPPC), 2022. New York, NY: IEEE.
31.
GlaessgenEStargelD (eds.). The digital twin paradigm for future NASA and US Air Force vehicles. In: 53rd AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics and materials conference 20th AIAA/ASME/AHS adaptive structures conference 14th AIAA, 2012. American Institute of Aeronautics and Astronautics (AIAA): Honolulu, Hawaii.
32.
ZhangZZouYZhouT, et al. Energy consumption prediction of electric vehicles based on digital twin technology. World Electr Veh J2021; 12(4): 160.
33.
TosoFTorchioRFavatoA, et al. (eds.). Digital twins as electric motor soft-sensors in the automotive industry. In: 2021 IEEE international workshop on metrology for automotive (MetroAutomotive), 2021. New York, NY: IEEE.
34.
XieYLiYZhaoZ, et al. Microsimulation of electric vehicle energy consumption and driving range. Appl Energy2020; 267: 115081.
35.
LouDZhaoYFangL, et al. Energy management based on D4QN reinforcement learning for a series-parallel multi-speed hybrid electric vehicle. SAE Technical Paper. Report no.: 0148-7191, 2023. Shanghai, China: SAE International.
36.
BullockDJohnsonBWellsRB, et al. Hardware-in-the-loop simulation. Transp Res C Emerg Technol2004; 12(1): 73–89.
37.
LamboraAGuptaKChopraK (eds.). Genetic algorithm – a literature review. In: 2019 international conference on machine learning, big data, cloud and parallel computing (COMITCon), 2019. New York, NY: IEEE.
38.
PanaretosVMZemelY.Statistical aspects of Wasserstein distances. Annu Rev Stat Appl2019; 6: 405–431.
39.
RahmanMSWagnerAB.On the optimality of binning for distributed hypothesis testing. IEEE Trans Inf Theory2012; 58(10): 6282–6303.
