This paper proposes a Soft Actor-Critic algorithm with prioritized experience replay (SAC-PER) based on deep reinforcement learning, which is used for speed planning and energy management for fuel cell vehicles (FCV) through intersection scenarios. A FCV model is established, and two traffic scenarios with 6-signal and 12-signal intersections are built for scenario validation. By optimizing the speed and power allocation of the FCV, we can reduce start-stop events, lower energy consumption, and extend the fuel cell’s lifespan. In the constructed simulation scenario, a comparison with traditional hierarchical optimization methods shows that the SAC-PER algorithm strategy can reduce energy consumption by at least 48.32% and decrease fuel cell degradation by at least 38.75%. These findings are significant for improving energy utilization efficiency, reducing hydrogen consumption, and extending fuel cell lifespan in FCV.
TengTZhangXDongH, et al. A comprehensive review of energy management optimization strategies for fuel cell passenger vehicle. Int J Hydrogen Energy2020; 45: 20293–20303.
2.
YangHCZhangCGongX, et al. Energy management strategy of fuel cell hybrid electric vehicle based on dynamic programming. In: Chinese Automation Congress (CAC), 2020, pp.3134–3139. New York: IEEE.
3.
LuoYGLiSZhangSW, et al. Green light optimal speed advisory for hybrid electric vehicles. Mech Syst Signal Process2017; 87: 30–44.
4.
WangSYLinXK.Eco-driving control of connected and automated hybrid vehicles in mixed driving scenarios. Appl Energy2020; 271: 115233.
5.
GuoHQLuSLHuiHZ, et al. Receding horizon control-based energy management for plug-in hybrid electric buses using a predictive model of terminal SOC constraint in consideration of stochastic vehicle mass. Energy2019; 176: 292–308.
6.
GuoNYZhangXDZouY, et al. Real-time predictive energy management of plug-in hybrid electric vehicles for coordination of fuel economy and battery degradation. Energy2021; 214: 119070.
7.
Chan-ChiaoLHueiPGrizzleJW, et al. Power management strategy for a parallel hybrid electric truck. IEEE Trans Control Syst Technol2003; 11: 839–849.
8.
NamwookKSukwonCHueiP.Optimal control of hybrid electric vehicles based on Pontryagin’s Minimum Principle. IEEE Trans Control Syst Technol2011; 19: 1279–1287.
9.
WuJLZouYZhangXD, et al. A hierarchical energy management for hybrid electric tracked vehicle considering velocity planning with pseudospectral method. IEEE Trans Transp Electrification2020; 6: 703–716.
10.
HuoWWZhaoTYYangF, et al. An improved soft actor-critic based energy management strategy of fuel cell hybrid electric vehicle. J Energy Storage2023; 72: 108243.
11.
HuoWWChenDTianS, et al. Lifespan-consciousness and minimum-consumption coupled energy management strategy for fuel cell hybrid vehicles via deep reinforcement learning. Int J Hydrogen Energy2022; 47: 24026–24041.
12.
ZhengCHZhangDFXiaoY, et al. Reinforcement learning-based energy management strategies of fuel cell hybrid vehicles with multi-objective control. J Power Sources2022; 543: 231841.
13.
ZhangZZhangTZHongJC, et al. Energy management strategy of a novel parallel electric-hydraulic hybrid electric vehicle based on deep reinforcement learning and entropy evaluation. J Clean Prod2023; 403: 136800.
14.
HuBLiJX.A deployment-efficient energy management strategy for connected hybrid electric vehicle based on offline reinforcement learning. IEEE Trans Ind Electron2022; 69: 9644–9654.
15.
ShiJQQiaoFXLiQ, et al. Application and evaluation of the reinforcement learning approach to eco-driving at intersections under infrastructure-to-vehicle communications. Transp Res Rec2018; 2672: 89–98.
16.
LiuBSunCWangB, et al. Bi-level convex optimization of eco-driving for connected Fuel Cell Hybrid Electric Vehicles through signalized intersections. Energy2022; 252: 123956.
17.
LiJWuXDXuM, et al. Deep reinforcement learning and reward shaping based eco-driving control for automated HEVs among signalized intersections. Energy2022; 251: 123924.
18.
WeiXDSunCRenQ, et al. Hierarchical energy management of fuel cell vehicles through signalized intersections. In: 2021 IEEE Vehicle Power and Propulsion Conference (VPPC), Gijon, Spain, 25–28 October 2021. New York: IEEE.
19.
WeiXDWangJQSunC, et al. Guided control for plug-in fuel cell hybrid electric vehicles via vehicle to traffic communication. Energy2023; 267: 126469.
20.
MaGQGhasemiMSongXY.Integrated powertrain energy management and vehicle coordination for multiple connected hybrid electric vehicles. IEEE Trans Veh Technol2018; 67: 2893–2899.
21.
LiuYGLiJChenZ, et al. Research on a multi-objective hierarchical prediction energy management strategy for range extended fuel cell vehicles. J Power Sources2019; 429: 55–66.
22.
GarcíaPTorreglosaJPFernándezLM, et al. Viability study of a FC-battery-SC tramway controlled by equivalent consumption minimization strategy. Int J Hydrogen Energy2012; 37: 9368–9382.
23.
ZhangDCuiYXiaoY, et al. An improved soft actor-critic-based energy management strategy of fuel cell hybrid vehicles with a nonlinear fuel cell degradation model. Int J Precis Eng Manuf Green Technol2024; 11: 183–202.
24.
Olaverri-MonrealCErrea-MorenoJDíaz-AlvarezA, et al. Connection of the SUMO microscopic traffic simulator and the unity 3D game engine to evaluate V2X communication-based systems. Sensors (Basel)2018; 18(12): 4399.
25.
HaarnojaTTangHAbbeelP, et al. Reinforcement learning with deep energy-based policies. In: Proceedings of the 34th international conference on machine learning, vol. 70, 2017, pp.1352–1361. PMLR.
26.
SchulmanJChenXAbbeelP.Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:170406440, 2017.
27.
HaarnojaTZhouAAbbeelP, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th international conference on machine learning, vol. 80, 2018, pp.1861–1870. PMLR.
28.
HaarnojaTZhouAHartikainenK, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:181205905, 2018.
