For patients diagnosed with a chronic condition, appropriately timing treatment is critically important. Many conditions, such as age-related macular degeneration (AMD), have a maximum safe treatment interval (MSTI) in which treatment is required to prevent disease progression. We introduce a Markov decision process (MDP) with an ordinal action space as a way to quickly and safely identify the MSTI for an individual patient. The MDP’s ordinal action space allows users to update the expected outcomes of multiple actions with only a single decision. We solve the MDP and show how the expected outcomes of each action can be presented to clinicians as a menu of treatment options. We describe conditions under which the MDP is guaranteed to find the MSTI under uncertainty. We illustrate the model’s effectiveness through an application to AMD, and show that following the MDP can reduce patient exposure to symptoms by 38% and find the optimal interval 7% faster when compared to current clinical protocols.
AhujaVBirgeJR (2016) Response-adaptive designs for clinical trials: Simultaneous learning from multiple patients. European Journal of Operational Research248(2): 619–633.
2.
AlagozOMaillartLMSchaeferAJ, et al. (2004) The optimal timing of living-donor liver transplantation. Management Science50(10): 1420–1430.
3.
AyerTAlagozOStoutNK (2012) OR forum-A POMDP approach to personalize mammography screening decisions. Operations Research60(5): 1019–1034.
4.
AyerTZhangCBonifonteA, et al. (2019) Prioritizing hepatitis C treatment in U.S. prisons. Operations Research67(3): 853–873.
5.
BatunSSchaeferAJBhandariA, et al. (2018) Optimal liver acceptance for risk-sensitive patients. Service Science10(3): 320–333. DOI: https://doi.org/10.1287/serv.2018.0215.
6.
BergKPedersenTRSandvikL, et al. (2015) Comparison of ranibizumab and bevacizumab for neovascular age-related macular degeneration according to LUCAS treat-and-extend protocol. Ophthalmology122(1): 146–152. DOI: https://doi.org/10.1016/J.OPHTHA.2014.07.041.
7.
BogunovicHAbramoffMDZhangL, et al. (2017) Prediction of treatment response from retinal OCT in patients with exudative age-related macular degeneration. Ophthalmic Medical Image Analysis First International Workshop1: 129–136. DOI: https://doi.org/10.17077/omia.1018.
8.
BrownGCBrownMMRapuanoSB, et al. (2021) A cost-benefit analysis of VEGF-inhibitor therapy for neovascular age-related macular degeneration in the United States. American Journal of Ophthalmology223: 405–429. DOI: https://doi.org/10.1016/J.AJO.2020.07.010.
9.
Centers for Medicare and Medicaid Services (2021) Physician fee schedule. https://www.cms.gov/medicare/physician-fee- schedule/.
10.
ChenQAyerTChhatwalJ (2018) Optimal M-switch surveillance policies for liver cancer in a hepatitis C-infected population. Operations Research66(3): 673–696. DOI: https://doi.org/10.1287/opre.2017.1706.
11.
DaiTTayurS (2021) Healthcare operations management: A snapshot of emerging research. Manufacturing and Service Operations Management22(5): 869–887. DOI: https://doi.org/10.1287/msom.2019.0778.
12.
DentonBT (2018) Optimization of sequential decision making for chronic diseases: From data to decisions. Recent Advances in Optimization and Modeling of Contemporary Problems1: 316–348. DOI: https://doi.org/10.1287/educ.2018.0184.
13.
FauserSSchwabeckerVMuetherPS (2014) Suppression of intraocular vascular endothelial growth factor during aflibercept treatment of age-related macular degeneration. American Journal of Ophthalmology158(3): 532–536. DOI: https://doi.org/10.1016/j.ajo.2014.05.025.
14.
FreundKBKorobelnikJFDevenyiR, et al. (2015) Treat-and-extend regimens with anti-VEGF agents in retinal diseases. Retina35(8): 1489–1506. DOI: https://doi.org/10.1097/IAE.0000000000000627.
15.
GuptaOPShienbaumGPatelAH, et al. (2010) A treat and extend regimen using ranibizumab for neovascular age-related macular degeneration: Clinical and economic impact. Ophthalmology117(11): 2134–2140. DOI: https://doi.org/10.1016/j.ophtha.2010.02.032.
HelmJELavieriMSVan OyenMP, et al. (2015) Dynamic forecasting and control algorithms of glaucoma progression for clinician decision support. Operations Research63(5): 979–999. DOI: https://doi.org/10.1287/opre.2015.1405.
18.
HolekampNDuffSRajputY (2016) Quality-adjusted life years for the retinal specialist. Retina Today3(3): 1–2.
19.
HörsterRRistauTSaddaSR, et al. (2011) Individual recurrence intervals after anti-VEGF therapy for age-related macular degeneration. Graefe’s Archive for Clinical and Experimental Ophthalmology249(5): 645–652. DOI: https://doi.org/10.1007/s00417-010-1588-2.
20.
KamalzadehHAhujaVHahslerM, et al. (2021) An analytics-driven approach for optimal individualized diabetes screening. Production and Operations Management30: 3161–3191.
21.
LeeELavieriMSVolkM (2019) Optimal screening for hepatocellular carcinoma: A restless bandit model. Manufacturing and Service Operations Management21(1): 198–212.
22.
MantelIDeliAIglesiasK, et al. (2013) Prospective study evaluating the predictability of need for retreatment with intravitreal ranibizumab for age-related macular degeneration. Graefe’s Archive for Clinical and Experimental Ophthalmology251(3): 697–704.
23.
NegoescuDMBimpikisKBrandeauML, et al. (2018) Dynamic learning of patient response types: An application to treating chronic diseases. Management Science64(8): 3469–3488.
24.
NgSPAjayiTSchaeferAJ, et al. (2020) Surveillance imaging for patients with head and neck cancer treated with definitive radiotherapy: A partially observed Markov decision process model. Cancer126(4): 749–756. DOI: https://doi.org/10.1002/cncr.32597.
25.
PayneJFWykoffCCClarkWL, et al. (2017) Randomized trial of treat and extend ranibizumab with and without navigated laser for diabetic macular edema: TREX-DME 1 year outcomes. Ophthalmology124(1): 74–81.
26.
PourchoAMSmithJWisniewskiSJ, et al. (2014) Intraarticular platelet-rich plasma injection in the treatment of knee osteoarthritis: Review and recommendations. American Journal of Physical Medicine and Rehabilitation93(11): S108–S121. DOI: https://doi.org/10.1097/PHM.0000000000000115.
27.
RohmMTrespVMüllerM, et al. (2018) Predicting visual acuity by using machine learning in patients treated for neovascular age-related macular degeneration. Ophthalmology125(7): 1028–1036. DOI: https://doi.org/10.1016/j.ophtha.2017.12.034.
28.
SabouriAHuhWTShechterSM (2017) Screening strategies for patients on the kidney transplant waiting list. Operations Research65(5): 1131–1146. DOI: https://doi.org/10.1287/opre.2017.1632.
29.
SchaeferAJBaileyMDShechterSM, et al. (2006) Modeling medical treatment using Markov decision processes. Operations Research and Health Care70: 593–612. DOI: https://doi.org/10.1007/1-4020-8066-2_23.
30.
SchellGJLavieriMSHelmJE, et al. (2014) Using filtered forecasting techniques to determine personalized monitoring schedules for patients with open-angle glaucoma. Ophthalmology121(8): 1539–1546.
31.
SchleglTWaldsteinSMBogunovicH, et al. (2018) Fully automated detection and quantification of macular fluid in OCT using deep learning. Ophthalmology125(4): 549–558. DOI: https://doi.org/10.1016/j.ophtha.2017.10.031.
32.
Schmidt-ErfurthUBogunovicHSadeghipourA, et al. (2018) Machine learning to analyze the prognostic value of current imaging biomarkers in neovascular age-related macular degeneration. Ophthalmology Retina2(1): 24–30. DOI: https://doi.org/10.1016/j.oret.2017.03.015.
33.
ShechterSMBaileyMDSchaeferAJ, et al. (2008) The optimal time to initiate HIV therapy under ordered health states. Operations Research56(1): 20–33.
SongCHSternSGiruparajahM, et al. (2013) Long-term efficacy of fixed-dose omalizumab for patients with severe chronic spontaneous urticaria. Annals of Allergy, Asthma and Immunology110(2): 113–117. DOI: https://doi.org/10.1016/j.anai.2012.11.022.
36.
SteimleLNDentonBT (2017) Markov decision processes for screening and treatment of chronic diseases. International Series in Operations Research and Management Science248: 189–222. Springer New York LLC. https://doi.org/10.1007/978-3-319-47766-4_6.
37.
LanzettaPLoewensteinA, The Vision Academy Steering Committee (2017) Fundamental principles of an anti-VEGF treatment regimen: Optimal application of intravitreal anti-vascular endothelial growth factor therapy of macular diseases. Graefe’s Archive for Clinical and Experimental Ophthalmology255(7): 1259. DOI: https://doi.org/10.1007/S00417-017-3647-4.
38.
WykoffCCOuWCBrownDM, et al. (2017) Randomized trial of treat-and-extend versus monthly dosing for neovascular age-related macular degeneration: 2-year results of the TREX-AMD study. Ophthalmology Retina1(4): 314–321.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
