In this paper, we provide a unified presentation of the Configurable Markov Decision Process (Conf-MDP) framework. A Conf-MDP is an extension of the traditional Markov Decision Process (MDP) that models the possibility to configure some environmental parameters. This configuration activity can be carried out by the learning agent itself or by an external configurator. We introduce a general definition of Conf-MDP, then we particularize it for the cooperative setting, where the configuration is fully functional to the agent’s goals, and non-cooperative setting, in which agent and configurator might have different interests. For both settings, we propose suitable solution concepts. Furthermore, we illustrate how to extend the traditional value functions for MDPs and Bellman operators to this new framework.
Peter Auer, Nicolò Cesa-Bianchi and Paul Fischer,
Finite-time analysis of the multiarmed bandit problem, Mach. Learn.47(2-3) (2002), 235–256.
2.
Richard Bellman, The theory of dynamic programming, Bulletin of the American Mathematical Society60(6) (1954), 503–515.
3.
A. Bensoussan and J. Frehse, Stochastic games for n players, Journal of Optimization Theory and Applications105(3) (2000), 543–565.
4.
Krishnendu Chatterjee, Rupak Majumdar and Marcin Jurdzinski,
On Nash Equilibria in Stochastic Games. In Computer Science Logic, 18th International Workshop (CSL), pages 26–40, 2004.
5.
Kamil Andrzej Ciosek and Shimon Whiteson, OFFER: off-environment reinforcement learning, In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 1819–1825, AAAI Press, 2017.
6.
Vincent Conitzer and Tuomas Sandholm, Computing the optimal strategy to commit to, In Proceedings 7th ACM Conference on Electronic Commerce (EC), pages 82–90. ACM, 2006.
7.
Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren and Qionghai Dai, Deep direct reinforcement learning for financial signal representation and trading, IEEE Trans. Neural Networks Learn. Syst.28(3) (2017), 653–664.
8.
Pierluca D’Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini and Marcello Restelli,
Gradient-aware model-based policy search, In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), pages 3801–3808, AAAI Press, 2020.
9.
Zehao Dou, Zhuoran Yang, Zhaoran Wang and Simon S. Du,
Gap-dependent bounds for two-player markov games, CoRR, abs/2107.00685, 2021.
10.
Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker and Sergey Levine, Learning to walk via deep reinforcement learning, In Robotics: Science and Systems XV, University of Freiburg, 2019.
11.
Sarah Keren, Luis Enrique Pineda, Avigdor Gal, Erez Karpas and Shlomo Zilberstein,
Equi-reward utility maximizing design in stochastic environments, In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI), pages 4353–4360. ijcai.org, 2017.
12.
Erwan Lecarpentier and Emmanuel Rachelson,
Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019)), pages 7214–7223, 2019.
13.
Boris Lesner and Bruno Scherrer,
Non-stationary approximate modified policy iteration, In Proceedings of the 32nd International Conference on Machine Learning (ICML)), volume 37, pages 1567–1575. JMLR.org, 2015..
14.
Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli and Danilo Romano,
Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving, Robotics and Autonomous Systems131 (2020), 103568.
15.
Littman MichaelL., Markov games as a framework for multi-agent reinforcement learning, In Proceedings of the Eleventh International Conference (ICML)), pages 157–163. Morgan Kaufmann, 1994.
16.
Yuzhe Ma, Xuezhou Zhang, Wen Sun and Jerry Zhu,
Policy poisoning in batch reinforcement learning and control, In Advances in Neural Information Processing Systems 32 (NeurIPS)), pages 14543–14553, 2019.
17.
Alberto Maria Metelli, Exploiting environment configurability in reinforcement learning, PhD thesis, Politecnico di Milano, March 2021.
18.
Alberto Maria Metelli,
Configurable environments in reinforcement learning: An overview, In Luigi Piroddi, editor, Special Topics in Information Technology, pages 101–113, Cham, 2022. Springer International Publishing.
19.
Alberto Maria Metelli, Emanuele Ghelfi and Marcello Restelli,
Reinforcement learning in configurable continuous environments, In Proceedings of the 36th International Conference on Machine Learning, (ICML), volume 97, pages 4546–4555. PMLR, 2019.
20.
Alberto Maria Metelli, Amarildo Likmeta and Marcello Restelli,
Propagating uncertainty in reinforcement learning via wasserstein barycenters, In Advances in Neural Information Processing Systems 32 (NeurIPS)), pages 4335–4347, 2019.
21.
Alberto Maria Metelli, Guglielmo Manneschi and Marcello Restelli,
Policy space identification in configurable environments, Machine Learning111(6) (2022), 2093–2145.
22.
Alberto Maria Metelli, Flavio Mazzolini, Lorenzo Bisi, Luca Sabbioni and Marcello Restelli,
Control frequency adaptation via action persistence in batch reinforcement learning, In Proceedings of the 37th International Conference on Machine Learning (ICML)), volume 119, pages 6862–6873. PMLR, 2020.
23.
Alberto Maria Metelli, Mirco Mutti and Marcello Restelli,
Configurable Markov decision processes, In Proceedings of the 35th International Conference on Machine Learning (ICML)), volume 80, pages 3488–3497. PMLR, 2018.
24.
Alberto Maria Metelli, Matteo Papini, Pierluca D’Oro and Marcello Restelli,
Policy optimization as online learning with mediator feedback, In The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), pages 8958–8966.
AAAI Press, 2021.
25.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio and Marcello Restelli,
Policy optimization via importance sampling, In Advances in Neural Information Processing Systems 31 (NeurIPS)), pages 5447–5459, 2018.
26.
Alberto Maria Metelli, Matteo Papini, Nico Montali and Marcello Restelli,
Importance sampling techniques for policy optimization, Journal of Machine Learning Research21(141) (2020), 1–75.
27.
Alberto Maria Metelli, Matteo Pirotta and Marcello Restelli,
On the use of the policy gradient and hessian in inverse reinforcement learning, Intelligenza Artificiale14(1) (2020), 91–124.
28.
Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti and Marcello Restelli,
Provably efficient learning of transferable rewards, In Proceedings of the 38th International Conference on Machine Learning (ICML) ), volume 139, pages 7665–7676. PMLR, 2021.
29.
Alberto Maria Metelli, Alessio Russo and Marcello Restelli,
Subgaussian and differentiable importance sampling for off-policy evaluation and learning, In Advances in Neural Information Processing Systems 34 (NeurIPS). 2021.
30.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg and Demis Hassabis,
Human-level control through deep reinforcement learning, Nat.518(7540) (2015), 529–533.
31.
Rémi Munos,
Performance bounds in lp-norm for approximate value iteration, SIAM J. Control. Optim.46(2) (2007), 541–561.
32.
John Nash,
Non-cooperative games, Annals of Mathematics54(2) (1951), 286–295.
33.
Andrew Y. Ng and Stuart J. Russell, Algorithms for inverse reinforcement learning, In Proceedings of the Seventeenth International Conference on Machine Learning (ICML)), pages 663–670. Morgan Kaufmann, 2000.
34.
Matteo Papini, Alberto Maria Metelli, Lorenzo Lupo and Marcello Restelli,
Optimistic policy optimization via multiple importance sampling, In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 4989–4999. PMLR, 2019.
35.
T. Parthasarathy,
Discounted, positive and noncooperative stochastic games, International Journal of Game Theory2(1) (1973), 25–37.
36.
Doina Precup, Richard S. Sutton and Satinder P. Singh, Eligibility traces for off-policy policy evaluation, In Pat Langley, editor, Proceedings of the Seventeenth International Conference on Machine Learning (ICML), pages 759–766. Morgan Kaufmann, 2000.
37.
Martin L Puterman, Markov decision processes: discrete stochastic dynamic programming, John Wiley & Sons, 2014.
38.
Giorgia Ramponi, Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni and Marcello Restelli,
Truly batch model-free inverse reinforcement learning about multiple intentions, In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108, pages 2359–2369. PMLR, 2020.
39.
Giorgia Ramponi, Alberto Maria Metelli, Alessandro Concetti and Marcello Restelli,
Learning in non-cooperative configurable markov decision processes, In Advances in Neural Information Processing Systems 34 (NeurIPS), 2021.
40.
Lloyd S. Shapley, Stochastic games, Proceedings of the National Academy of Sciences39(10) (1953), 1095–1100.
41.
Rui Silva, Gabriele Farina, Francisco S. Melo and Manuela Veloso,
A theoretical and algorithmic analysis of configurable mdps, In Proceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling (ICAPS)), pages 455–463. AAAI Press, 2019.
42.
Rui Silva, Francisco S. Melo and Manuela Veloso,
What if the world were different? gradient-based exploration for new optimal policies, In 4th Global Conference on Artificial Intelligence (GCAI)), volume 55 of EPiC Series in Computing, pages 229–242. EasyChair, 2018.
43.
Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, MIT Press, 2018.
44.
Csaba Szepesvári and Michael L. Littman,
Generalized markov decision processes: Dynamic-programming and reinforcement-learning algorithms, In Proceedings of International Conference of Machine Learning (ICML), volume 96, 1996.
45.
Heinrich Von Stackelberg, Marktform und gleichgewicht, J. Springer, 1934.
46.
Yevgeniy Vorobeychik and Satinder P. Singh,
Computing Stackelberg equilibria in discounted stochastic games, In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI), 2012.
47.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel and Jimmy Ba,
Benchmarking model-based reinforcement learning, CoRR, abs/1907.02057, 2019.
48.
Haoqi Zhang, Yiling Chen and David C. Parkes,
A general approach to environment design with one agent, In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI)), pages 2002–2014, 2009.
49.
Haoqi Zhang and David C. Parkes,
Value-based policy teaching with active indirect elicitation, In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (AAAI), pages 208–214. AAAI Press, 2008.
50.
Haoqi Zhang, David C. Parkes and Yiling Chen,
Policy teaching through reward function learning. In Proceedings 10th ACM Conference on Electronic Commerce (EC), pages 295–304. ACM, 2009.
51.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell and Anind K. Dey, Maximum entropy inverse reinforcement learning, In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (AAAI), pages 1433–1438. AAAI Press, 2008.
