In this article we investigate the role of abstraction principles for knowledge transfer in agent control learning tasks. We analyze abstraction from a formal point of view and characterize three distinct facets: aspectualization, coarsening, and conceptual classification. The taxonomy we develop allows us to interrelate existing approaches to abstraction, leading to a code of practice for designing knowledge representations that support knowledge transfer. We detail how aspectualization can be utilized to achieve knowledge transfer in reinforcement learning. We propose the use of so-called structure space aspectualizable knowledge representations that explicate structural properties of the state space and present a posteriori structure space aspectualization (APSST) as a method to extract generally sensible behavior from a learned policy. This new policy can be used for knowledge transfer to support learning new tasks in different environments. Finally, we present a case study that demonstrates transfer of generally sensible navigation skills from simple simulation to a real-world robotic platform.
Bertel, S., Freksa, C., & Vrachliotis, G. (2004). Aspectualize and conquer in architectural design. In J. S. Gero , B. Tversky & T. Knight (Eds.), Visual and spatial reasoning in design III (pp. 255-279). Key Centre of Design, Computing and Cognition , University of Sydney.
2.
Bittner, T., & Smith, B. ( 2001). A taxonomy of granular partitions. In D. Montello (Ed.), Spatial Information Theory: Cognitive and Computational Foundations of Geographic Information Science (COSIT), volume 2205 of Lecture Notes in Computer Science (pp. 28-43). Berlin : Springer.
3.
Dean, T., & Givan, R. ( 1997). Model minimization in Markov decision processes. In Proceedings of the 14th National Conference on Artificial Intelligence (AAAI) (pp. 106-111).
4.
Dietterich, T.G. ( 1998). The MAXQ method for hierarchical reinforcement learning . In Proceedings of the Fifteenth International Conference on Machine Learning (ICML) (pp. 118-126).
5.
Dietterich, T.G. ( 2000). State abstraction in MAXQ hierarchical reinforcement learning. In S. A. Solla, T. K. Leen & K.-R. Muller (Eds.), Advances in Neural Information Processing Systems 12: Proceedings of the 1999 Conference (pp. 994-1000). Cambridge, MA: MIT Press.
6.
Dorigo, M., & Colombetti, M. (1994). Robot shaping: Developing autonomous agents through learning. Artificial Intelligence , 71(2), 321-370.
7.
Fernández, F., & Veloso, M. ( 2006). Probabilistic policy reuse in a reinforcement learning agent. In Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS) (pp. 720-727).
8.
Frommberger, L. ( 2006). A qualitative representation of structural spatial knowledge for robot navigation with reinforcement learning. In Proceedings of the ICML-06 Workshop on Structural Knowledge Transfer for Machine Learning.
9.
Frommberger, L. ( 2008). Learning to behave in space: A qualitative spatial representation for robot navigation with reinforcement learning. International Journal on Artificial Intelligence Tools, 17(3), 465-482.
10.
Grisetti, G., Stachniss, C., & Burgard, W. ( 2007). Improved techniques for grid mapping with Rao-Blackwellized particle filters. IEEE Transactions on Robotics, 23, 34-46.
11.
Gutmann, J.-S., Weigel, T., & Nebel, B. ( 2001). A fast, accurate and robust method for self-localization in polygonal environments using laser range finders. Advanced Robotics, 14(8), 651-667.
12.
Herskovits, A. ( 1998). Schematization. In P. Olivier & K. P. Gapp (Eds.), Representation and processing of spatial expressions (pp. 149-162). Mahwah, NJ: Lawrence Erlbaum Associates.
13.
Hobbs, J.R. ( 1985). Granularity. In Proceedings of the Ninth International Joint Conference on Artificial Intelligence (IJCAI) (pp. 432-435).
14.
Jong, N.K., & Stone, P. ( 2005). State abstraction discovery from irrelevant state variables . In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJCAI) (pp. 752-757).
15.
Kaelbling, L.P., Littman, M.L., & Moore, A.W. ( 1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237-285.
16.
Klippel, A., Richter, K.-F., Barkowsky, T., & Freksa, C. ( 2005). The cognitive reality of schematic maps. In L. Meng, A. Zipf & T. Reichenbacher (Eds.), Map-based mobile services - theories, methods and implementations (pp. 57-74). Berlin : Springer.
17.
Konidaris, G.D. ( 2006). A framework for transfer in reinforcement learning. In Proceedings of the ICML-06 Workshop on Structural Knowledge Transfer for Machine Learning.
18.
Konidaris, G.D., & Barto, A.G. ( 2006). Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings of the Twenty Third International Conference on Machine Learning (ICML) (pp. 489-496).
19.
Konidaris, G.D., & Barto, A.G. ( 2007). Building portable options: Skill transfer in reinforcement learning. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (IJCAI).
20.
Li, L., Walsh, T.J., & Littman, M.L. ( 2006). Towards a unified theory of state abstraction for MDPs . In Proceedings of the 9th International Symposium on Artificial Intelligence and Mathematics (pp. 531-539).
21.
Lovejoy, W.S. ( 1991). A survey of algorithmic methods for partially observable Markov decision processes. Annals of Operations Research, 1, 47-66.
22.
Mackaness, W.A., & Chaudhry, O. ( 2008). Generalization and symbolization. In S. Shekhar & H. Xiong (Eds.), Encyclopedia of GIS (pp. 330-339). Berlin: Springer.
23.
McCallum, A.K. ( 1995). Reinforcement learning with selective perception and hidden state. Unpublished doctoral dissertation, University of Rochester.
24.
Moravec, H.P., & Elfes, A.E. ( 1985). High resolution maps from wide angle sonar. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
25.
Munos, R., & Moore, A. ( 1999). Variable resolution discretizations for high-accuracy solutions of optimal control problems. In Proceedings of the Sixteenth International Conference on Artificial Intelligence (IJCAI) (pp. 1348- 1355).
26.
Parr, R., & Russell, S.J. ( 1998). Reinforcement learning with hierarchies of machines. In M. I. Jordan, M. J. Kearns & S. A. Solla (Eds.), Advances in Neural Information Processing Systems 10: Procedings of the 1997 conference (pp. 1043-1049). Cambridge, MA: MIT Press.
27.
Reynolds, S.I. ( 2000). Adaptive resolution model-free reinforcement learning: Decision boundary partitioning. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML).
28.
Schlieder, C. ( 1993). Representing visible locations for qualitative navigation . In N. P. Carrete & M. G. Singh (Eds.), Qualitative reasoning and decision technologies (pp. 523-532).
29.
Stell, J.G., & Worboys, M.F. ( 1999). Generalizing graphs using amalgamation and selection . In R. H. Güting, D. Papadias & F. Lochovsky (Eds.), Advances in Spatial Databases: Proceedings of the 6th International Symposium on Spatial Databases (SSD’99), volume 1651 of Lecture Notes in Computer Science (pp. 19-32). Springer: Berlin, Heidelberg.
30.
Sutton, R.S., & Barto, A.G. ( 1998). Reinforcement learning: an introduction. Cambridge, MA: MIT Press.
31.
Sutton, R.S., Precup, D., & Singh, S. ( 1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 112(1-2), 181-211.
32.
Talmy, L. ( 1983). How language structures space. In H. L. Jr Pick & L. P. Acredolo (Eds.), Spatial orientation: Theory, research, and application (pp. 225-282). NewYork, NJ: Plenum.
33.
Taylor, M., & Stone, P. ( 2007). Cross-domain transfer for reinforcement learning. In Proceedings of the Twenty Fourth International Conference on Machine Learning (ICML).
34.
Taylor, M., Stone, P., & Liu, Y. ( 2005). Value functions for rlbased behavior transfer: A comparative study. In Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI) (pp. 880-885).
35.
Thrun, S., & Schwartz, A. ( 1995). Finding structure in reinforcement learning. In G. Tesauro, D. Touretzky & T. Leen (Eds.), Advances in Neural Information Processing Systems: Proceedings of the 1994 conference. Cambridge, MA: MIT Press.
36.
Torrey, L., Shavlik, J., Walker, T., & Maclin, R. ( 2006). Skill acquisition via transfer learning and advice taking . In Proceedings of the Seventeenth European Conference on Machine Learning (ECML) (pp. 425-436).
37.
Watkins, C. ( 1989). Learning from delayed rewards. Unpublished doctoral dissertation , Cambridge University.
38.
Zadeh, L.A. ( 1979). Fuzzy sets and information granularity. In M. M. Gupta, R. K. Ragade & R. R. Yager (Eds.), Advances in fuzzy set theory and applications (pp. 3-18). Amsterdam : North Holland Publishing Company.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
