Clutter creates challenges for robot manipulation, including a lack of non-contact trajectories and reduced visibility for line-of-sight sensors. We demonstrate that robots can use whole-arm tactile sensing to perceive clutter and maneuver within it, while keeping contact forces low. We first present our approach to manipulation, which emphasizes the benefits of making contact across the entire manipulator and assumes the manipulator has low-stiffness actuation and tactile sensing across its entire surface. We then present a novel controller that exploits these assumptions. The controller only requires haptic sensing, handles multiple contacts, and does not need an explicit model of the environment prior to contact. It uses model predictive control with a time horizon of length one and a linear quasi-static mechanical model. In our experiments, the controller enabled a real robot and a simulated robot to reach goal locations in a variety of environments, including artificial foliage, a cinder block, and randomly generated clutter, while keeping contact forces low. While reaching, the robots performed maneuvers that included bending objects, compressing objects, sliding objects, and pivoting around objects. In simulation, whole-arm tactile sensing also outperformed per-link force–torque sensing in moderate clutter, with the relative benefits increasing with the amount of clutter.
AbbeelPCoatesANgA (2010) Autonomous helicopter aerobatics through apprenticeship learning. International Journal of Robotics Research29: 1608–1639.
2.
Albu-SchafferAOttCFreseUHirzingerG (2003) Cartesian impedance control of redundant robots: Recent results with the DLR-light-weight-arms. In: IEEE international conference on robotics and automation (ICRA ‘03), 14–19 September 2003, vol. 3, pp. 3704–3709. Piscataway: IEEE Press.
3.
AlexanderR (1990) Three uses for springs in legged locomotion. International Journal of Robotics Research9: 53–61.
4.
BellinghamJRichardsAHowJ (2002) Receding horizon control of autonomous aerial vehicles. In: American control conference, vol. 5, pp. 3741–3746. Piscataway: IEEE Press.
5.
BianchiL (2007) Mechanotransduction: Touch and feel at the molecular level as modeled in caenorhabditis elegans. Molecular Neurobiology36(3): 254–271.
6.
BicchiA (1993) Force distribution in multiple whole-limb manipulation. In: IEEE international conference on robotics and automation (ICRA ‘93). 2–6 May 1993, vol. 2, pp. 196–201. Piscataway: IEEE Press.
7.
BicchiAKumarV (2000) Robotic grasping and contact: A review. In: IEEE international conference on robotics and automation (ICRA ‘00). Vol. 1, pp. 238–353. Piscataway: IEEE Press.
8.
BicchiASalisburyJBrockD (1993) Contact sensing from force measurements. International Journal of Robotics Research12: 249–262.
9.
BierbaumAAsfourTDillmannR (2009) Dynamic potential fields for dexterous tactile exploration. Human Centered Robot Systems6: 23–31.
10.
BoydSVandenbergheL (2004) Convex Optimization. Cambridge: Cambridge University Press.
11.
BuergerSHoganN (2007) Complementary stability and loop shaping for improved human–robot interaction. IEEE Transactions on Robotics23(2): 232–244.
12.
BylKTedrakeR (2008) Approximate optimal control of the compass gait on rough terrain. In: IEEE international conference on robotics and automation (ICRA‘08). Pasadena, CA, 19–23 May 2008, p. 1258–1263. Piscataway: IEEE Press.
13.
CataniaK (1999) A nose that looks like a hand and acts like an eye: the unusual mechanosensory system of the star-nosed mole. Journal of Comparative Physiology A: Neuroethology, Sensory, Neural, and Behavioral Physiology185: 367–372.
14.
ChittaSSturmJPiccoliMBurgardW (2011) Tactile sensing for mobile manipulation. IEEE Transactions on Robotics27(99): 1–11.
15.
CottleRDantzigG (1968) Complementary pivot theory of mathematical programming. Linear Algebra and its Applications1: 103–125.
16.
CottleRPangJStoneR (2009) The Linear Complementarity Problem. Philadelphia, PA: Society for Industrial & Applied Mathematics.
17.
CutkoskyMRUlmenJV (2012) Dynamic tactile sensing. In: SantosVBalasubramanianR (eds) The Human Hand: A Source of Inspiration for Robotic Hands. Berlin: Springer, ch. 24.
18.
De LucaAAlbu-SchafferAHaddadinSHirzingerG (2006) Collision detection and safe reaction with the DLR-III lightweight manipulator arm. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘06), Beijing, China, October 2006, pp. 1623–1630. Piscataway: IEEE Press.
19.
De LucaAMattoneR (2004) An adapt-and-detect actuator FDI scheme for robot manipulators. In: IEEE international conference on robotics and automation (ICRA ‘04), 26 April 1 May, vol. 5, p. 4975–4980. Piscataway: IEEE Press.
20.
De SchutterJBruyninckxHDutréSDe GeeterJKatupitiyaJDemeyS. (1999) Estimating first-order geometric parameters and monitoring contact transitions during force-controlled compliant motion. International Journal of Robotics Research18: 1161–1184.
21.
DiankovRKuffnerJ (2008) Openrave: A planning architecture for autonomous robotics. Report, Robotics Institute, USA.
22.
DogarMHemrajaniVLeedsDKaneBSrinivasaS (2010) Proprioceptive localization for mobile manipulators. Report, Carnegie Mellon University, USA.
23.
DogarMSrinivasaS (2011) A framework for push-grasping in clutter. Robotics: science and systems. Cambridge, MA: The MIT Press.
24.
DominyN (2004) Fruits, fingers, and fermentation: the sensory cues available to foraging primates. Integrative and Comparative Biology44: 295–303.
DupontPYamajakoS (1994) Jamming and wedging in constrained rigid-body dynamics. In: IEEE international conference on robotics and automation (ICRA ‘94), San Diego, CA, USA, 8–13 May, 1994, vol. 3, pp. 2349–2354. Piscataway: IEEE Press.
27.
EbermanB (1989) Whole-arm manipulation: kinematics and control. MSc Thesis, Massachusetts Institute of Technology, USA.
28.
EbermanBSalisburyJ (1990) Determination of manipulator contact information from joint torque measurements. In: 1st international symposium on experimental robotics, vol. 139, pp. 463–473. New York: ACM Press.
29.
EdsingerAKempCC (2007 a) Human–robot interaction for cooperative manipulation: Handing objects to one another. In: 16th IEEE international symposium on robot and human interactive communication (RO-MAN ‘07), Jeju, Korea, 26–29 August, pp. 1167–1172. Piscataway: IEEE Press.
30.
EdsingerAKempCC (2007 b) Two arms are better than one: A behavior-based control system for assistive bimanual manipulation. In: 13th international conference on advanced robotics (ICAR ‘07), vol. 370, pp. 345–355.
31.
ErezTTassaYTodorovE (2011) Infinite-horizon model predictive control for periodic tasks with contacts. In: Robotics: science and systems. Cambridge, MA: The MIT Press, pp. 73–80.
32.
EscandeAKheddarA (2009) Contact planning for acyclic motion with tasks constraints. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘09), St Louis, MO, USA, 10-15 October 2009, pp. 435–440. Piscataway: IEEE Press.
33.
FeatherstoneROrinDE (2008) Dynamics. In: SicilianoBKhatibO (eds) Handbook of Robotics. New York: Springer, ch. 2.
34.
FrankBStachnissCAbdoNBurgardW (2011) Using Gaussian process regression for efficient motion planning in environments with deformable objects. Workshops at: 25th AAAI conference on artificial intelligence, San Francisco, CA, USA, August 7–11, 2011.
35.
FromPGravdahlJLillehagenTAbbeelP (2011) Motion planning and control of robotic manipulators on seaborne platforms. Control Engineering Practice19: 809–819.
36.
GaneshGJarrasseNHaddadinSAlbu-SchaefferABurdetE (2012) A versatile biomimetic controller for contact tooling and haptic exploration. In: IEEE international conference on robotics and automation (ICRA ‘12), Saint Paul, MN, USA, 14–18 May 2012, pp. 3329–3334. Piscataway: IEEE Press.
37.
GarciaCPrettDMorariM (1989) Model predictive control: Theory and practice–a survey. Automatica25: 335–348.
38.
GarciaMChatterjeeARuinaAColemanM (1998) The simplest walking model: Stability, complexity, and scaling. Journal of Biomechanical Engineering120: 281–288.
39.
GrossoMQuachRBadlerN (1989) Anthropometry for computer animated human figures. State-of-the Art in Computer Animation. pp. 83. Berlin: Springer.
40.
GuXBallardD (2006) An equilibrium point based model unifying movement control in humanoids. In: Robotics: science and systems.
41.
HaddadinSAlbu-SchafferADe LucaAHirzingerG (2008) Collision detection and reaction: A contribution to safe physical human–robot interaction. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘08), Nice, Italy, 22–26 September 2008, pp. 3356–3363. Piscataway: IEEE Press.
42.
HaddadinSBelderRAlbu-SchäfferA (2011) Dynamic motion planning for robots in partially unknown environments. In: IFAC world congress, Milan, Italy, 28August-2September, vol. 18, pp. 6842–6850.
43.
HauserKBretlTLatombeJ (2005) Non-gaited humanoid locomotion planning. In: IEEE conference on humanoid robots, Tsukuba, Japan, December 2005, pp. 7–12. Piscataway: IEEE Press.
44.
HerschMBillardA (2006) A biologically-inspired controller for reaching movements. In: IEEE/RAS-EMBS international conference on biomedical robotics and biomechatronics (BIOROB ‘06), Pisa, Italy, 20-22 February 2006, pp. 1067–1072. Piscataway: IEEE Press.
45.
HoganN (1984) Adaptive control of mechanical impedance by coactivation of antagonist muscles. IEEE Transactions on Automatic Control29: 681–690.
46.
HoganN (1988) On the stability of manipulators performing contact tasks. IEEE Journal of Robotics and Automation4(6): 677–686.
47.
HoganNBuergerS (2005). Impedance and interaction control. In: SicilianoBKhatibO (eds) Handbook of Robotics. New York: Springer, ch. 19.
48.
HsiaoKChittaSCiocarlieMJonesE (2010 a) Contact-reactive grasping of objects with partial shape information. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘10)Taipei, Taiwan, 18–22 October 2010, vol.10, pp. 1228–1235. Piscataway: IEEE Press.
49.
HsiaoKKaelblingLLozano-PérezT (2010 b) Task-driven tactile exploration. Robotics: science and systems.
50.
HsiaoKLozano-PerezT (2006) Imitation learning of whole-body grasps. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘06), Beijing, China, 9–15 October 2006, pp. 5657–5662. Piscataway: IEEE Press.
51.
IvaldiSFumagalliMNoriFBagliettoMMettaGSandiniG (2010) Approximate optimal control for reaching and trajectory planning in a humanoid robot. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘10)Taipei, Taiwan, 18–22 October, pp. 1290–1296. Piscataway: IEEE Press.
52.
IwaniukAWhishawI (1999) How skilled are the skilled limb movements of the raccoon (procyon lotor)?Behavioural Brain Research99(1): 35–44.
53.
PrattJPDChewMPrattG (2001) Virtual model control: An intuitive approach for bipedal locomotion. International Journal of Robotics Research20(2): 129–143.
54.
JainAKempCC (2009 a) Behavior-based door opening with equilibrium point control. RSS Workshop Mobile Manipulation in Human Environments at: Robotics: science and systems.
55.
JainAKempCC (2009 b) Pulling open novel doors and drawers with equilibrium point control. In: IEEE conference on humanoid robots, Paris, France, 7–10 December, pp. 498–505. Piscataway: IEEE Press.
56.
JainAKempCC (2010 a) EL-E: An assistive mobile manipulator that autonomously fetches objects from flat surfaces. Autonomous Robots28: 45–64.
57.
JainAKempCC (2010 b) Pulling open doors and drawers: coordinating an omni-directional base and a compliant arm with equilibrium point control. In: IEEE international conference on robotics and automation (ICRA ‘10), Anchorage, AK, USA, 3–7 May 2010, pp. 1807–1814. Piscataway: IEEE Press.
58.
JohnsonKL (1985) Contact Mechanics. Cambridge: Cambridge University Press, ch. 4
59.
KanekoMTanieK (1994) Contact point detection for grasping an unknown object using self-posture changeability. IEEE Transactions on Robotics and Automation3: 355–367.
60.
KaoILynchKBurdickJW (2008). Contact modeling and manipulation. In: SicilianoBKhatibO (eds) Handbook of Robotics. New York: Springer, ch. 27.
61.
KavrakiLELaValleSM (2008) Motion planning. In: SicilianoBKhatibO (eds) Handbook of Robotics. New York: Springer, ch. 5.
62.
KhatibO (1987) A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal of Robotics and Automation3: 43–53.
63.
KillpackMDeyleTAndersonCKempC (2010) Visual odometry and control for an omnidirectional mobile robot with a downward-facing camera. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘10), Taipei, Taiwan, 18–22 October 2010, pp. 139–146. Piscataway: IEEE Press.
64.
KraftDBierbaumAKjaergaardMRatkeviciusJKjaer-NielsenARybergC. (2009) Tactile object exploration using cursor navigation sensors. In: EuroHaptics symposium on haptic interfaces for virtual environment and teleoperator systems, Salt Lake City, UT, USA, 18–20 March 2009, pp. 296–301. Piscataway: IEEE Press.
65.
KulchenkoPTodorovE (2011) First-exit model predictive control of fast discontinuous dynamics: Application to ball bouncing. In: IEEE international conference on robotics and automation (ICRA ‘11)Shanghai, China, 9–13 May 2011, pp. 2144–2151. Piscataway: IEEE Press.
66.
LaValleSKuffnerJ (2001) Randomized kinodynamic planning. International Journal of Robotics Research20(5): 378–400.
67.
LedermanSJKlatzkyRL (2009) Haptic perception: A tutorial. Attention, Perception and Psychophysics (71): 1439–1459.
68.
LeeperAHsiaoKCiocarlieMTakayamaLGossowD (2012) Strategies for human-in-the-loop robotic grasping. In: ACM/IEEE international conference on human robot interaction, New York, NY, USA, pp. 1–8. New York: ACM Press.
69.
LegagneSKheddarAYoshidaE (2011) Generation of optimal dynamic multi-contact motions: Application to humanoid robots. IEEE Transactions on Robotics, submitted.
70.
Lozano-PerezT (1987) A simple motion-planning algorithm for general robot manipulators. IEEE Transactions on Robotics and Automation3: 224–238.
71.
LumpkinEAMarshallKLNelsonAM (2010) The cell biology of touch. Journal of Cell Biology191(2): 237–248.
72.
MaladenRDingYUmbanhowarPKamorAGoldmanD (2010) Biophysically inspired development of a sand-swimming robot. Robotics: science and systems.
73.
ManchesterIMettinUIidaFTedrakeR (2011) Stable dynamic walking over uneven terrain. International Journal of Robotics Research3: 265–279.
74.
MasonM (2001) Mechanics of Robotic Manipulation. Cambridge: MIT Press.
75.
MasonMRodriguezASrinivasaSVazquezA (2011) Autonomous manipulation with a general-purpose simple hand. International Journal of Robotics Research31: 688–703.
76.
MayneDRawlingsJRaoCScokaertP (2000) Constrained model predictive control: Stability and optimality. Automatica36(6): 789–814.
77.
McClaveJBensonPSincichT (2008) Statistics for Business and Economics. New Jersey: Pearson Education.
78.
McKennaJAnhaltDBronsonFBrownHSchwerinMShammasE. (2008) Toroidal skin drive for snake robot locomotion. In: IEEE international conference on robotics and automation (ICRA ‘08), Pasadena, CA, USA, 19–23 May 2008, pp. 1150–1155. Piscataway: IEEE Press.
79.
MettaGNataleLNoriFSandiniG (2011) Force control and reaching movements on the ICUB humanoid robot. International symposium on robotics research.
80.
MiglioreS (2009) The role of passive joint stiffness and active knee control in robotic leg swinging: Applications to dynamic walking. PhD Thesis, Georgia Institute of Technology, USA.
81.
MiglioreSBrownEDeWeerthS (2005) Biologically inspired joint stiffness control. In: IEEE international conference on robotics and automation (ICRA ‘05), 18–22 April 2005, pp. 4508–4513. Piscataway: IEEE Press.
82.
MorariMLeeJH (1999) Model predictive control: past, present and future. Computers & Chemical Engineering23(4–5): 667–682.
83.
MurrayRLiZSastryS (1994) A Mathematical Introduction to Robotic Manipulation. New York: CRC Press.
84.
NataleLTorres-JaraE (2006) A sensitive approach to grasping. In: International workshop on epigenetic robotics, Paris, France, 20–22 September 2006, pp. 87–94.
85.
OkamuraACurkoskyM (2001) Feature-guided exploration with a robotic finger. In: IEEE international conference on robotics and automation (ICRA ‘01), vol. 1, pp. 589–596. Piscataway: IEEE Press.
86.
OkamuraACutkoskyM (1999) Haptic exploration of fine surface features. In: IEEE international conference on robotics and automation (ICRA ‘99), Detroit, MI, USA, 10–15 May, vol. 4, pp. 2930–2936. Piscataway: IEEE Press.
PastorPRighettiLKalakrishnanMSchaalS (2011) Online movement adaptation based on previous sensor experiences. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘11)San Francisco, CA, 25–30 September 2011, pp. 365–371. Piscataway: IEEE Press.
89.
PatilSvan den BergJAlterovitzR (2011) Motion planning under uncertainty in highly deformable environments. In: Robotics: science and systems: 242–248. Cambridge, MA: The MIT Press.
90.
PetrovskayaAKhatibO (2011) Global localization of objects via touch. IEEE Transactions on Robotics27(3): 569–585.
91.
PetrovskayaAParkJKhatibO (2007) Probabilistic estimation of whole body contacts for multi-contact robot control. IEEE international conference on robotics and automation (ICRA ‘07), Rome, Italy, 10–14 April 2007, pp. 568–573. Piscataway: IEEE Press.
92.
PezzementiZPlakuEReydaCHagerG (2011) Tactile-object recognition from appearance information. IEEE Transactions on Robotics27(3): 473–487.
93.
PlattRJrFaggAGrupenR (2003) Extending fingertip grasping to whole body grasping. In: IEEE international conference on robotics and automation (ICRA ‘03), 14–16 September 2003, vol. 2, pp. 2677–2682. Piscataway: IEEE Press.
94.
PlattRJrPermenterFPfeifferJ (2011) Using Bayesian filtering to interpret tactile data during flexible materials manipulation. IEEE Transactions on Robotics27: 586–598.
PrattGWilliamsonM (1995) Series elastic actuators. In: IEEE/RSJ international conference on intelligent robots and systems (IROS ‘95), Pittsburgh, PA, USA, 5–6 August 1995, vol. 1, pp. 399–406. Piscataway: IEEE Press.
97.
PrescottTPearsonMFoxCEvansMMitchinsonBAndersonS. (2010) Towards biomimetic vibrissal tactile sensing for robot exploration, navigation, and object recognition in hazardous environments. In: 4th international workshop on robotics for risky interventions and environmental surveillance-maintenance (RISE ‘10).
98.
QuigleyMGerkeyBConleyKFaustJFooteTLeibsJ. (2009) ROS: An open-source robot operating system. ICRA Open-Source Software workshop at: IEEE international conference on robotics and automation (ICRA ‘09). Piscataway: IEEE Press.
99.
RaibertMBlankespoorKNelsonGPlayterR. (2008) Bigdog, the rough–terrain quadruped robot. In: 17th world congress. The International Federation of Automatic Control, Seoul, South Korea, July 6–11, pp. 10822–10825.
100.
RaibertMCraigJ (1981) Hybrid position/force control of manipulators. Journal of Dynamic Systems, Measurement, and Control102(127): 126–133.
101.
RodriguezSLienJAmatoN (2006) Planning motion in completely deformable environments. In: IEEE international conference on robotics and automation (ICRA ‘06), Orlando, FL, USA, 15–19 May 2006, pp. 1067–1079. Piscataway: IEEE Press.
102.
RomanoJHsiaoKNiemeyerGChittaSKuchenbeckerK (2011) Human-inspired robotic grasp control with tactile sensing. IEEE Transactions on Robotics27: 1067–1079.
103.
SalisburyJJ (1984) Interpretation of contact geometries from force measurements. In: IEEE international conference on robotics and automation (ICRA ‘84), March 1984, vol. 1, pp. 240–247. Piscataway: IEEE Press.
104.
SaranliUBuehlerMKoditschekD (2001) Rhex: A simple and highly mobile hexapod robot. International Journal of Robotics Research20: 616–631.
105.
SaxenaADriemeyerJNgA (2008) Robotic Grasping of Novel Objects using Vision. International Journal of Robotics Research27(2): 157–173.
106.
SentisLKhatibO (2005) Synthesis of whole-body behaviors through hierarchical control of behavioral primitives. International Journal of Humanoid Robotics2: 505–518.
107.
SentisLParkJKhatibO (2010) Compliant control of multicontact and center-of-mass behaviors in humanoid robots. IEEE Transactions on Robotics26(3): 483–501.
108.
ShadmehrR (1993) Control of equilibrium position and stiffness through postural modules. Journal of Motor Behavior25: 228–241.
109.
SrinivasaSFergusonCHelfrichDBerensonDColletADiankovR. (2009) Herb: A home exploring robotic butler. Autonomous Robots28: 5–20.
110.
StilmanMNishiwakiKKagamiSKuffnerJ (2007 a) Planning and executing navigation among movable obstacles. Advanced Robotics21(14): 1617–1634.
111.
StilmanMSchamburekJKuffnerJAsfourT (2007 b) Manipulation planning among movable obstacles. In: IEEE international conference on robotics and automation (ICRA ‘07), Rome, Italy, 10–14 April 2007, pp. 3327–3332. Piscataway: IEEE Press.
112.
StulpFKresseIMaldonadoARuizFFedrizziABeetzM (2009) Compact models of human reaching motions for robotic control in everyday manipulation tasks. In: IEEE international conference on development and learning, Shanghai, China, 5–7 June, pp. 1–7. Piscataway: IEEE Press.
113.
TanakaHKushihamaKUedaNHiraiS (2003) A vision-based haptic exploration. In: IEEE international conference on robotics and automation (ICRA ‘03), 14–16 September, vol. 3, pp. 3441–3448. Piscataway: IEEE Press.
114.
TilleyADreyfussH (2001) The Measure of Man and Woman: Human Factors in Design. New York: Wiley.
115.
WhitneyD (1982) Quasi-static assembly of compliantly supported rigid parts. Journal of Dynamic Systems, Measurement, and Control104: 65–77.
116.
WhitneyDNevinsJ (1979) What is the remote center compliance (RCC) and what can it do. In: 9th international symposium on industrial robots, pp. 135–152.
117.
WieberP (2006) Trajectory free linear model predictive control for stable walking in the presence of strong perturbations. In: IEEE-RAS international conference on humanoid robots, Genoa, Italy, 4–6 December, pp. 137–142. Piscataway: IEEE Press.
118.
WilliamsonM (1996) Postural primitives: Interactive behavior for a humanoid robot arm. In: 4th international conference on simulation of adaptive behavior, pp. 124–131. Cambridge, MA: The MIT Press.
119.
WilliamsonM (1999) Robot arm control exploiting natural dynamics. PhD Thesis, Massachusetts Institute of Technology, USA.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
