GelSight family of vision-based tactile sensors has proven to be effective for multiple robot perception and manipulation tasks. These sensors are based on an internal optical system and an embedded camera to capture the deformation of the soft sensor surface, inferring the high-resolution geometry of the objects in contact. However, customizing the sensors for different robot hands requires a tedious trial-and-error process to redesign the optical system. In this paper, we formulate the GelSight sensor design process as a systematic and objective-driven design problem and perform the design optimization with a physically accurate optical simulation. The method is based on modularizing and parameterizing the sensor’s optical components and designing four generalizable objective functions to evaluate the sensor. We implement the method with an interactive and easy-to-use toolbox called OptiSense Studio. With the toolbox, non-sensor experts can quickly optimize their sensor design in both forward and inverse ways following our predefined modules and steps. We demonstrate our system with four different GelSight sensors by quickly optimizing their initial design in simulation and transferring it to the real sensors.
AbdiHWilliamsLJ (2010) Principal component analysis. WIREs Computational Statistics2(4): 433–459.
2.
AgarwalAManTYuanW (2021) Simulation of vision-based tactile sensors using physics based rendering. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). Piscataway: IEEE, 1–7.
3.
AndrussowISunHKuchenbeckerKJ, et al. (2023) Minsight: a fingertip-sized vision-based tactile sensor for robotic manipulation. Advanced Intelligent Systems5(8): 2300042.
4.
CalandraROwensAJayaramanD, et al. (2018) More than a feeling: learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters3(4): 3300–3307.
5.
CaoGJiangJLuC, et al. (2023) Touchroller: a rolling optical tactile sensor for rapid assessment of textures for large surface areas. Sensors23(5): 2661.
6.
CommunityBO (2018) Blender - a 3D modelling and rendering package. In: Blender Foundation. Amsterdam: Stichting Blender Foundation. https://www.blender.org/
7.
DoWKKennedyM (2022) Densetact: optical tactile sensor for dense shape reconstruction. In: 2022 International Conference on Robotics and Automation (ICRA). Piscataway: IEEE, 6188–6194.
8.
DoWKDhawanAKKitzmannM, et al. (2023) Densetact-mini: an optical tactile sensor for grasping multi-scale objects from flat surfaces. arXiv preprint arXiv:2309.08860.
9.
DongSYuanWAdelsonEH (2017) Improved gelsight tactile sensor for measuring geometry and slip. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE, 137–144.
10.
DonlonEDongSLiuM, et al. (2018) Gelslim: A High-Resolution, Compact, Robust, and Calibrated Tactile-Sensing Finger. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE, pp. 1927–1934.
11.
DupuyJJakobW (2018) An adaptive parameterization for efficient material acquisition and rendering. ACM Transactions on Graphics37(6): 1–14.
JakobWSpeiererSRousselN, et al. (2022) Dr. jit: a just-in-time compiler for differentiable rendering. ACM Transactions on Graphics41(4): 1–19.
17.
JohnsonMKAdelsonEH (2009) Retrographic sensing for the measurement of surface texture and shape. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 1070–1077.
18.
LambetaMChouPWTianS, et al. (2020) Digit: a novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters5(3): 3838–3845.
19.
LiRPlattRYuanW, et al. (2014) Localization and manipulation of small parts using gelsight tactile sensing. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 3988–3993.
20.
LinCZhangHXuJ, et al. (2023a) 9dtact: a compact vision-based tactile sensor for accurate 3d shape reconstruction and generalizable 6d force estimation. Piscataway: IEEE Robotics and Automation Letters.
21.
LinZZhuangJLiY, et al. (2023b) Gelfinger: a novel visual-tactile sensor with multi-angle tactile image stitching. IEEE Robotics and Automation Letters8(9): 5982–5989.
22.
LiuSQAdelsonEH (2022) Gelsight fin ray: incorporating tactile sensing into a soft compliant robotic gripper. In: 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft). Edinburgh, United Kingdom: IEEE, 925–931.
23.
LiuSQMaYAdelsonEH (2023a) Gelsight baby fin ray: a compact, compliant, flexible finger with high-resolution tactile sensing. In: 2023 IEEE International Conference on Soft Robotics (RoboSoft). Piscataway: IEEE, 1–8.
24.
LiuSQYañezLZAdelsonEH (2023b) Gelsight endoflex: a soft endoskeleton hand with continuous high-resolution tactile sensing. In: 2023 IEEE International Conference on Soft Robotics (RoboSoft). Piscataway: IEEE, 1–6.
25.
MirzaeeMAHuangHJYuanW (2025) Gelbelt: a vision-based tactile sensor for continuous sensing of large surfaces. IEEE Robotics and Automation Letters10(2): 2016–2023.
26.
PagnuttiMRyanRECazenavetteG, et al. (2017) Laying the foundation to use raspberry pi 3 v2 camera module imagery for scientific and engineering purposes. Journal of Electronic Imaging26(1): 013014.
27.
PatelROuyangRRomeroB, et al. (2021) Digger finger: gelsight tactile sensor for object identification inside granular media. In: SicilianoBLaschiCKhatibO (eds) Experimental Robotics. Cham: Springer International Publishing, 105–115.
28.
PharrMJakobWHumphreysG (2023) Physically Based Rendering: From Theory to Implementation. Cambridge: MIT Press.
29.
QiHYiBSureshS, et al. (2023) General in-hand object rotation with vision and touch. In: Conference on Robot Learning. Westminster: PMLR, 2549–2564.
30.
RomeroBVeigaFAdelsonE (2020) Soft, round, high resolution tactile fingertip sensors for dexterous robotic manipulation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). Paris, France: IEEE, 4796–4802.
31.
SheYLiuSQYuP, et al. (2020) Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). Paris, France: IEEE, 10075–10081.
32.
SiZYuanW (2022) Taxim: an example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters7(2): 2361–2368.
33.
SiZZhuZAgarwalA, et al. (2022) Grasp stability prediction with sim-to-real transfer from tactile sensing. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Paris, France: IEEE, 7809–7816.
SureshSSiZMangelsonJG, et al. (2022) Shapemap 3-d: efficient shape mapping through dense touch and vision. In: 2022 International Conference on Robotics and Automation (ICRA). Piscataway: IEEE, 7073–7080.
36.
TaylorIHDongSRodriguezA (2022) Gelslim 3.0: high-resolution measurement of shape, force and slip in a compact tactile-sensing finger. In: 2022 International Conference on Robotics and Automation (ICRA). Philadelphia, PA, USA: IEEE, 10781–10787.
37.
TippurMHAdelsonEH (2023) Gelsight360: an omnidirectional camera-based tactile sensor for dexterous robotic manipulation. arXiv preprint arXiv:2304.04268.
38.
VirtanenPGommersROliphantTE, et al. (2020) SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods17: 261–272.
39.
WangSSheYRomeroB, et al. (2021) Gelsight wedge: measuring high-resolution 3d contact geometry with a compact robot finger. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). Xi’an, China: IEEE, pp. 6468–6475.
40.
Ward-CherrierBPestellNCramphornL, et al. (2018) The tactip family: soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft Robotics5(2): 216–227.
41.
XuJChenTZlokapaL, et al. (2021a) An end-to-end differentiable framework for contact-aware robot design. arXiv preprint arXiv:2107.07501.
42.
XuJChenTZlokapaL, et al. (2021b) An end-to-end differentiable framework for contact-aware robot design. arXiv preprint arXiv:2107.07501.
43.
YuanWDongSAdelsonEH (2017) Gelsight: high-resolution robot tactile sensors for estimating geometry and force. Sensors17(12): 2762.
44.
ZhangSSunYSunF, et al. (2024) Pfs 1.0: a development tool applied to vision-based tactile sensor process formulation and fabrication. Sensors and Actuators A: Physical367: 115090.
45.
ZhaoJAdelsonEH (2023) Gelsight svelte: a human finger-shaped single-camera tactile robot finger with large sensing coverage and proprioceptive sensing. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE, 8979–8984.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
