Pose-and-shear-based tactile servoing

Abstract

Tactile servoing is an important technique because it enables robots to manipulate objects with precision and accuracy while adapting to changes in their environments in real-time. One approach for tactile servo control with high-resolution soft tactile sensors is to estimate the contact pose relative to an object surface using a convolutional neural network (CNN) for use as a feedback signal. In this paper, we investigate how the surface pose estimation model can be extended to include shear, and utilise these combined pose-and-shear models to develop a tactile robotic system that can be programmed for diverse non-prehensile manipulation tasks, such as object tracking, surface-following, single-arm object pushing and dual-arm object pushing. In doing this, two technical challenges had to be overcome. Firstly, the use of tactile data that includes shear-induced slippage can lead to error-prone estimates unsuitable for accurate control, and so we modified the CNN into a Gaussian-density neural network and used a discriminative Bayesian filter to improve the predictions with a state dynamics model that utilises the robot kinematics. Secondly, to achieve smooth robot motion in 3D space while interacting with objects, we used SE(3) velocity-based servo control, which required re-deriving the Bayesian filter update equations using Lie group theory, as many standard assumptions do not hold for state variables defined on non-Euclidean manifolds. In future, we believe that pose-and-shear-based tactile servoing will enable many object manipulation tasks and the fully-dexterous utilisation of multi-fingered tactile robot hands.

1. Introduction

Tactile sensing is an essential component for enabling controlled physical interaction between a robot and its surroundings. A technique known as tactile servoing (Li et al. (2013); Lepora and Lloyd (2021)) uses tactile feedback to adjust the position or velocity of a robot’s end-effector, such as a gripper, hand or tool. Tactile servoing is an important technique in robotics because it enables robots to manipulate objects with precision and accuracy while adapting to changes in their environments in real-time. Tactile servo control is based on its more established counterpart: visual servo control (e.g. Espiau et al. (1992); Hutchinson et al. (1996); Chaumette and Hutchinson (2006)), which since the 1990 s has led to many applications in vision-based robotics. However, tactile servoing cannot yet be widely deployed because it is far more difficult to perceive relevant state information from a soft tactile sensor interacting physically with its environment than from a non-contact sense such as vision. For example, the recent time history of the combined normal and shear motion will affect the deformation of a soft tactile sensor. This paper aims to close that gap by showing how tactile pose-and-shear state information can enable many robot manipulation tasks.

One approach for tactile servo control with high-resolution soft tactile sensors is to estimate the contact pose relative to an object surface using a convolutional neural network (CNN) and use this as a feedback signal: the tactile sensor can then slide over unknown curved surfaces and push unknown objects to goal locations (Lepora and Lloyd (2020, 2021); Lloyd and Lepora (2021)). However, soft tactile sensors shear after contact, which our previous work treated as an undesirable ‘nuisance variable’ that interferes with the primary goal of contact pose estimation. The new viewpoint taken in this paper is that post-contact shear is a useful attribute that can aid servoing and manipulation tasks. For example, when tracking a moving object that rotates and translates relative to a contact region, it is essential that a robot can detect and respond to both shear and normal contact interactions with the tactile sensor.

In this paper, we investigate how the surface pose estimation model can be extended to include shear effects, and we utilise these combined pose-and-shear models to develop a tactile robotic system that can be programmed for diverse non-prehensile manipulation tasks, such as object tracking, surface-following, single-arm object pushing and dual-arm (i.e. stabilised) object pushing. While we have previously demonstrated tactile surface-following and single-arm pushing, our new system is capable of performing these tasks with a continuous smooth motion instead of using discrete, position-controlled movements. Significant new capabilities such as tactile object tracking and dual-arm pushing are only possible because the tactile robotic system can respond to shearing motion.

Two technical challenges had to be overcome to realise a pose-and-shear-based tactile servoing system with a soft high-resolution tactile sensor. Firstly, slippage under shear can cause tactile aliasing (Lloyd et al. (2021)) whereby very similar sensor images may become associated with significantly different poses and shears, which can lead to error-prone pose-and-shear estimation that is not suitable for accurate control. To address this, we modified the CNN into a Gaussian-density neural network (GDN) that predicts both the values and the associated uncertainties of the pose-and-shear components, which we then feed through a Bayesian filter to accurately predict pose-and-shear. The second technical challenge relates to controlling smooth continuous robot motion in 3D space while interacting with an object or surface. This challenge necessitated the use of SE(3) velocity-based servo control methods, which introduced numerous difficulties into the formalism. For example, the standard assumption underlying many Bayesian filters (e.g. Kalman filters) that the normalised product of two Gaussian probability distributions is itself Gaussian does not hold on non-Euclidean manifolds such as SE(3). Hence, the techniques in this paper rely on technical derivations using Lie group theory within the context of robot manipulation control in SE(3), which we present in the Appendices.

Our main novel contributions are summarised below and pictured in Figure 1:

1. A Gaussian-density neural network model that predicts contact pose and post-contact shear with uncertainty from tactile images, and which represents the pose-and-shear as a single, unified vector that transforms under SE(3).

2. A discriminative Bayesian filter that reduces the error and uncertainty of the combined pose-and-shear predictions in SE(3), which enables the use of accurate, noise-reduced estimates for tactile servo control.

3. Feedforward-feedback control methods using velocity control that are driven by tactile pose-and-shear estimation for tactile servo control, supplemented with controllers for goal-based tasks such as object pushing.

4. The application and assessment of these techniques for smoothly and accurately controlling single- and dual-armed tactile robotic systems for object tracking, surface-following, single-arm pushing and dual-arm pushing.

Figure 1.

Pose-and-shear-based tactile servo control (left) applied to four tasks (right): (a) object tracking; (b) surface-following; (c) single-arm object pushing; (d) dual-arm object pushing. Here the servo control loop for each tactile robot has: (1) a Gaussian-density network (GDN) model for predicting the contact pose and post-contact shear with uncertainty from a tactile image; (2) an SE(3) discriminative Bayesian filter for reducing the error and uncertainty of pose/shear estimates; (3) a feedforward-feedback controller that outputs a velocity control signal to (4) a robot arm fitted with a vision-based tactile sensor as an end-effector. Examples of tactile data under different contact poses and post-contact shears are given later in Figure 11.

This paper is organised as follows. Section 2 gives an overview of tactile pose-and-shear estimation followed by a survey of tactile servoing and object pushing. In the methodology, Section 3.1 introduces our definition of surface pose-and-shear in SE(3), then goes on to describe the data collection and modelling procedures for CNNs with a regression head, and a GDN architecture. Section 3.2 derives our algorithms for discriminative Bayesian filtering over SE(3) random variables with uncertainty, referring to results derived in Appendices A-C. Section 3.3 details our methods for feedforward-feedback control of pose-and-shear in SE(3), considering a tactile servoing controller and a combined tactile servoing/pushing controller applied to single- and dual-arm robot systems. Section 4 describes our tactile robot experimental platform and software, comprising a dual robot arm system with soft high-resolution tactile sensors and various test objects. In the experimental results, Section 5.1 begins with examining pose-and-shear information in tactile images. Section 5.2 quantifies and compares the CNN regression and GDN model predictions of pose-and-shear with uncertainty. Then Section 5.3 assesses the error and uncertainty reduction with SE(3) Bayesian filtering. The tactile controller performance is then assessed on four tasks: (1) object pose tracking (Section 5.4) for single and multiple pose components; (2) surface-following (Section 5.5) on a curved ramp and hemisphere; (3) single-arm object pushing (Section 5.6) of geometric objects; and (4) dual-arm object pushing (Section 5.7) with the same geometric objects and taller versions, along with some tall household objects. Finally, Section 6 discusses these results and their limitations, focussing on the tactile pose-and-shear estimation, then the tactile servo control task performance.

A video of the experiments is included as supplementary material and is publicly available on YouTube. We have released the data and code for this paper on github.com/dexterousrobot with a guide and summary on lepora.com/pose-and-shear.

2. Background and related work

2.1. Tactile pose-and-shear estimation

Contemporary methods for tactile pose estimation can be broadly categorised according to whether they estimate a local contact pose or a global object pose. Local contact pose estimation tends to be easier because it can always use tactile information provided in a single contact, although accuracy can be improved by using a sequence of observations or contacts. As such, it has been studied over a longer period of time than the second problem. The global object pose estimation problem is generally harder to solve because it involves fusing information from several different contact interactions, and using this information together with some form of object model to estimate the object pose (Bimbo et al. (2015); Suresh et al. (2021); Villalonga et al. (2021); Bauza et al. (2022); Kelestemur et al. (2022); Caddeo et al. (2023)). As such, work on this type of pose estimation problem is more recent, and this has largely been driven by progress in deep learning models. Comprehensive reviews of pose estimation in the context of robotic tactile perception can be found in Luo et al. (2017) and Li et al. (2020).

In early work on tactile pose estimation, Bicchi et al. proposed a theoretical model for estimating pose-and-shear information, and described a framework for designing tactile sensors that have this capability (Bicchi et al. (1993)). More specifically, their theoretical model addressed the problem of how to determine the location of a contact, the force at the interface and the moment about the contact normals.

In the context of high-resolution vision-based tactile sensors, Yuan et al. showed that the GelSight sensor can be used to estimate the normal contact pose between the sensor and an object surface, but was limited in its contact angle range due to its rather flat sensor geometry (Yuan et al. (2017)). Similarly, Lepora et al. showed that the TacTip soft biomimetic optical tactile sensor could be used to predict 2D contact poses (Lepora et al. (2017; 2019)) and, more recently, 3D contact poses (Lepora and Lloyd (2020; 2021)).

The estimation of post-contact shear is less well-explored than pose estimation. Yuan et al. showed how a GelSight sensor can be used to measure post-contact shear by including printed markers on the sensing surface (Yuan et al. (2015)). Cramphorn et al. described a similar approach for the TacTip sensor based on the shear of marker-tipped pins (Cramphorn et al. (2018)). More recent work has considered data-efficient methods of decoupling the confounding post-contact shear from the primary goal of contact pose estimation, either through Principal Component Analysis (Aquilina et al. (2019)) or the latent feature space of a CNN model (Gupta et al. (2022)). The present study takes a different approach in seeking a predictive model of the components of post-contact shear that can be used alongside a model of contact pose for tactile servo control.

2.2. Tactile servoing and object pushing

Methods for robotic tactile servoing can be grouped according to whether they control attributes in the signal space or feature space of tactile sensor signals or features, or attributes in the task space associated with the problem under consideration. For vision-based tactile sensors or tactile sensors that produce a taxel image, if control is performed in the sensor feature space it can be referred to as image-based tactile servoing (IBTS). Conversely, if control is performed in the task space and the task involves tracking a reference pose with respect to a surface feature it can be referred to as pose-based tactile servoing (PBTS) (Lepora and Lloyd (2021)). In principle, a hybrid approach could also be used, where some aspects of control are performed in the task space and some in the signal or feature space. The tactile servo control methods used in this paper can be viewed as pose-based tactile servoing methods (and more generally as task-space methods) because we combine the contact pose-and-shear motion into a single ‘surface contact pose’ and use it in a feedback loop to control the robot arm motion. However, because of the importance of shear in the control, we refer to it as pose-and-shear-based tactile servoing.

Historically, Berger and Khoslar first used image-based tactile feedback on the location and orientation of edges together with a feedback controller to track straight and curved edges in 2D (Berger and Khosla (1991)). Chen et al. used a task-space tactile servoing approach, using an ‘inverse tactile model’ similar in concept to a pose-based tactile servoing model, to follow straight-line and curved edges in 2D (Chen et al. (1995)). Zhang and Chen used an image-based tactile servoing approach and introduced the concept of a ‘tactile Jacobian’ to map image feature errors to task space errors (Zhang and Chen (2000)). They used their system to track straight and curved edges in 2D and to follow cylindrical and spherical surfaces in 3D. Sikka et al. drew inspiration from image-based visual servoing to develop a tactile analogy using the taxel images produced by a tactile sensor to control the robot arm movement. They applied their tactile servoing system to the task of rolling a cylindrical pin on a planar surface (Sikka et al. (2005)).

Later on, Li et al. advanced the tactile servoing approach of Zhang and Chen to demonstrate a wider selection of servoing tasks including 3D object tracking and surface-following (Li et al. (2013)). Lepora et al. used a TacTip soft optical tactile sensor with a bio-inspired active touch perception method and a simple proportional controller to demonstrate contour following around several complex 2D edges and ridges (Lepora et al. (2017)), following a related contour-following method with an iCub fingertip (Martinez-Hernandez et al. (2017; 2013)). Sutanto et al. used learning-from-demonstration to build a tactile servoing dynamics model and used it to demonstrate 3D contact-point tracking (Sutanto et al. (2019)). Kappassov et al. developed a task-space tactile servoing system, similar to the earlier system developed by Li et al., and used it for 3D edge following and object co-manipulation (Kappassov et al. (2020)). More recently, Lepora and Lloyd described a pose-based tactile servoing approach that uses a deep learning model to map from the tactile image space to pose space, firstly in 2D (Lepora et al. (2019)) and then in 3D (Lepora and Lloyd (2021)). They used this approach to demonstrate robotic surface and edge following on complex 2D and 3D objects.

Most current approaches for robotic object pushing also fall into two main categories: analytical physics-based approaches, which are used in conventional robot planning and control systems; and data-driven approaches for learning forward or inverse models of pusher-object interactions, or for directly learning control policies (e.g. using reinforcement learning). We summarise work on these two approaches in the following paragraphs. A comprehensive survey on robotic object pushing can be found in Stüber et al. (2020).

In the case of analytical, physics-based object pushing, Mason derived a simple rule known as the voting theorem for determining the direction of rotation of a pushed object (Mason (1986)). Goyal et al. introduced the concept of a limit surface to describe how the sliding motion of a pushed object depends on its frictional properties (Goyal et al. (1989)). Lee and Cutkosky derived an ellipsoid approximation to the limit surface, aiming to reduce the computational overhead of applying it in real applications (Lee and Cutkosky (1991)). Lynch et al. used the ellipsoid approximation to obtain closed-form analytical solutions for sticking and sliding pushing interactions (Lynch et al. (1992)). Howe and Cutkosky explored other, non-ellipsoidal geometric forms of limit surface and provided guidelines for selecting them (Howe and Cutkosky (1996)). Lynch and Mason analysed the mechanics, controllability and planning of object pushing and developed a planner for finding stable pushing paths between obstacles (Lynch and Mason (1996)).

In the case of data-driven approaches, Kopicki et al.used a modular data-driven approach for predicting the motion of pushed objects (Kopicki et al. (2011)). Bauza et al. developed models that describe how an object moves in response to being pushed in different ways and embedded these models in a model-predictive control (MPC) system (Bauza et al. (2018)). Zhou et al. developed a hybrid analytical/data-driven approach that approximated the limit surface for different objects using a parametrised model (Zhou et al. (2018)). Other researchers have used deep learning to model the forward or inverse dynamics of pushed object motion (Agrawal et al. (2016); Byravan and Fox (2017); Li et al. (2018)), or to learn end-to-end control policies for pushing (Clavera et al. (2017); Dengler et al. (2022)). In general, analytical approaches are more computationally efficient and transparent in their operation than data-driven approaches, but may not perform well if their underlying assumptions and approximations do not hold in practice (Yu et al. (2016)).

While most object pushing methods rely on computer vision systems to track the pose and other state information of the pushed object, a few (including ours) use tactile sensors to perform this function. Lynch et al. were the first to employ tactile sensing to manipulate a rectangular object and circular disk on a moving conveyor belt (Lynch et al. (1992)). Jia and Erdmann used a theoretical analysis to show that the pose and motion of a planar object with known geometry can be determined using only the tactile contact information generated during pushing (Jia and Erdmann (1999)). More recently, Meier et al. used a tactile-based method for pushing an object using frictional contact with its upper surface (Meier et al. (2016)).

From a control perspective, the most similar approaches to our method for single-arm robotic pushing are the ones described by Hermans (Hermans et al. (2013)) and Krivic (Krivic and Piater (2019)). The similarities and differences are described in more detail in Lloyd and Lepora (2021), but the main difference from our method is that they both used computer vision techniques to track the state of the pushed object, rather than tactile sensing and proprioception.

3. Computational methods

3.1. Contact pose-and-shear prediction with uncertainty

3.1.1. Surface contact pose-and-shear

In previous work on tactile pose estimation (Lepora and Lloyd (2020, 2021); Lloyd and Lepora (2021)), we assumed that a surface contact pose can be represented by a 6-component vector, $\left(\mathrm{0,0},z,\alpha ,\beta ,0\right)$ , where the z-component denotes the contact depth and the $\left(\alpha ,\beta \right)$ -components denote the two orientation angles of the sensor with respect to the surface normal. The three remaining components were set to zero because we assumed that all surface contacts are invariant to $\left(x,y\right)$ -translation and γ-rotation parallel to an idealised flat surface.

In this paper, we instead train a model to estimate all six non-zero components of a surface contact pose, $\left(x,y,z,\alpha ,\beta ,\gamma \right)$ . To do this, we identify the $\left(z,\alpha ,\beta \right)$ components with surface contact pose, as previously, and the $\left(x,y,\gamma \right)$ components with post-contact shear. This combination of pose-and-shear into a single vector arises from the geometry of a flat planar surface, in that the pose components where the surface remains invariant $\left(x,y,\gamma \right)$ happen to be along the primary motions that cause shear.

Thus, in our new definition of a surface contact pose and post-contact shear, we make the following two simplifying modelling assumptions:

1. All contacted surfaces can be locally approximated as flat.

2. All sensor-surface contacts that produce a sensor output can be approximately decomposed into an equivalent normal contact motion followed by a tangential post-contact shear motion.

In practise, we find that the tactile servo control methods apply equally well to curved objects and in situations when the sensor output depends on the time history of the combined normal and shear motion. Thus, with these two simplifying assumptions in mind, we now define the surface contact poses we use to train our pose-and-shear estimation models (Figure 2) and describe the process we use to sample and generate the data.

Figure 2.

Definition and generation of surface contact poses using a two-step process of normal contact motion followed by translational and rotational shear. Step 1a: prior to normal contact motion, the sensor is rotated by Euler angles (α, β) with respect to the surface plane. Step 1b: the sensor is brought into normal contact with the surface through distance z. Step 2: the sensor is translated by (x, y) parallel to the surface and rotated through angle γ about the normal contact axis.

We start by attaching a sensor coordinate frame {s} to the centre of the hemispherical sensor tip so that the z-axis is directed outwards from the tip of the sensor, along its radial axis. We also attach a surface feature frame {f} to the surface so that its z-axis is normal to and directed inwards towards the surface. The {f}-frame is also located so that it is aligned with the sensor frame {s} when the sensor is in its initial position, just out of normal contact with the surface.

As discussed above, the surface contact motion is assumed equivalent to one that is carried out in two stages: a normal contact motion followed by a post-contact, tangential shear motion. The normal contact motion, represented by an {f}-frame SE(3) transform X_⊥, rotates the sensor by Euler angles (α, β) with respect to the surface (assuming an extrinsic-xyz Euler convention) and then brings it into normal contact with the surface through distance z. The tangential shear motion is represented by another {f}-frame SE(3) transform X_‖, which translates the sensor by a displacement (x, y) parallel to the surface, while simultaneously rotating it about the normal contact axis through an angle γ. Composing these transformations into a single SE(3) transform: X _fs = X_‖X_⊥, we take components in the Euler representation (x, y, z, α, β, γ), to represent the surface contact pose of the sensor in the surface feature frame {f}. Combining the contact pose and post-contact shear information in this way simplifies the subsequent filtering and control stages because it avoids the need for two separate filters and two pose-based controllers.

3.1.2. Training data collection

We collect data for training the pose-and-shear prediction models by using a robot arm to move the sensor into different surface contact poses with a flat surface, then apply a shear motion before recording the tactile sensor image. Each data sample consists of a tactile image together with the corresponding surface pose-and-shear in extrinsic-xyz Euler format.

The tactile images associated with surface contact poses and shears (x, y, z, α, β, γ) are sampled according to a two-step procedure that mirrors the definition of the surface contact pose-and-shears given in the previous section (Algorithm 1). Ranges for data collection were as follows:

1. x and y are sampled so that the translational shear displacements are distributed uniformly over a disk of radius r_max = 5 mm centred on the initial point of normal contact with the surface.

2. z is sampled uniformly in the range [0.5, 6] mm, chosen to provide sufficient variation but not damage the sensor.

3. α and β are sampled so that contacts with the sensor are distributed uniformly over a spherical cap of the sensor, which is subtended by angle ϕ_max = 25° with respect to its central axis.

4. γ is sampled uniformly at random over a [−5°, 5°] range, chosen to provide sufficient variation but limit slippage.

The sampling of x and y over a disk, is specified as follows: $\begin{array}{l}x=r\hspace{0.17em}\cos \hspace{0.17em}\theta , y=r\hspace{0.17em}\sin \hspace{0.17em}\theta , r=r_{\max }\sqrt{r^{\prime }},\\ r^{\prime }\sim \mathcal{U}(0,1), \theta \sim \mathcal{U}(0,360°),\end{array}$ (1)where $\mathcal{U}\left(a,b\right)$ denotes a continuous uniform probability density function (PDF) over the interval, $\left[a,b\right]$ . The sampling of α and β over a spherical cap is based on the method outlined by Simon (2015): $\begin{array}{l}\alpha =-\hspace{0.17em}\arcsin \hspace{0.17em}q, \beta =-\hspace{0.17em}\mathrm{atan}\hspace{0.17em}2\left(p,r\right),\\ p=\sin \hspace{0.17em}\varphi \hspace{0.17em}\cos \hspace{0.17em}\theta , q=\sin \hspace{0.17em}\varphi \hspace{0.17em}\sin \hspace{0.17em}\theta , r=\cos \hspace{0.17em}\varphi ,\\ \varphi =\arccos \left(1-\left(1-\hspace{0.17em}\cos \hspace{0.17em}{\varphi }_{\max }\right){\varphi }^{\prime }\right),\\ {\varphi }^{\prime }\sim \mathcal{U}(0,1), \theta \sim \mathcal{U}(0,360°).\end{array}$ (2)

Three distinct data sets were used to develop the pose-and-shear models: a training set of 6000 samples, a validation set of 2000 samples for model selection and hyperparameter tuning, and a test set of 2000 samples for independently verifying the model performance post training. All data were collected using a 3D-printed flat surface (VeroWhite material, shown later in Figure 6(c)).

3.1.3. Pre- and post-processing

We used the following steps to collect and pre-process the tactile images of the training, validation and test sets, and to pre-process tactile images after the model is deployed:

1. Collect at 640 × 480 pixel resolution and convert to 8-bit grayscale.

2. Crop to a 430 × 430 pixel square enclosing the circular region within which the markers are located.

3. Apply a 5 × 5 median blur to remove sensor noise.

4. Apply an adaptive threshold to binarize the image.

5. Resize the images to 128 × 128 pixels.

6. Convert the 8-bit integer pixel intensities to floating point and normalise to lie in the range [0, 1].

We also pre-processed the pose-and-shear labels so that the trained model predictions are in the correct format for the subsequent filtering and controller stages:

1. Convert pose labels from their Euler representations to 4 × 4 homogeneous matrices ${\mathbf{X}}_{fs}\in {\mathbb{R}}^{4\times 4}$ .

2. Invert the 4 × 4 matrices, so that instead of representing sensor poses in the surface feature frames {f} they now represent surface feature poses in the sensor frames {s}: ${\mathbf{X}}_{sf}={\mathbf{X}}_{fs}^{-1}$ .

3. Convert the inverted matrices to exponential coordinates: ${\mathit{\xi }}_{sf}=\ln {\left({\mathbf{X}}_{sf}\right)}^{\vee }\in {\mathbb{R}}^6$ (see Appendix A).

3.1.4. Convolutional neural network for pose-and-shear estimation

Following previous work on tactile pose estimation (Lepora and Lloyd (2020, 2021); Lloyd and Lepora (2021)), we consider a baseline model using a CNN with a multi-output regression head. We configured this CNN architecture to be effective for predicting pose-and-shear, resulting in a sequence of convolutional layer blocks, where each block is composed of a sequence of sub-layers: a 3 × 3 2D convolution; batch normalisation (Ioffe and Szegedy (2015)); a rectified linear unit (ReLU) activation function; and 2 × 2 max-pooling. The feature map dimensions are reduced by half at each block as we move forwards through the blocks, due to the max-pooling. Hence, we balance the progressive loss of feature resolution by doubling the number of features in consecutive layer blocks.

The output of the convolutional base feeds into a fully-connected, multi-output regression head, composed of a flatten layer, dropout layer with dropout probability p = .1 (Srivastava et al. (2014)) and a single fully-connected layer with a linear activation function. When a pre-processed sensor image is applied as input to the CNN, it outputs a surface contact pose-and-shear estimate $\mathit{\xi }\in {\mathbb{R}}^6$ . Where required, we convert these estimates ξ from exponential coordinates in the vector space to 4 × 4 homogeneous matrices using X = exp ( ξ ^∧) (see Appendix A).

We train this CNN regression model by minimising a weighted mean-squared error (MSE) loss function, defined over N training examples and M = 6 network outputs: $\text{MSE}=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{j=1}^M{\alpha }_j{\left({\xi }_{ij}^{\text{label}}-{\xi }_{ij}\right)}^2.$ (3)Here, ${\xi }_{ij}={\left({\xi }_j\right)}_i$ is the jth pose-and-shear component of the ith sample in exponential coordinates, with ξ _ij the regression output and ${\xi }_{ij}^{\text{label}}$ its corresponding label. The loss weights α _j are hyperparameters that can compensate for different output scales and avoid over-fitting when some outputs have larger errors than others. Through trial and error, we found a good set of weights to be α = (1, 1, 1, 100, 100, 100).

We trained these CNN regression models using the Adam optimiser with a batch size of 16 and a linear rise, polynomial decay (LRPD) learning rate schedule. In our implementation of this schedule, we initialised the learning rate to 10⁻⁵ and linearly increased it to 10⁻³ over 3 epochs; we then maintained it for a further epoch before decaying it to 10⁻⁷ over e_max = 50 epochs using a $\sqrt{1-e/e_{\max }}$ polynomial decay weighting factor. We found that a good learning rate schedule can make the training process less sensitive to a particular choice of learning rate and generally improves the performance of the trained model. We used ‘early stopping’ to terminate the training process when the validation loss reached its minimum value over a ‘patience’ of 25 epochs.

3.1.5. Gaussian-density network for pose-and-shear with uncertainty

In this paper, we introduce a modification of the CNN regression head to estimate the parameters of a (Gaussian) pose-and-shear distribution, rather than produce a single-point estimate. This allows us to estimate both the surface contact pose/shear and its associated uncertainty (Figure 3). The motivation for doing this was discussed in our previous work on tactile aliasing (Lloyd et al. (2021)): if we know the uncertainty associated with a pose, this information can be used to reduce the error and uncertainty using other system components such as the Bayesian filter we describe in the next section. In our previous work, we considered a Mixture Density Network (MDN) composed of a mixture of Gaussians. In the present work, we use a single Gaussian to be consistent with assumptions for deriving the update equations for the Bayesian filter in Algorithm 2. We refer to this model as a Gaussian-Density Network (GDN) because it predicts the parameters of a multivariate Gaussian PDF that captures uncertainty in the pose-and-shear outputs.

Figure 3.

CNN with regression and GDN architectures used for surface contact pose estimation. (a) Convolutional base for CNN and GDN models. (b) Convolutional block sub-layer structure. (c) CNN multi-output regression head. (d) GDN PDF estimation head.

Specifically, we use the GDN outputs μ _i and ${\mathit{\sigma }}_i^{-1}$ for the ith tactile data sample to estimate the parameters of a multivariate Gaussian PDF over the pose-and-shear components in exponential coordinates ${\mathit{\xi }}_{\mathbf{i}}\in {\mathbb{R}}^6$ : $p\left({\mathit{\xi }}_i\right)={\displaystyle \prod }_{j=1}^M\frac{{\sigma }_{ij}^{-1}}{\sqrt{2\pi }}\exp \left(-\frac{1}{2}{\left({\sigma }_{ij}^{-1}\left({\xi }_{ij}-{\mu }_{ij}\right)\right)}^2\right),$ (4)where ${\mu }_{ij}={\left({\mu }_j\right)}_i$ and ${\sigma }_{ij}^{-1}={\left({\sigma }_j^{-1}\right)}_i$ . To simplify the model and reduce the amount of training data needed, we have assumed a diagonal covariance matrix, conditioned on the ith data sample: $\text{diag}\left({\mathit{\sigma }}_i^2\right)=\text{diag}\left(1/{\left({\sigma }_{i1}^{-1}\right)}^2,\dots ,1/{\left({\sigma }_{iM}^{-1}\right)}^2\right)$ .

We train the GDN model by minimising a mean negative log-likelihood loss function over the label values: $\begin{array}{l}\overline{\text{NLL}}=-\frac{1}{N}{\displaystyle \sum }_{i=1}^N\ln \hspace{0.17em}p\left({\mathit{\xi }}_i^{\text{label}}\right)\\ =\frac{1}{N}{\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{j=1}^M\left[\frac{1}{2}{\left({\sigma }_{ij}^{-1}\left({\xi }_{ij}^{\text{label}}-{\mu }_{ij}\right)\right)}^2-\hspace{0.17em}\ln \hspace{0.17em}{\sigma }_{ij}^{-1}\right]+c,\end{array}$ (5)where c = 1/2M ln (2π) is a constant term that can be ignored. Comparing this definition with equation (3), if ${\sigma }_{ij}^{-1}={\sigma }_j^{-1}$ for all i, minimising $\overline{\text{NLL}}$ is equivalent to minimising the weighted MSE. Moreover, the squared inverse standard deviations play the same role as the loss function weights α .

As mentioned above, the GDN model can be viewed as a single-component mixture density network (MDN), which performs a similar function to the GDN but uses a Gaussian mixture model to model the output distribution (Bishop (1994; 2006)). This is relevant because the difficulties encountered when training MDNs are well-documented and include problems such as training instability and mode collapse (Hjorth and Nabney (1999); Makansi et al. (2019)). To overcome these difficulties, we incorporated several novel extensions to our architecture, described below.

1. First, rather than directly estimating the component means and standard deviations of a multivariate Gaussian pose-and-shear distribution (assuming a diagonal covariance matrix), we instead estimate the means and inverse standard deviations. As a result of this, the estimated values appear as products in the mean negative log-likelihood loss function instead of quotients. Otherwise, we found that using a neural network to simultaneously estimate two variables that appear as quotients in a loss function can cause instability or slow progress during training.

2. We introduce a new softbound activation function layer. We use this layer to bound the values of the (inverse) standard deviation within a pre-specified range [x_min, x_max] to prevent it from becoming too large or small: $\begin{array}{l}\text{softbound}\left(x\right)=x_{\min }+\text{softplus}\left(x-x_{\min }\right)\\ -\text{softplus}\left(x-x_{\max }\right),\end{array}$ (6)

where $\text{softplus}\left(x\right)=\ln \left(1+\exp \left(x\right)\right)$ . This softbound layer also helps speed-up training and reduce instability.

• 3. We introduce a novel multi-dropout configuration for multi-output neural networks, which allows us to apply distinct dropout probabilities to different outputs. In general, we found that for our pose-and-shear estimation task, dropout is more effective than other forms of regularisation such as L2 regularisation, and so we needed a way to vary the amount of dropout across the different outputs.

Our GDN architecture uses the same convolutional base as the original CNN architecture, but instead feeds its output through a modified GDN head that includes the enhancements discussed above. Inside the GDN head, the output of the convolutional base is flattened and replicated to a set of 12 dropout layers, one for each of the 12 network outputs (μ _j ) and $\left({\sigma }_j^{-1}\right)$ . Each of these dropout layers feeds into a distinct single-output, fully-connected output layer with a linear activation function. The 6 outputs for $\left({\sigma }_j^{-1}\right)$ are each passed through a softbound layer that bounds them to the range ${\sigma }_j^{-1}\in$ [10⁻⁶, 10⁶]. Since each single-output output layer has its own dedicated dropout layer, we use a higher level of dropout to increase the regularisation on the noisier shear-related outputs, and a lower level of dropout on the remaining pose-related outputs. Through manual tuning, we found a good set of dropout probabilities to be p _μ = (0.7, 0.7, 0.1, 0, 0, 0.4) and ${\mathit{p}}_{{\sigma }^{-1}}=\left(\mathrm{0.1,0.1,0,0,0,0.05}\right)$ .

We trained the GDN model the same way as the CNN model, using the Adam optimiser with a batch size of 16 and the same LRPD learning rate schedule. As before, we terminated the training process when the validation loss reached its minimum value over a ‘patience’ of 25 epochs.

3.2. Bayesian filtering of pose-and-shear

3.2.1. Discriminative Bayesian filtering

We model the sequential pose-and-shear estimation problem using a probabilistic state-space model (Figure 4) that is defined by two interrelated sequences of conditional PDFs. This type of state-space model and inference equations form the basis of many Bayesian filtering algorithms, including the Kalman filter (Kalman (1960); Kalman and Bucy (1961)), extended Kalman filter (EKF) (see Gelb (1974)), unscented Kalman filter (UKF) (Julier et al. (1995); Julier and Uhlmann (1997)) and particle filters (see Särkkä (2013)). To simplify the notation in this section, we use lower-case italic letters to represent continuous random variables, regardless of whether they are scalars, vectors or SE(3) elements.

Figure 4.

Probabilistic state-space model used to describe the relationship between surface contact poses and shears (states) and tactile sensor images (observations) in our Bayesian filter.

The state dynamics model describes how states x _k ∼ p (x _k |x_k−1) are transformed across steps k = 1, 2, … beginning from x₀ ∼ p (x₀). The observation model describes how observations y _k ∼ p (y _k |x _k ) are related to the state x _k at time step k. As is conventional for this type of model, we assume first-order Markov state dynamics and conditional independence of observations: $\begin{array}{l}p\left(x_k|x_{1:k-1},y_{1:k-1}\right)=p\left(x_k|x_{k-1}\right),\\ p\left(y_k|x_{1:k},y_{1:k-1}\right)=p\left(y_k|x_k\right).\end{array}$

The (conditional) PDF over states x _k can then be inferred recursively from the following pair of equations: $\begin{array}{l}p\left(x_k|y_{1:k-1}\right)=\int p\left(x_k|x_{k-1}\right)p\left(x_{k-1}|y_{1:k-1}\right)d{x}_{k-1},\end{array}$ (7) $\begin{array}{l}p\left(x_k|y_{1:k}\right)=\frac{1}{Z_k}p\left(y_k|x_k\right)p\left(x_k|y_{1:k-1}\right)\end{array},$ (8)where the normalisation coefficient Z _k is $Z_k=\int p\left(y_k|x_k\right)p\left(x_k|y_{1:k-1}\right)d{x}_k.$ (9)Here, the first relation (Equation (7)) is known as the prediction step or the Chapman–Kolmogorov equation, and it computes an interim PDF over states x _k at time step k given observations up to time step k − 1. Since the integral marginalises over the state distribution at the previous time step, it can be viewed as computing the PDF of the probabilistic transformation of the previous state by the state dynamics model. Meanwhile, the second relation (Equation (8)) is known as the correction step and uses Bayes’ rule to compute the PDF over states at time step k given observations up to time step k. This step can be viewed as probabilistic fusion of the current observation with the interim state computed in the prediction step.

The observation model can be viewed as a generative model because it specifies how to generate observations y _k given a state x _k . However, as pointed out in Burkhart et al. (2020), we do not always have access to such a model but instead have a discriminative model of the form x _k ∼ p (x _k |y _k ). This alternative type of model corresponds to the situation we are dealing with here, where the GDN model estimates a PDF over states (poses and shears) given an observation (tactile image). To use this type of model in the Bayesian filter equations, we must first invert the original observation model using a second application of Bayes’ rule and then substitute the result back into the correction step of equation (7) to give a modified correction step: $p\left(x_k|y_{1:k}\right)=\frac{1}{Z_k^{\prime }}\frac{p\left(x_k|y_k\right)p\left(x_k|y_{1:k-1}\right)}{p\left(x_k\right)}.$ (10)Here, the p (y _k ) term has been absorbed in the modified normalisation constant $Z_k^{\prime }$ . If we also assume a constant prior p (x _k ), we can further simplify this modified correction step to a normalised product of PDFs: $p\left(x_k|y_{1:k}\right)=\frac{1}{Z_k^{″}}p\left(x_k|y_k\right)p\left(x_k|y_{1:k-1}\right),$ (11)where the constant prior p (x _k ) has been absorbed in the modified normalisation constant $Z_k^{″}$ .

Then we can reinterpret Equations (7) and (11) as a discriminative Bayesian filter that updates a filtered state PDF $p\left(x_k^{\text{fil}}\right)$ over steps k = 0, 1, 2, …. This filter uses an intermediate computation of the belief PDF $p\left(x_k^{\text{bel}}\right)$ in the prediction step, which is fused with the observation PDF $p\left(x_k^{\text{obs}}\right)$ in the correction step: $\begin{array}{l}p\left(x_k^{\text{bel}}\right)=\int p\left(x_k^{\text{bel}}|x_{k-1}^{\text{fil}}\right)p\left(x_{k-1}^{\text{fil}}\right)d{x}_{k-1}^{\text{fil}},\end{array}$ (12) $\begin{array}{l}p\left(x_k^{\text{fil}}\right)=\frac{1}{Z}p\left(x_k^{\text{obs}}\right)p\left(x_k^{\text{bel}}\right)\end{array}.$ (13)

For Kalman filters, it is standard to assume Gaussian PDFs. Then the filter is equivalent to updating the means and covariance matrix for $x_k^{\text{fil}}\sim \mathcal{N}\left({\mathit{\mu }}_k^{\text{fil}},{\mathbf{\Sigma }}_k^{\text{fil}}\right)$ using an intermediate estimate of the belief PDF $x_k^{\text{bel}}\sim \mathcal{N}\left({\mathit{\mu }}_k^{\text{bel}},{\mathbf{\Sigma }}_k^{\text{bel}}\right)$ that is then fused with the observation PDF $x_k^{\text{obs}}\sim \mathcal{N}\left({\mathit{\mu }}_k^{\text{obs}},{\mathbf{\Sigma }}_k^{\text{obs}}\right)$ . The state dynamics model $p\left(x_k^{\text{bel}}|x_{k-1}^{\text{fil}}\right)$ can be as simple as shifting the filtered mean ${\mathit{\mu }}_{k-1}^{\text{fil}}$ by an approximate displacement and increasing the covariance matrix ${\mathbf{\Sigma }}_{k-1}^{\text{fil}}$ by a constant amount.

A similar approach has been used to derive some discriminative variations of the Kalman filter, referred to as the Discriminative Kalman Filter (DKF) and robust DKF (Burkhart et al. (2020)). However, in that work the authors modified the inference equations after specialising the state-space model to a linear-Gaussian model. We do not follow that approach here because we cannot specialise to standard Gaussian state distributions due to the complexities with SE(3) state variables discussed in the next section. However, it is nevertheless reassuring to know that if we had assumed a linear-Gaussian model with our more general equations (equation (7) together with equation (10) or Equation (11)), we would obtain the same filter update equations as other works.

3.2.2. Discriminative Bayesian filtering in SE(3)

We now describe how the discriminative Bayesian filtering from the previous section is implemented on a sequence of SE(3) pose-and-shear observations with uncertainty. Algorithm 2 iteratively implements the filter by applying the prediction step in equation (12) and correction step in equation (13) to the sequence of observation PDFs. As part of this computation, Algorithm 3 implements the correction step as this requires combining two SE(3) PDFs, which is complicated by the fusion of two SE(3) PDFs being no longer of the same form, so an approximate method is needed (see Appendix C).

We assume a sequence of observations of the surface pose-and-shear with uncertainty $\left({\overline{\mathbf{X}}}_k^{\text{obs}},{\mathbf{\Sigma }}_k^{\text{obs}}\right)$ estimated by the GDN model from the kth tactile sensor image I _k , along with a corresponding sequence of sensor poses ${\mathbf{X}}_k^{\text{sens}}$ from the robot arm kinematics. The pose-and-shear (mean) estimates from the GDN model also need converting from the ${\mathit{\mu }}_k\in {\mathbb{R}}^6$ exponential coordinate representation of $\mathfrak{s}\mathfrak{e}\left(3\right)$ to 4 × 4 matrices ${\overline{\mathbf{X}}}_k\in SE\left(3\right)$ representing the translational and rotational components of the pose-and-shear (Appendix A): ${\overline{\mathbf{X}}}_k^{\text{obs}}=\exp \left({\mathit{\mu }}_k^{\wedge }\right), {\mathbf{\Sigma }}_k^{\text{obs}}=\mathcal{J}\left({\mathit{\mu }}_k\right)\text{diag}\left({\mathit{\sigma }}_k^2\right)\mathcal{J}{\left({\mathit{\mu }}_k\right)}^{\top }.$ (14)Here, the covariance matrix ${\mathbf{\Sigma }}_k=\text{diag}\left({\mathit{\sigma }}_k^2\right)\in {\mathbb{R}}^{6\times 6}$ is transformed from the GDN output by the Jacobian $\mathcal{J}\left(\mathit{\mu }\right)$ because our chosen method of representing SE(3) random variables is to use a mean that is left-perturbed by a zero-mean Gaussian random variable. With reference to Appendix B, we invert the covariance expression in equation (38) to find this covariance matrix Σ _k .

The sequence of sensor poses ${\mathbf{X}}_k^{\text{sens}}$ is used in the state dynamics model to find the deterministic component ${\overline{\mathbf{T}}}_k\in SE\left(3\right)$ of the probabilistic transformation from the change in sensor pose between steps k − 1 and k: ${\overline{\mathbf{T}}}_k={\left({\mathbf{X}}_k^{\text{sens}}\right)}^{-1}{\mathbf{X}}_{k-1}^{\text{sens}},$ (15)which we assume approximates the change in object pose between those steps. Around this deterministic component, a zero-mean Gaussian noise term ϕ (with covariance Σ _ϕ ) represents the uncertainty in the change in object pose between steps (e.g. due to motion of the object relative to the sensor). Unless otherwise specified, we use the following state dynamics noise covariance in the Bayesian filter algorithm for the experiments in this paper: ${\mathbf{\Sigma }}_{\varphi }={\sigma }_{\varphi }^2{\mathbf{1}}_{6\times 6},$ (16)with σ _ϕ = 0.5, which is equivalent to using a standard deviation (i.e. uncertainty) of 0.5 mm/s and 0.5 deg/s in the three translational and three rotational components of ϕ _k .

The observations and states described above are combined using an SE(3) Bayesian filter (Algorithm 2) that computes the filtered surface pose-and-shear with uncertainty. The prediction step implements a state dynamics model that updates $\left({\overline{\mathbf{X}}}_k^{\text{bel}},{\mathbf{\Sigma }}_k^{\text{bel}}\right)$ according to the change in sensor pose ${\overline{\mathbf{T}}}_k$ , adding a noise term Σ _ϕ to the covariance matrix. The correction step updates $\left({\overline{\mathbf{X}}}_k^{\text{fil}},{\mathbf{\Sigma }}_k^{\text{fil}}\right)$ by combining the belief output of the prediction step and the observed poses and shears with uncertainty, using an approximate SE(3) data fusion method (Algorithm 3).

The derivation of the prediction step of the SE(3) Bayesian filter is given in Appendix B and the derivation of the correction/data fusion step is given in Appendix C.

3.3. Feedforward-feedback control of pose-and-shear

3.3.1. Feedforward-feedback control in SE(3)

In our past work, we made extensive use of feedback control systems for pose-based tactile servo control (Lepora and Lloyd (2021); Lloyd and Lepora (2021)). For the feedback control, we defined the pose error as the SE(3) transformation that moves the observed sensor pose to a reference pose in the same coordinate frame. We do the same here, with pose error: ${\mathbf{E}}_{\mathbf{X}}={\mathbf{X}}^{-1}{\mathbf{X}}_{\text{ref}},$ (17)where E _X and X_ref are specified in the local frame associated with the pose X ∈ SE(3). Then the pose error right-multiplies the observed pose X to give the reference pose X_ref = XE _X .

To define the control operations, we project this pose error into the exponential coordinates for the Lie algebra $\mathfrak{s}\mathfrak{e}\left(3\right)$ , mapped onto the vector tangent space ${\mathbb{R}}^6$ : ${\mathbf{e}}_{\mathbf{X}}=\ln {\left({\mathbf{X}}^{-1}{\mathbf{X}}_{\text{ref}}\right)}^{\vee }.$ (18)Likewise, e _X is regarded as the pose error in the local tangent space to the SE(3) transformations at pose X ∈ SE(3). This error is also the right-perturbation that transforms the observed pose X to the reference pose ${\mathbf{X}}_{\text{ref}}=\mathbf{X}\hspace{0.17em}\exp \left({\mathbf{e}}_{\mathbf{X}}^{\wedge }\right)$ .

This control signal ${\mathbf{e}}_{\mathbf{X}}\in {\mathbb{R}}^6$ can be used to directly control the robot for velocity-based control, or treated as a right-perturbation of the current SE(3) pose for position-based control. Since the error is defined in a Euclidean vector space ${\mathbb{R}}^6$ , we can employ all the control frameworks that have been developed for such spaces (e.g. state feedback). Another advantage of using this representation is that for velocity-based control, it can be convenient to use the control signals generated in these spaces to directly control the robot.

In the case of multi-input multi-output (MIMO) proportional control, we use equation (18) to map the observed pose X ∈ SE(3) to a 6-component vector ${\mathbf{e}}_{\mathbf{X}}\in {\mathbb{R}}^6$ , and then compute the control signal using u(t) = K _p e _X (t), where K _p is a 6 × 6 diagonal gain matrix that contains the corresponding proportional gain coefficients. For full MIMO proportional-integral-derivative (PID) control, we use: $\mathbf{u}\left(t\right)=\mathbf{v}\left(t\right)+{\mathbf{K}}_p{\mathbf{e}}_{\mathbf{X}}\left(t\right)+{\mathbf{K}}_i{\int }_{t^{\prime }=0}^t{\mathbf{e}}_{\mathbf{X}}\left(t^{\prime }\right)d{t}^{\prime }+{\mathbf{K}}_d\frac{d{\mathrm{e}}_{\mathbf{X}}}{dt}\left(t\right),$ (19)

where K _i and K _d are the 6 × 6 diagonal gain matrices associated with the integral and derivative errors at time t. For this type of controller, we include a feedforward term v(t) that can generate a control signal in the absence of any error. This term is useful in surface-following tasks, where the tactile sensor on the robot arm should move tangentially to a surface while the sensor remains normal to the surface at a fixed contact depth. Similarly, for object pushing tasks, the tactile sensor on the robot arm should move forwards oriented normal to the contacted surface of the pushed object.

Since our system operates in discrete time, we typically use simple backward-Euler approximations for computing the integral and derivative errors. To reduce noise in the error signal before computing the derivative, we smooth the error using an exponentially-weighted moving average filter with decay coefficient 0.5. We also sometimes clip the integral error between pre-defined limits to mitigate any integral wind-up problems, and clip the output to limit the control signal range. Details of gain coefficients, error or output clipping ranges, feedback reference poses and feedforward velocities (velocity twists) are provided in Appendix D.

3.3.2. Tactile servoing controller

For object tracking and surface-following, we use a tactile servoing controller (Figure 5, top part only) that performs MIMO feedforward-feedback PID control as described in the previous section (see Equation (19)). The goal of this controller is to align the sensor with a reference contact pose in the surface feature frame, while at the same time moving it with a feedforward velocity (set to zero for object tracking) specified in relation to the desired pose. The reference contact pose is usually set so that the sensor is normal to the surface at a fixed contact depth. For surface-following, the feedforward velocity is usually set to be tangential to the surface. The overall effect is that the sensor moves smoothly to track or follow a surface while maintaining normal contact at a constant depth.

Figure 5.

(a) Tactile servoing controller used for all tasks, which is the sole controller for the object tracking and surface-following tasks. (b) For the tactile pushing controller the tactile servoing controller is supplemented with a target alignment controller.

In each control cycle, we start by computing the SE(3) error in the sensor coordinate frame using: ${\mathbf{E}}_{\mathbf{X}}={\mathbf{X}}_{s{s}^{\prime }}={\mathbf{X}}_{fs}^{-1}{\mathbf{X}}_{s^{\prime }f}^{-1}={\mathbf{X}}_{sf}{\mathbf{X}}_{s^{\prime }f}^{-1},$ (20)where X _sf is the observed feature pose (i.e. the surface contact pose) in the current sensor frame that is predicted by the GDN model and subsequently filtered by the Bayesian filter; the other term ${\mathbf{X}}_{\text{ref}}={\mathbf{X}}_{f{s}^{\prime }}={\mathbf{X}}_{s^{\prime }f}^{-1}$ is the target/reference sensor pose in the feature frame. This error is then mapped onto the ${\mathbb{R}}^6$ representation of $\mathfrak{s}\mathfrak{e}\left(3\right)$ using the logarithmic map (Equation (18)). Then the transformed error is sent to a 6-channel MIMO PID controller with the resulting control signal added to the feedforward velocity.

For surface-following, we set the reference sensor frame so that its z-axis is normal to and pointing towards the surface and the feedforward velocity lies in the xy-plane of that frame (tangential to the surface). The adjoint representation of the SE(3) error is used to map the feedforward velocity to the observed sensor frame before adding it to the feedback signal, so that v in equation (19) is Ad (X_ss′)u_s′2. Finally, the resulting control signal is used to update the robot end-effector velocity during each control cycle.

3.3.3. Tactile pushing controller

For pushing objects across a surface towards a target, we augment the tactile servoing controller with an additional feedback control element that we refer to as the target alignment controller (Figure 5, bottom part). The target alignment controller tries to steer the object towards the target as it is pushed forward, using sideways tangential movements while the sensor remains in frictional contact with the object. In this configuration, the controller feedforward velocity is specified normal to and into the object surface (rather than tangential to the surface as was done for surface-following). The combined effect of the tactile servoing and target alignment controllers is to get the sensor to push the object towards the target point while trying to maintain normal contact with the pushed object’s surface. Since the tactile serviong controller was discussed in the previous section, we only describe its integration with the target alignment controller here.

The object pushing target is specified as a target pose X _wt in the robot work frame {w}. The target pose is transformed to the reference sensor frame using the sensor pose X _ws obtained from the robot (i.e. proprioceptive information), and the sensor error X_ss′ computed by the tactile servoing controller in equation (20): ${\mathbf{X}}_{s^{\prime }t}={\mathbf{X}}_{s{s}^{\prime }}^{-1}{\mathbf{X}}_{ws}^{-1}{\mathbf{X}}_{wt}.$ (21)The target bearing and distance are computed in the reference sensor frame using: ${\theta }_{s^{\prime }}=\mathrm{atan}\hspace{0.17em}2\left(y,z\right), r_{s^{\prime }}=\sqrt{y^2+z^2},$ (22)where y and z are the target pose translation components extracted from X_s′t ∈ SE(3). The target bearing is subtracted from the reference bearing, which is zero in our case θ_s′r = 0, to obtain the bearing error in the reference sensor frame. This error is sent to a single-input single-output (SISO) PID controller, which generates a scalar control signal that is used as the tangential y-component of the velocity control signal in the reference sensor frame (with other components zero). Since the tactile servoing controller generates a control signal relative to the current sensor pose, the target alignment control signal must be transformed from the reference sensor frame to that of the current sensor frame. We do this using the adjoint representation of the SE(3) error, in the same way that we transform the feedforward velocity signal in the tactile servoing controller. The transformed target alignment control signal is then added to the tactile servoing control signal. Finally, we use the resulting control signal to update the robot end-effector velocity during each control cycle.

As in our previous work on pushing (Lloyd and Lepora (2021)), we zero the output of the target alignment controller when the sensor is less than a pre-defined distance ρ* away from the target (manually tuned to ρ* = 120 mm), so as to maintain stability close to the target. After this point, only the tactile servoing controller remains active. The pushing sequence is terminated when the centre of the sensor tip is closer than its radius of 20 mm from the target. This ensures that the sensor-object contact-point is moved close to the target with minimal overshooting.

3.3.4. Single-arm and dual-arm control configurations

The tactile servoing and object pushing controllers described above can either be used in isolation to control a single robot arm for object tracking, surface-following or single-arm pushing tasks, or they can be used in combination to control multiple robot arms. In the dual-arm pushing task, one arm is controlled by an active/leader pushing controller, while the second arm is controlled using a passive/follower object tracking controller. This dual-arm configuration allows the active pushing arm to control the movement of the object towards the target while the second passive arm helps to stabilise the object to prevent it from toppling.

Another way of viewing the operation of these multi-arm configurations is that each robot arm is attempting to follow a control signal via the feedforward path, while simultaneously trying to satisfy the constraints imposed by the reference contact pose specified in the feedback path. In this scenario, the feedforward control signals can either be generated separately for each arm in a decentralised approach or they can be generated in a centralised, more coordinated manner. The ‘leader-follower’ configuration we use in our dual-arm pushing task is an example of the decentralised approach.

4. Tactile robot experimental platform

4.1. Dual-arm robot platform

For our experiments and demonstrations, we use a dual robot arm system with two Franka Emika Panda, 7 degree-of-freedom (DoF) robot arms. The robot arms are mounted on custom aluminium trolleys with base plates, which are bolted together so that the arms are separated by 1.0 m at their bases and can be used individually or together for collaborative tasks (Figure 1(a)-(d)). Depending on the task, the robots can either be fitted with a TacTip tactile sensor (Figure 6) or a stimulus adaptor as an end-effector (Figure 7(a)-(b)). The tactile sensor can be mounted in a standard downwards-pointing configuration or at a right angle using an adaptor mount (Figure 6(b)).

Figure 6.

TacTip soft biomimetic optical tactile sensor, with (a) sensor mount for a robot arm and (b) right-angled mount. (c) Robot-arm mounted tactile sensor collecting training data.

Figure 7.

End-effector mounts used for object tracking: (a) Flat mount with non-slip foam pads (used for Rubik’s cube and mustard bottle), (b) Concave curved mount (used for foam ball). Objects: (c) Rubik’s cube, (d) Mustard bottle, (e) Soft foam ball. Object (d) is from the YCB Object set (Calli et al. (2015)).

4.2. Tactile sensor

The TacTip soft biomimetic optical tactile sensor (Figure 6) has been used in a wide variety of robotic touch applications and integrated into many robot hands (for reviews, see Ward-Cherrier et al. (2018); Lepora (2021)). The 3D-printed sensor tip consists of a black, gel-filled, rubber-like skin with an internal array of pins capped with white markers, which are imaged with a standard USB camera and LED lighting contained within the sensor body. The TacTip is considered biomimetic because these pins mimic the epidermal papillae structure in human skin on the boundary of the epidermal (outer) and dermal (inner) skin layers (Chorley et al. (2009)), as verified in a comparison to real sensory neuronal data on matched stimuli (Pestell et al. (2022)). Practically, the use of marker tips on pins means the sensor is highly sensitive to both normal contact and shear, because the pins act as levers that amplify small contacts into larger patterns of shear.

The TacTip is well-suited for investigating tactile control because its 3D-printed outer surface (Agilus 30, Stratesys) is fairly robust to abrasion and tears, while also being inexpensive and easy to replace. The soft inner gel (Techsil, Shore A hardness 15) gives a conformability similar to the soft parts of the human hand, making the sensor responsive and forgiving of errors in physical contact. Many variations of the TacTip have been created, from fingertip-sized sensors for anthropomorphic robot hands (Ford et al. (2023)) to the DigiTac version of the low-cost DIGIT (Lepora et al. (2022)).

In this work, we use one or two TacTip sensors with 40 mm diameter hemispherical tips containing 331 marker-tipped pins arranged in a circular array. As in other work using the TacTip with deep learning, we use the raw sensor image with minimal pre-processing as input to a neural network model.

4.3. Test objects

Various test objects and mounts (Figures 6-10) were used in the experiments reported in the results sections of this paper. For training/validation/testing tactile data collection, the tactile sensor was mounted vertically on the end-effector of the arm and brought into contact with a flat 3D-printed surface mounted to the base plate (Figure 6(c)). For the object tracking experiments, end-effector mounts were used to attach flat or concave curved objects to the end of the leader arm (Figure 7(a)-(b)), using a similar flat surface to the one used to collect training data and a set of everyday objects (Rubik’s cube, mustard bottle and soft foam ball) held against an adaptor by the tactile sensor mounted on the second robot arm (Figure 7(c)-(e)). For the surface-following experiments, we used a 3D-printed curved ramp and hemisphere (Figure 8) attached to mounts bolted onto the base plate. For the single-arm pushing experiments, we used four distinct plastic regular geometric objects (Figure 9). For the dual-arm pushing experiments, we used double-height (stacked) versions of the four geometric objects and five everyday objects (mustard bottle, cream cleaner bottle, window cleaner spray bottle, glass bottle and large coffee tin) as tall objects that are challenging because they usually topple when pushed (Figure 10).

Figure 8.

3D-printed objects used in surface-following experiments: (a) curved ramp, (b) hemispherical dome.

Figure 9.

Regular geometric objects used in single-arm pushing experiments: (a) Large blue square prism (479 g), (b) Blue circular prism (363 g), (c) Small red square prism (264 g), (d) Yellow hexagonal prism (310 g).

Figure 10.

Tall (double-height) geometric objects used in dual-arm object pushing experiments: (a) Large blue square prism (566 g), (b) Blue circular prism (436 g), (c) Small red square prism (324 g), (d) Yellow hexagonal prism (373 g). Tall everyday objects used in dual-arm object pushing experiments: (e) Mustard bottle (237 g), (f) Cream cleaner bottle (161 g), (g) Windex window cleaner spray bottle (339 g), (h) Glass bottle (641 g), (i) Large coffee tin (214 g). Objects (e), (f), (g) and (i) are from the YCB Object set (Calli et al. (2015)).

4.4. Software infrastructure

We control the robot arms using a layered software API built on top of the libfranka C++ library (version 0.8.0) of the Franka Control Interface (Franka (2017)). This library provides several software methods that allow users to specify callback functions within a 1 kHz real-time control loop. On top of this, we have a developed an in-house library called pyfranka that provides smooth trajectory generation for position- and velocity-based control, so that velocity, acceleration and jerk constraints are not violated; it also handles any background threads needed for velocity-based control and provides a python wrapper via pybind11. The pyfranka library sits underneath the Common Robot Interface (CRI) python library that has been used in most recent work on tactile sensing with the TacTip. Since the only critical functionality needed for our experiments and demonstrations is the ability to perform Cartesian position/velocity control and query the state of the robot, it should be possible to replace the Franka robot arms and API with any 6-DOF or 7-DOF robot arms that support this functionality.

The OpenCV library (version 4.5.2) is used to capture and process images from the tactile sensor, and TensorFlow (version 2.4) with the included Keras API to develop our neural network models for those tactile images. We also use the transforms3d python library and the software provided with the book Modern Robotics (Lynch and Park (2017)) to manipulate 3D poses, transforms and velocity twists.

We run all of the software components in a Pyro5 distributed object environment on an Ubuntu 18.04 desktop PC. The Pyro5 environment allows us to run several communicating python processes in parallel to ensure real-time performance. Using this approach, we were able to run the low-level 1 kHz control loops, image capture, neural network inference (but not training) and high-level control loops for both robot arms and tactile sensors on a single PC.

5. Experimental results

5.1. Pose-and-shear information in tactile images

In the first experiment, we examine images from the tactile sensor used here (the TacTip) to check that the six considered components of contact pose-and-shear are represented in the tactile data. To do this, we visualise the marker densities of tactile images using a kernel density model (see Silverman (2018)), with Gaussian kernels located at marker centroids and a constant kernel width (15 pixels) equal to the mean distance between adjacent markers.

From these visualisations, we were confident that the sensor images contained enough information to produce these estimates (Figure 11). The size of the low-density blue region in the centre of the image depends on the contact depth, while its location in the image depends on the sensor orientation. Changes in marker density around the periphery of the sensor depend on the post-contact translational shear, and subtler changes within the contact region depend on the post-contact rotational shear. These are a type of feature that CNNs can easily replicate if required by applying a sequence of convolution and down-sampling operations.

Figure 11.

Visualisation of tactile images as corresponding changes in marker density with respect to an undeformed tactile image across the relative surface contact poses and shears annotated below the marker density images.

5.2. Neural network-based pose-and-shear estimation

In this experiment, we compare the performance of our GDN pose-and-shear estimation model against a baseline CNN model with regression head. To ensure a fair comparison, both models were developed using the same three data sets (6000 training set samples, 2000 validation set samples and 2000 test set samples), which were collected as described in Section 3.1.2 and pre-processed as described in Section 3.1.3. We trained the CNN regression model as described in Section 3.1.4, and the GDN models as described in Section 3.1.5. For statistical robustness, we trained 10 models of each from different random weight initialisations and then computed the mean and standard deviation loss (MSE loss for the CNN regression model and mean NLL for the GDN model) and component Mean Absolute Errors (MAEs) for all models on the test data set (Table 1).

Table 1.

Overall MSE / mean NLL loss and pose component MAEs for 10 CNN regression and 10 GDN models (mean values ± standard deviation across 10 models). The lowest mean MAE values for each component are highlighted in bold.

Metric	CNN regression	GDN
MSE/NLL loss	1.43 ± 0.02	−7.20 ± 0.25
MAE v x (mm/s)	0.448 ± 0.005	0.426 ± 0.005
MAE v y (mm/s)	0.451 ± 0.004	0.422 ± 0.005
MAE v z (mm/s)	0.164 ± 0.002	0.123 ± 0.002
MAE ω x (deg/s)	0.88 ± 0.03	0.45 ± 0.01
MAE ω y (deg/s)	1.04 ± 0.02	0.64 ± 0.01
MAE ω z (deg/s)	1.44 ± 0.02	1.16 ± 0.02

The results show that the GDN model produces lower component MAEs than the CNN regression model when evaluated on the test data set. An explanation for this is that the mean NLL loss function used to train the GDN model directly has a variable, estimated uncertainty for each pose component, which in effect increases the error weighting on more confident estimates and decreases the error weighting on less confident ones (see Section 3.1.5). This contrasts with the MSE loss function used to train the CNN regression model, which implicitly assumes a constant, pre-specified uncertainty for each pose component and hence is unable to incorporate variations in the estimated uncertainty.

We visualise the distribution of test set errors by plotting the estimated pose components against the ground-truth pose components for the best-performing model of each type (Figure 12). For the GDN model, we also colour each point according to the precision (inverse variance) estimated by the model for the corresponding pose component. Points coloured in red denote high precision (low uncertainty) estimates and points coloured in blue denote low precision (high uncertainty) estimates.

Figure 12.

Distribution of test errors for best-performing (lowest loss) CNN regression and GDN models. Predicted pose-and-shear values are plotted against their actual values. GDN estimates are coloured by their predicted precision (reciprocal variance).

With reference to these plots, we make the following observations. Firstly, while the errors are significant across all pose components estimated by both models, they are larger for the shear-related components than the normal contact ones. This could be due to aliasing effects, which are more prevalent during shear motion than normal contact motion; for example, at small contact depths, the tactile sensor is prone to slip under translational shear, which would lead to a similar tactile image for a range of shear values (see Lloyd et al. (2021) for an explanation of the effects of tactile aliasing). Secondly, the GDN estimates appear more accurate than the CNN regression estimates, in that the distribution of predicted values is more concentrated around the true values for the GDN model than the CNN regression model, which is consistent with the statistical results presented in Table 1. Finally, the precision (uncertainty) values estimated by the GDN model appear to correlate with the errors, in the sense that the red points tend to lie closer to the imaginary ground-truth line than the blue points. In the following section, we consider the impact that our SE(3) Bayesian filter has on reducing these estimation errors and the associated uncertainty.

5.3. Error and uncertainty reduction using an SE(3) Bayesian filter

To evaluate the effect of the SE(3) discriminative Bayesian filter on the pose-and-shear predictions and uncertainty values produced by the GDN model, we treated the test data set as a sequence of sensor images with corresponding pose-and-shear estimates on consecutive random contacts.

Since the pose changes between consecutive sensor contacts can be computed from the test data labels, we can compute the state dynamics transformation T _k whose mean is computed in Algorithm 2 for time step k: ${\mathbf{T}}_k=\exp \left({\mathbf{\psi }}_k^{\wedge }\right){\mathbf{X}}_k{\mathbf{X}}_{k-1}^{-1}.$ (23)Here, X _k and X_k−1 are the contact poses (i.e. the test data labels) at time steps k and k − 1, and ${\mathit{\psi }}_k\sim \mathcal{N}\left(\mathbf{0},{\mathbf{\Sigma }}_{\mathit{\psi }}\right)$ is a Gaussian noise perturbation applied at time step k, which represents simulated noise in the state dynamics model. We specify the noise perturbation covariance as: ${\mathbf{\Sigma }}_{\mathit{\psi }}={\sigma }_{\psi }^2{\mathbf{1}}_{6\times 6}.$ (24)

The discriminative Bayesian filter (Algorithm 2) was then applied to the GDN pose estimates generated in response to the sequence of test inputs, using equation (23) to compute the SE(3) transformation in the state dynamics model at each time step. We set the corresponding perturbation noise covariance Σ _ϕ in Algorithm 2 equal to the perturbation noise covariance defined in equation (24): Σ _ϕ = Σ _ψ .

As in the single-prediction results, we improved statistical robustness by applying the Bayesian filter to each of the 10 GDN models we trained from different random weight initialisations and evaluated the mean and standard deviation component MAEs for all models on the test data set. We repeated the experiment for four different noise levels σ _ψ , which are specified on a logarithmic scale between minimum and maximum values σ _ψ = 0.01 and σ _ψ = 10.0.

The statistical results presented in Table 2 show that the filtered estimates become more accurate as the noise levels in the real state dynamics and the state dynamics model are reduced. As the noise levels increase, the accuracy reduces to the single time-step prediction results for the GDN models in Table 1, which we consider as the σ _ψ = ∞ case and have included for comparison in Table 2.

Table 2.

Pose component MAEs for 10 GDN models followed by Bayesian filter with different state dynamics noise levels, σ _ψ (mean values ± standard deviation across 10 models). The lowest mean MAE values are highlighted in bold.

Metric	σ ψ = ∞	σ ψ = 10.0	σ ψ = 1.0	σ ψ = 0.1	σ ψ = 0.01
MAE v x (mm/s)	0.426 ± 0.005	0.422 ± 0.006	0.360 ± 0.007	0.160 ± 0.004	0.062 ± 0.004
MAE v y (mm/s)	0.422 ± 0.005	0.421 ± 0.006	0.352 ± 0.006	0.161 ± 0.005	0.065 ± 0.005
MAE v z (mm/s)	0.123 ± 0.002	0.123 ± 0.002	0.121 ± 0.002	0.098 ± 0.001	0.069 ± 0.003
MAE ω x (deg/s)	0.50 ± 0.01	0.49 ± 0.01	0.40 ± 0.01	0.18 ± 0.01	0.08 ± 0.01
MAE ω y (deg/s)	0.64 ± 0.01	0.62 ± 0.01	0.42 ± 0.02	0.18 ± 0.01	0.07 ± 0.01
MAE ω z (deg/s)	1.16 ± 0.01	1.15 ± 0.02	0.82 ± 0.02	0.30 ± 0.02	0.11 ± 0.02

To show explicitly how the GDN model depends on the state dynamics noise, we visualise the distribution of test sequence errors by plotting the filtered pose-and-shear predictions against the actual components for the best-performing GDN model at the different noise levels (Figure 13). With reference to these plots, we make the following observations. Firstly, the accuracy of the filtered estimates increases as the state dynamics noise level σ _ψ is decreased (the magnitude of errors about the imaginary ground-truth line decreases). Secondly, the filtered uncertainty estimates get smaller as the state dynamics noise level σ _ψ is decreased (the proportion of points coloured red and blue increases and decreases, respectively). Both of these observations are a consequence of the state dynamics model becoming more accurate as the noise levels are reduced. This allows more effective combination of consecutive pose estimates, which increases their accuracy and reduces the associated uncertainty.

Figure 13.

Distribution of test errors for best-performing (lowest loss) GDN model followed by Bayesian filter with different state dynamics noise levels σ _ψ . Pose-and-shear values (predicted vs actual) are coloured by their predicted precision.

In the above analysis, the state dynamics noise level σ _ψ is known, so the noise covariance in the Bayesian filter update can be set to that value, σ _ϕ = σ _ψ . In the following experiments, the noise covariance in the state dynamics model is set to a constant σ _ϕ = 0.5 (see text following Equation (16)), which allows for precise control while being able to react quickly to changes in the environment.

5.4. Task 1: Object pose tracking

In this experiment, we show how our tactile robotic system can be configured to track the pose of a moving object. We demonstrate this capability using two robot arms: the first arm (the leader robot) moves an object around in 3D space, while a second arm fitted with a tactile sensor (the follower robot) tracks the motion of the object using the tactile servoing controller described in Section 3.3.2.

There are two parts to this experiment. In the first part, we show that the follower arm can track changes to individual pose components of a moving object. More specifically, we track translational motion along the x, y and z axes of the robot work frame, and α, β and γ rotational motion around these axes. In the second part of the experiment, we show that the follower arm can track simultaneous changes to all pose components while the leader arm moves the object in a complex periodic motion. Another key difference between these experiments is that in the first part we only track a flat surface attached to the end of the leader arm, whereas in the second part we also track several everyday objects that are held in position against the leader arm by the follower arm (Figure 7). Hence, the second part of the experiment also demonstrates a form of dual-arm object manipulation.

For both parts of this experiment, we used the controller parameters listed in Table 6 (Appendix D) in the tactile servoing controller (Figure 5(a), top controller only). The feedback reference pose specifies that the tactile sensor should be orientated normal to the contacted surface at a contact depth of 6 mm. Since the feedforward velocity is not required for object tracking tasks, it is set to zero.

5.4.1. Tracking changes to single pose components

In the first part of the experiment, we initially positioned the follower arm tactile sensor in direct contact with the leader arm flat surface at a contact depth of approximately 6 mm and so that its central axis was normal to the flat surface. We then used the leader robot to move the flat surface through a sequence of 200 mm translations along the −x, y and z axes (of the robot work frame), and then through 60° α, β and γ rotations about these axes (Figure 14(a)). During the tracking sequence, we recorded the end-effector poses and corresponding time stamps for both robot arms at the start of each control cycle. This allowed us to (approximately) match up the corresponding poses for the two arms and plot them in 3D for different points in the trajectory after the experiment had finished (Figure 14(b)-(d)).

Figure 14.

Using the follower arm to track changes to individual components of the leader arm pose. (a) Tracking sequence: translation along −x → y → z axes (1-3), followed by −α → β → γ rotation around these axes (4-6). (b) Leader and follower arm pose-trajectory as leader arm translates along −x, y and z axes. (c) Leader and follower arm pose-trajectory as leader arm rotates about x and y axes (α and β). (d) Leader and follower arm pose-trajectory as leader arm rotates about z axis (γ).

For plots that relate to translational pose components (Figure 14(b))), we removed variation in the rotational components from the response of the follower arm. Similarly, for plots that relate to rotational pose components (Figures 14(c)-(d)), we removed any variation in the translational components from the response of the follower arm. This allows us to focus on individual pose components when evaluating how the follower arm responds to changes in leader arm pose. If we did not do this, but instead plotted the raw unaltered poses, it would make it extremely difficult to compare individual pose components of the leader and follower arms at any point in time, particularly for the rotational components (α, β and γ). That said, in the second part of the experiment below where we vary all components of the pose together, we will be able to plot the raw unaltered poses for both arms to see the correspondence.

The pose-trajectory plots (Figures 14(b)-(d)) show that the tactile sensor on the follower arm tracks changes to individual pose components of the flat surface on the end-effector of the leader arm. The coordinates are for the tool centre point of each robot, which on the leader arm is in the centre of the flat surface and on follower arm is in the centre of the sensor tip. For translations along −x, y and z, the follower pose is displaced by z ≈ 15 mm from the coordinate frame of the leader end-effector, where z points towards the follower end-effector and the follower coordinate frame of the sensor is reversed compared to that of the leader (Figure 14(b)). This displacement is consistent with the 6 mm contact depth and sensor tip radius of 40 mm. Likewise for rotations α and β around the x and y axes, the follower coordinate frame of the sensor tracks along circular arcs of radius ∼15 mm around a point centred on the leader, with the leader coordinate frame reversed and pivoting around that same point (Figure 14(c)). For rotations γ around the z-axis, the follower again tracks the leader with coordinate frame displaced by ∼ 15 mm and reversed in z (Figure 14(d)).

5.4.2. Tracking simultaneous changes to all pose components

In the second part of the experiment, we moved the leader robot arm in a more complex velocity trajectory v(t) where all of the pose components were varied at the same time using the periodic function: ${\mathrm{v}}_j\left(t\right)=\frac{2\pi {\mathrm{b}}_j}{T}\cos \left(\frac{2\pi t}{T}+{\varphi }_j\right),$ (25)where we set an amplitude $\mathbf{b}={\left[\mathrm{75,75,75,25,25,25}\right]}^{\top }$ , phase $\varphi ={\left[\pi /\mathrm{2,0,0,0,0,0}\right]}^{\top }$ and period T = 30 s. The translational and rotational components of the amplitude b have units of mm and degrees, respectively. The velocity trajectory was tracked over three full periods (i.e. 90 s).

In addition to tracking a flat surface attached to the leader arm, as in the first part of the experiment above, in this second part we also tracked several everyday objects (Rubik’s cube, mustard bottle and soft foam ball; Figure 7) held between the two arms as they followed the leader arm trajectory. Again, we recorded the end-effector poses and corresponding time stamps for both robot arms at the start of each control cycle to match up the corresponding poses from both robots and plot them after the experiment had finished.

The time-lapse photos and pose-trajectory plots (Figure 15) show that the follower arm tracks simultaneous changes to all components of the leader arm pose as the leader arm follows a complex periodic trajectory. Moreover, it can also hold an object against the leader arm while it is following its trajectory, thereby implementing a form of 3D object manipulation guided by the leader arm.

Figure 15.

Using the follower arm to track simultaneous changes to all components of the leader arm pose. (a) Tracking sequences for direct (flat) surface contact, Rubik’s cube, mustard bottle and soft foam ball. Leader and follower robot pose-trajectory for a single period of the leader arm trajectory for, (b) direct surface contact, (c) Rubik’s cube, (d) mustard bottle, and (e) soft foam ball. In (c)-(e), the numbered points indicate corresponding poses of the two robot arms at several points in the pose-trajectory.

The leader pose-trajectory is the same for all objects and forms an approx. 150 mm diameter circle with the z-axis of the coordinate frame (blue axis) rotating from orthogonal to tangential to the plane containing the circle at the bottom and top antipodal points (Figures 15(b)-(d)). This z-axis points towards the position of the follower arm, which has its coordinate frame reversed compared to the leader arm. For direct contact, the two arm end-effector positions are about 15 mm apart along the z-axis, resulting in a slightly larger, tilted pose-trajectory for the follower compared to that of the leader. As the object size increases, the pose-trajectory of the follower becomes larger relative to the leader but is still separated along the same orientation of the z-axis, increasing from direct contact ( $\sim 15$ mm) to the Rubik’s cube and mustard bottle (both $\sim 60$ mm) to the ball (∼ 80 mm).

5.5. Task 2: Surface-following

In this experiment, we show how our tactile robotic system can be configured for surface-following tasks for two scenarios: traversing a straight-line projection on the surface of a curved ramp, and traversing a sequence of eight straight-line projections outwards from the centre of a hemispherical dome at 45° intervals. The surfaces used in these two scenarios are shown in Figure 8.

For both parts of this experiment, we use the tactile servoing controller described in Section 3.3.2 with the controller parameters listed in Table 7 of Appendix D. The feedback reference pose specifies that the sensor should be orientated normal to the contacted surface at a contact depth of 3 mm. Since, the feedforward velocity depends on the particular surface-following task being performed, it is specified in each of the following subsections that describe each task.

5.5.1. Surface-following on a curved ramp

For this first surface-following task, we initially positioned the robot arm so that the tactile sensor made contact with the highest part of the curved ramp at a contact depth of 3 mm with the y-axis of the sensor aligned with the y-axis of the robot work frame pointing along the length of the ramp. We set the feedforward velocity to 10 mm/s, with u_s′2 = (0, 10, 0, 0, 0, 0) in the tactile servoing controller (Figure 5(a), top controller only). During surface-following, we recorded the end-effector poses and associated time stamps during each control cycle.

For this surface-following task, the time-lapse photos and pose-trajectory plots show that the robot arm successfully follows this type of gently curving surface while the sensor remains in contact with it and orientated normal to the surface (Figure 16). The pose-trajectory is almost straight along the x-axis of the work frame, but has a small drift, presumably because of a slight surface tilt in the x direction.

Figure 16.

Tactile servoing to follow a curved ramp surface. (a) Time-lapse photos. (b)-(d) Robot arm end-effector pose-trajectory. The red/green/blue arrows in this and the other figures correspond to about one every second, although precise timings vary due to factors such as individual controller details and the real-time processing requirements.

5.5.2. Surface-following on a hemispherical dome

For this second surface-following task, we initially positioned the robot arm so that the tactile sensor made contact with the centre of the dome at a contact depth of 3 mm, with the sensor y-axis aligned with the y-axis of the robot work frame.

When following the ith radial path from the centre of the hemisphere, at angle θ _i = 0, 45°, 90°, …, 315°, we set the feedforward velocity ${\mathbf{u}}_{s^{\prime }2}=\left(10\hspace{0.17em}\cos \hspace{0.17em}{\theta }_i,10\hspace{0.17em}\sin \hspace{0.17em}{\theta }_i,\mathrm{0,0,0,0}\right)$ , over 8 radial paths (1 ≤ i ≤ 8). Specifying the reference pose and feedforward velocity in this way causes the sensor to move at 10 mm/s tangentially to the surface in direction θ _i while remaining at a contact depth of 3 mm. During the surface-following sequence, we recorded the end-effector poses and corresponding time stamps as the sensor moved along each of the radial paths.

For this surface-following task, the time-lapse photos and pose-trajectory plots show that the robot arm end-effector successfully follows this curved surface while the sensor remains in contact and orientated normal to the surface (Figure 17).

Figure 17.

Tactile servoing to follow radial paths from the centre of the surface of a hemispherical dome. (a) Time-lapse photos. (b)-(d) Robot arm end-effector pose trajectories. In (b), the numbered points correspond to different radial paths over the surface.

5.6. Task 3: Single-arm object pushing

In our first object pushing experiment, we demonstrate how our tactile robotic system can be used for single-arm object pushing tasks, similar to those demonstrated in earlier work (Lloyd and Lepora (2021)). A major improvement on that earlier work is that the present system can push an object in a smooth continuous manner rather than the previous discrete point motion, because we now use velocity control rather than the position control used previously. We also now show that our new system can push objects over surfaces with different frictional properties, considering both medium-density fibreboard (MDF) and a soft foam surface.

For the single-arm pushing configuration, we mounted the tactile sensor as an end-effector of the robot arm using a right-angle adaptor (Figure 1(c)) so that it can be moved parallel to the surface during the pushing sequence without the arm getting caught on the surface. At the start of each trial, we positioned the tactile sensor end-effector 45 mm above the surface with central axis parallel to the y-axis of the robot work frame at position (y, z) = (−250, 100) mm in the yz-plane parallel to the surface. We then placed the object centrally in front of the tactile sensor so that the contacted surface of the object was about normal to the sensor axis.

For this experiment, we pushed several regular geometric objects (Figure 9) across the MDF and foam surface. These objects were also used in previous work (Lloyd and Lepora (2021)), except we do not use the triangular prism because it cannot be used for dual-arm object pushing below with both arms contacting a flat surface.

For each trial of the experiment, the robot arm was used to push the object towards the target at position (y, z) = (0, 375) mm while remaining in contact with the object. The location of the target relative to the object’s starting pose means that the robot has to push the object around a bend to reach the target. To control the robot arm, we used the pushing controller described in Section 3.3.3, with the parameters listed in Table 8 of Appendix D.

During each trial, we recorded the end-effector poses and corresponding time stamps over each control cycle. To improve statistical robustness, we repeated the trial five times for each object and computed the mean ± standard deviation final target error across all five trials (Table 3). As in Lloyd and Lepora (2021), we define the final target error as the perpendicular distance from the target to the sensor-object contact normal on completion of the push sequence. This provides a measure of how close the pusher is able to approach the target with the object.

Table 3.

Single-arm pushing final target error (mean ± standard deviation perpendicular distance from target to sensor-object contact normal on completion of push sequence). All statistics are computed over 5 independent trials.

Object	Foam surface (mm)	MDF surface (mm)
Blue square	0.50 ± 0.56	4.45 ± 0.78
Blue circle	6.93 ± 0.06	9.45 ± 0.13
Red square	1.88 ± 0.71	4.38 ± 0.77
Yellow hexagon	0.33 ± 1.32	2.87 ± 0.68

The push sequences are visualised by plotting the end-effector poses in 2D overlaid with approximate poses of the pushed objects at the start and finish points of the trajectory (Figure 18). Our tactile robotic system can push all these regular geometric objects over foam and MDF surfaces to the target, approaching within 10 mm for the blue circular prism and within 5 mm for the other objects (Table 3).

Figure 18.

Using a single robot arm to push regular geometric objects across: (a) a foam surface (time-lapse photos show sequence for blue square prism), and (b) an MDF surface (time-lapse photos show sequence for blue circular prism). In the 2D pose plots, the target is identified by a small red circle and dot.

5.7. Task 4: Dual-arm object pushing

In the second pushing experiment, we use a follower robot arm to constrain and stabilise objects as they are pushed across a flat surface by the leader arm. In many ways, this configuration is similar to that used in the object tracking experiment (Section 5.4.2), where a leader robot arm moved an object in a complex trajectory while a follower arm tracked its motion and held the object with the first arm.

The experiment is split into two parts. In the first part, we use two robot arms to push the objects used in the previous single-arm experiment across foam and MDF surfaces. In the second part of the experiment, we replace the original set of geometric objects with a set of taller, double-height versions together with several taller everyday objects (e.g. bottles and containers). These taller objects cannot be pushed by a single robot arm without toppling over, so the second stabilising follower arm is essential for the task.

For this dual-arm configuration, we mounted tactile sensors on both robot arms using right-angle adaptors (see Figure 1(d)). At the start of each trial, the leader arm and object were positioned as they were positioned at the start of each single-arm pushing trial. Then we positioned the follower arm so that its tactile sensor was approximately opposite the leader arm tactile sensor and normal to the opposite contacted surface. During each trial, we used the leader robot arm to push the object towards the same target as before, at position (y, z) = (0, 375) mm while both end-effectors remained in contact with the object.

To control the leader robot arm, we used the same pushing controller and parameters as for the single-arm configuration. To control the stabilising follower arm, we used the tactile servoing controller described in Section 3.3.2 with the parameters listed in Table 9 of Appendix D.

During each trial for both parts of the experiment, we recorded the end-effector poses and corresponding time stamps at each control cycle to match up the corresponding poses at different trajectory points for plotting.

5.7.1. Pushing regular geometric objects

In the first part of the dual-arm experiment, we used two robot arms to push the same geometric objects we pushed in the single-arm experiment (Figure 19). We repeated the experiment five times for each object and then computed the mean ± standard deviation target error across all five trials (Table 4).

Figure 19.

Using a leader and follower robot arm to push regular geometric objects across: (a) a foam surface (time-lapse photos show sequence for red square prism), and (b) an MDF surface (time-lapse photos show sequence for yellow hexagonal prism). In the 2D pose plots, the target is identified by a small red circle and dot.

Table 4.

Dual-arm pushing target error for short geometric objects (mean ± standard deviation) of the distance from target to sensor-object contact normal on completion of push sequence. All statistics are over 5 independent trials.

Object	Foam surface (mm)	MDF surface (mm)
Blue square	4.43 ± 0.25	5.24 ± 0.24
Blue circle	4.46 ± 3.35	3.58 ± 2.04
Red square	4.97 ± 0.18	4.28 ± 0.15
Yellow hexagon	4.43 ± 0.31	4.44 ± 0.62

We visualised examples of the push sequences by plotting the end-effector poses of both robot arms in 2D and overlaid the approximate poses of the pushed objects at the start and finish points of the trajectory (Figure 19).

The results in Table 4 and Figure 19 show that our dual-arm system can push the regular geometric objects over foam and MDF surfaces, approaching the target to within less than 5 mm for all objects. In contrast to the results for the single-arm configuration, the accuracy achieved for the blue circular prism did not appear much worse than for the other objects. In fact, for the MDF surface, the accuracy obtained for the circular prism was slightly better than the other objects.

5.7.2. Pushing tall objects that are prone to toppling

In the second part of the dual-arm experiment, we used the two robot arms to push a set of taller (double-height) geometric objects and tall everyday objects (Figure 10) across a surface.

For this part of the experiment, we found that we needed to modify the (feedback) reference contact pose used in the pushing controller of the leader robot to (0.5,0,0,0,0,0) and the reference contact pose used in the servoing controller of the follower robot to (−0.5, 0, 3, 0, 0, 0). These modified poses only differ from their defaults (Tables 8 and 9) by 0.5 mm in the first components. The effect is to apply a slight downward force on the pushed side of the object and slight upward force on the stabilised side. This helps prevent these taller objects from catching their leading edges on the surface as they are being pushed. Even so, we were not able to push these taller objects across the foam surface without their leading edges catching, and so we could only perform this part of the experiment on the harder MDF surface.

Once again, we visualised examples of the push sequences by plotting the end-effector poses of both robot arms in 2D and overlaid the approximate poses of the pushed objects at the start and finish points of the trajectory (Figure 20).

Figure 20.

Using a leader and follower robot arm to push tall objects across an MDF surface: (a) tall geometric objects (time-lapse photos show sequence for tall blue square prism), and (b) tall everyday objects (time-lapse photos show sequence for mustard bottle). In the 2D pose plots, the target is identified by a small red circle and dot.

The results in Table 5 and Figure 20 show that our dual-arm system can push the taller geometric and everyday objects over the MDF surface, approaching the target to within less than 7.5 mm for all objects.

Table 5.

Dual-arm pushing target error for tall objects (mean ± standard deviation perpendicular distance from target to sensor-object contact normal on completion of push sequence). All statistics are over 5 independent trials.

Object	MDF surface (mm)
Tall blue square	3.62 ± 0.31
Tall blue circle	7.11 ± 2.18
Tall red square	4.94 ± 0.19
Tall yellow hexagon	4.73 ± 0.38
Mustard bottle	6.24 ± 2.33
Cleaner bottle	7.02 ± 0.95
Windex spray	4.59 ± 1.36
Glass bottle	6.42 ± 0.39
Coffee tin	4.81 ± 2.09

6. Discussion and limitations

In this paper, we proposed and evaluated a tactile robotic system that uses contact pose and post-contact shear estimation to facilitate object tracking, surface-following, and single- and dual-arm object pushing using various configurations of velocity-based control. Our tactile robotic system has two key aspects that enable its generality and ease of control: (a) it estimates both contact pose and post-contact shear, and (b) it enables smooth continuous control of the robot arm by using these estimates to control velocity directly. These aspects enable the robot arm to track objects in six degrees of freedom; this control is either the primary goal, such as in object tracking, or as a secondary constraint on a primary goal, such as when pushing an object along a trajectory while maintaining a contact pose. To achieve these goals, we employed SE(3) Lie group theory to leverage techniques from probability and control theory that were developed originally for Euclidean vector spaces. This novel perspective for our perception and control methodology provides a robust theoretical foundation underlying the experimental demonstrations and results presented here.

6.1. Contact pose and post-contact shear estimation

A key simplifying assumption was to merge the contact pose and the post-contact shear into a unified surface contact pose-and-shear vector (x, y, z, α, β, γ). This was facilitated by the contact pose for a flat planar surface only having meaningful components (z, α, β) in contact depth and angle; the other components translate and rotate parallel to the plane and can be ignored in pose-based tactile servo control (Lepora and Lloyd (2021); Lloyd and Lepora (2021)). Here we use these other components (x, y, γ) to represent post-contact shear, which in combination with the contact pose is represented by a single SE(3) transformation (Section 3.1.1).

This combined surface contact pose-and-shear vector can be estimated using multi-output regression CNNs directly from tactile sensor images, using methods similar to previous work on tactile pose estimation of surfaces and edges (Lepora and Lloyd (2020)). However, for the shear components, the estimates are inaccurate with a larger error even after hyperparameter tuning, particularly for the rotational γ component (Figure 12). A pose-and-shear estimate with this level of error is not suitable for smooth and accurate robot control.

Consequently, we developed a Gaussian-density network (GDN) model that combines the CNN base (feature encoding) architecture with output layers that predict both an estimate of the mean and its uncertainty for each pose-and-shear component (Section 3.1.5). These estimates are slightly more accurate with the GDN model than with the regression CNN model (Table 1), but more importantly the predicted uncertainties become lower as the predicted means fall closer to the ground truths (Figure 12). Hence, the GDN model predictions of the means and uncertainties are suitable for Bayesian filtering to reduce error and uncertainty in a sequence of pose-and-shear estimates.

Therefore, we proposed a novel SE(3) discriminative Bayesian filter to decrease the error and uncertainty of the GDN pose-and-shear estimates. These filtered estimates can be highly accurate for all pose-and-shear components (Table 2 and Figure 13), with this accuracy depending upon the assumed noise in the state dynamics model used with the filter and how well this matches changes in the pose-and-shear across time steps.

Both the GDN model and the SE(3) Bayesian filter were technically challenging to implement. The GDN model can be viewed as single-component mixture density network (Bishop (1994; 2006)), which suffer from problems such as training instability and mode collapse (Hjorth and Nabney (1999); Makansi et al. (2019)). To overcome these difficulties, we incorporated several novel extensions to the model architecture including a softbound activation function, a new multi-dropout regularisation of the outputs and a loss function that depends on the inverse standard deviation. Furthermore, there is no simple closed-form method for an SE(3) Bayesian filter because the standard assumption that the normalised product of two Gaussian distributions is Gaussian does not hold on non-Euclidean manifolds such as SE(3). Instead, we used an iterative approximation to the prediction and correction steps of the filter that we re-derived using techniques in Lie group theory (Appendices A-C).

The most obvious limitation of our pose-and-shear estimation methods is the inaccurate CNN regression and GDN model performance on the shear components (Figure 12). We believe that much of this estimation error is due to tactile aliasing (Lloyd et al. (2021)), whereby similar tactile images in the training set become associated with very different shear labels. Specifically, when the sensor is sheared sufficiently after contacting a surface, it can slip across the surface to result in similar tactile images for a range of post-contact shear labels. If we could prevent this slippage, the complexity of the system might be reduced as the GDN and Bayesian filter could become redundant. However, to do this, the training data would need restricting to samples where slip does not occur, which may be difficult to arrange in practice and could overly restrict the model’s applicability, for example, just to large contact depths. That said, the TacTip is known to be effective at detecting slip (James and Lepora (2020); James et al. (2018)), which potentially could be used to minimise slip in the data collection or provide a label of slip occurrence. Our expectation is that the single-step errors may be reduced but Bayesian filtering will still be needed to reduce the error for accurate control.

Note that including further variation in the trajectories during training could also improve the system performance by giving better model generalisation during the task. At present, our training data collection (Algorithm 1) first moves the sensor normal to the surface to make contact then parallel with the surface to produce shear. Introducing motions that are more like those during the task could give better model predictions, such as trajectories that vary in shape and have both normal and parallel components of motion while in contact. However, the view taken in this paper is that the issue of aliasing from slippage is of greater initial concern and thus the primary issue to focus upon.

Another limitation is that in this study we concentrated on estimating contact poses and shears with flat or gently curving surfaces. Clearly, there is a much wider range of surface features that could be tracked, such as following around edges (Lepora et al. (2019); Lepora and Lloyd (2021)). In principle, one can train pose estimation models on other object features, for example, to predict a 5-component contact pose with a straight edge. However, it would then not be possible to combine a contact pose with a 3-component post-contact shear to form a single 6D pose. Even so, we could still use a GDN model to predict both the contact pose and the post-contact shear motion simultaneously. In this scenario, we would need to combine the outputs of two feedforward-feedback pose controllers (e.g. one for the contact pose and one for post-contact shear), and there would be subtleties to be addressed in how this would be best realised to transform appropriately under SE(3).

6.2. Experimental servo control task performance

In developing our new tactile robotic system, a primary objective was to achieve smooth and continuous motion of the robot arm using velocity control driven by tactile pose-and-shear estimation. We accomplished this aim by updating the velocity of the end-effector during each control cycle, instead of updating its pose-based on tactile pose estimation as considered previously (Lepora and Lloyd (2021); Lloyd and Lepora (2021)). The underlying tactile servoing uses an MIMO PID feedback controller on the SE(3) error between the estimated and reference pose-and-shear (Figure 5, top) with an additional feedforward velocity supplied for some tasks. For object manipulation, this tactile servo controller is augmented with a target alignment SISO PID feedback controller that uses the end-effector pose relative to the goal to steer the object (Figure 5, bottom). Specifically, we control the continuous tangential motion, which improves on our previous method of repeatedly breaking contact and discretely pushing the object (Lloyd and Lepora (2021)).

The pose-and-shear-based tactile servo controller was applied successfully to four distinct tasks: (1) object pose tracking, where a leader robot arm moves an object in 3D space while a follower robot arm uses a tactile sensor to track and hold the object; (2) surface-following, where a robot arm uses a tactile sensor to move smoothly over curved surfaces; (3) single-arm object pushing, where a robot arm uses a tactile sensor to smoothly push an object to a goal location; and (4) dual-arm object pushing, where a leader robot arm uses a tactile sensor to push an object while a follower robot arm uses a tactile sensor to track and hold the object.

Three of these four tasks were made possible by the tactile robotic system’s ability to estimate post-contact shear motion in addition to the contact pose. For object pose tracking, the shear motion is essential to track tangential and rotational motion with respect to the contacted surface while maintaining contact, as is visible in the experiments (Figures 14, 15). For single-arm object pushing, controlling shear is essential both to maintain contact with the object and to steer the object via a tangential motion (Figure 18). Likewise, for dual-arm object pushing, estimating post-contact shear is essential for the follower arm to remain in contact (Figure 19) and hold the tall objects to prevent them being toppled (Figure 20).

For the other task of surface-following, we did not need to control shear to complete the task, so the corresponding gains were set to zero; nevertheless, the pose-and-shear components are mixed in the Bayesian filter over SE(3) so the estimated shear was still used implicitly. In principle, shear control could be used to limit the sliding motion velocity, for example, to move slowly when the tactile sensor is pressed strongly into a high friction surface to avoid damage to the sensor or surface.

One limitation of the tactile robotic system is that we could only successfully push tall objects with two arms on the smooth (MDF) surface, as they kept catching on the foam surface. This is partly due to the nature of the task, as humans can struggle with this too, before they adopt a strategy of partially lifting the object. In principle, the tactile controllers could do this too, but this would be a new task of guiding a lifted object, which is beyond the scope of this investigation.

Another limitation of the present system is the absence of a planning component, which hinders its ability to anticipate and prevent undesired situations, such as collisions between robot arms, or motion trajectories that approach or reach joint limits or singularities. Implementing this planning capability would also be beneficial in scenarios where the system is unable to achieve a global task objective by following a local control objective, such as non-holonomic object pushing and manipulation tasks where an object must be rotated to a target orientation while also being moved to a target position.

For future developments of this type of tactile robotic system, we believe there are many more manipulation tasks that can be achieved by enabling both robot arms to operate in an active configuration, where they are both functioning as leaders and followers to some extent. This would broaden their capacity to collaborate, particularly on tasks relating to more complex types of tactile-enabled object manipulation. Such tasks could span from guided manipulation of an object to a goal pose or insertion/assembly of one object into/onto another, to more general tasks involving multiple tactile sensors to enable fully-dexterous manipulation with pairs of tactile grippers or multi-fingered tactile robot hands.

Supplemental Material