Touchy Subject: A review of Visual-Tactile Learning for Scalable Robotic Manipulation

important definitions

(For more context on these terms, visit the Morpheus Robotics Research Glossary)

Action Chunking

Instead of predicting a single action for the immediate next timestep ((a_t)), the model predicts a sequence of future actions (e.g., (a_t) to (a_{t+10})). This is primarily a smoothing mechanism. If the robot only predicts one step at a time, it tends to jitter; predicting a short trajectory makes motion more coherent.

Backbone

In VISK, the backbone refers to the transformer-based policy trunk (a transformer decoder) that tokenizes each modality (camera views, tactile, proprioception) and appends a learnable action token. The action token output is then passed to the action head.

Behavior Cloning (BC)

Behavior cloning is imitation learning framed as supervised learning. The robot copies expert actions from demonstrations by learning a direct mapping from observations to the action the expert took. It is simple and effective, but it can overfit to shortcuts in the observation space.

Policy

A policy is the mapping from the robot's current observations (vision, touch, proprioception, etc.) to an action (or an action chunk). A policy can be trained by imitation learning (behavior cloning), by reinforcement learning, or other approaches.

Visual-Tactile Learning

Learning methods that fuse visual observations with tactile sensing so the robot can combine what it sees with what it feels, especially for contact-rich manipulation.

Ablation

An ablation removes one component at a time (e.g., the wrist camera, proprioception, or touch) to identify which inputs or architectural choices drive performance.

BAKU Architecture

BAKU proposed and evaluated a set of architectural choices (encoders, backbones, action heads, and other components) for multi-task behavior cloning. VISK uses this best-performing template and focuses its comparisons on sensor and observation ablations, rather than running a full architecture ablation study itself.

MLP (Multi-Layer Perceptron)

A simple feed-forward neural network. In VISK, MLPs are effective for low-dimensional magnetic tactile data, while optical tactile images generally require larger convolutional or transformer encoders.

the paper's hypothesis

The authors propose that tactile sensing is the missing link for contact-rich manipulation, but that the type of tactile data matters. Their core hypotheses are:

1. Visuo-tactile learning outperforms vision-only policies in tasks requiring precision (e.g., insertion).
2. Low-dimensional magnetic touch can perform as well as, or better than, high-dimensional optical touch because it has less sim-to-real gap and fewer spurious correlations for the model to overfit.
3. Policies trained in this multi-modal setup generalize better to unseen object variations.

the paper's approach

The researchers used a modified BAKU-style policy to integrate vision and touch. They compared two primary tactile sensor modalities: optical (DIGIT) and magnetic (AnySkin). These are compared with vision-only options (the control group), and they also conduct observation ablations, including removing proprioception and varying camera configurations (wrist camera and 3rd-person/ego-centric cameras).

Sensor Selection: High Dimensionality (Optical) vs. Low Dimensionality (Magnetic)

High dimensionality (optical): sensors like DIGIT or GelSight place a camera inside a rubber finger. The output is an image (e.g., 64×64). This is information-rich, but it requires a heavy encoder and is prone to visual noise (lighting changes, internal reflections).

Low dimensionality (magnetic): sensors like AnySkin embed a magnet in the skin and Hall effect sensors on the rigid structure. As the skin deforms, the magnetic field shifts. The output is a compact vector (e.g., ~15 values). This is lower bandwidth and often easier to learn from robustly.

The Optical Approach (High Dimensionality)

Following a standard pipeline for optical tactile sensors, the authors treat touch as an image.

1. Keypoint detection: the raw tactile image is encoded to predict contact keypoints.
2. Gaussian blobs: to simplify the state, the system reconstructs a small set of Gaussian blobs representing contact.
3. Dynamics model: a learned model predicts blob evolution to support control.

This approach forces the model to learn which pixels represent contact vs lighting artifacts. It is computationally heavy and can be brittle to visual noise. DIGIT-style pipelines often use multiple steps to estimate contact location and intensity, effectively approximating the state of each contact point. One advantage of optical tactile images is that a single sensor can capture multiple simultaneous contact points, which helps with features like edge contact.

The Magnetic Approach (Low Dimensionality)

Magnetic systems measure magnetic flux in the X, Y, and Z axes relative to each Hall effect sensor.

1. The hardware: a flexible skin covers multiple Hall effect sensors on the fingertip.
2. The data: when the skin deforms (contact), the magnetic field changes and the sensors output a low-dimensional vector.
3. The simplicity: there is no image processing. The raw vector correlates directly with force and shear.
4. Triangulation: multiple Hall sensors allow the point of contact to be triangulated.
5. Maintenance: the skin is replaceable without needing to retrain or replace expensive electronics.

The downside is that the magnetic baseline can shift due to skin sagging under gravity and wear over time. Readings can also be affected by electromagnetic interference when manipulating conductive or magnetic objects.

System Architecture

The system inputs images from multiple cameras (including a wrist camera) and tactile data from the fingertips.

• Encoders: camera inputs are encoded with ResNet-18. AnySkin is encoded with a two-layer MLP. DIGIT tactile images (when used) are encoded with the same ResNet-18 as the cameras.
• Transformer backbone: the encoded camera, tactile, and (optionally) proprioception streams are treated as separate tokens, and a learnable action token is appended before passing through a transformer decoder.
• Action head: a deterministic action head predicts an action chunk, the next ten steps the robot must take to achieve the target action. During inference, the policy uses temporal smoothing to produce smoother motion.

This paper relies on findings from the BAKU research that analyzed which component combinations produce the highest-performing system for multi-modal tasks trained via imitation learning. As a byproduct, VISK does not vary architectural decisions to run a full component ablation; instead, it evaluates the impact of tactile sensing and observation ablations on top of a strong base architecture.

Evaluation Setup

They evaluated the approach on a Ufactory xArm 7 with a standard two-finger gripper, using three static third-person cameras and a wrist-mounted camera. For AnySkin, a tactile tip is attached to the left gripper finger, whereas the right finger has a plain silicone tip of a similar shape and size. For the optical baseline, DIGIT sensors are used on both fingertips.

They conducted real-world trials on four tasks.

1. Plug Insertion. Plugging in a power brick into a power strip.
2. USB insertion. Plugging a USB drive into a port.
3. Card swiping. Sliding a credit card through a reader.
4. Book retrieval. Extracting a book from a shelf.

Note: in three of the four tasks, the robot started already holding the object. Only the book retrieval task required grasp acquisition.

findings and conclusions

The experiments validated the less is more hypothesis regarding tactile data dimensionality.

• Touch beats vision: across all tasks, policies with tactile sensing outperformed vision-only baselines. Vision-only approaches often send a pass that is not aligned with the target, whereas touch adds an alignment step (pause, probe, then commit).
• Magnetic sensors outperformed their optical counterparts: despite being cheaper and lower resolution, magnetic AnySkin performed comparably to (and often better than) DIGIT for manipulation. For static texture detection or detailed surface mapping, optical is still the stronger modality.
• Proprioception hurt performance: adding proprioception (joint angles) as an input feature decreased success rates, likely due to shortcut learning (memorizing arm configuration instead of object-relative state).
• Wrist camera performance: wrist camera and touch outperformed touch alone in almost all cases. The exception was card swiping, where the card itself can occlude the relevant area.
• Alignment behavior: the robot exhibited emergent behavior and spent time wiggling to align using tactile feedback before insertion.

reviewer's opinions & analysis

Tactile Sensors vs. Camera Only

A robot without touch is like a human under local anesthesia. You can see your hand, but you cannot perform fine motor skills because you lack the feedback loop of interaction. It blows my mind that the general approach to date has been to build anesthetized robots. The most common workaround has been to add wrist cameras to maintain a view of the object on approach. It turns out that this stopgap measure continues to be valuable in a world with tactile sensing, since the approach and release are such critical parts of object manipulation.

The most interesting qualitative finding in VISK is the alignment process. The robot would pause, wiggle the USB against the port, and use pressure feedback to verify orientation before pushing. This is a behavior distinct from visual servoing, which requires the robot to guess its position from pixels and then execute blindly. This is a strong data point indicating that touch makes robots more spatially aware. I would like to see this replicated using more challenging tasks and other learning paradigms, such as reinforcement learning or world models.

The Critique of Evaluation Starting Conditions

A significant limitation of this study is that, in 3 of 4 tasks, the robot already held the object. By starting from the premise that the object is already grasped, the paper presents a rosy view of the state of the art (SoTA) in robotic manipulation today. The most likely reason is that this assumption is required to compare performance across options; otherwise, most approaches would have failed to complete the task end-to-end. Even if they could grasp, they would likely struggle to pivot and reorient objects in their hands to execute the desired action successfully (e.g., plugging something in).

However, I commend the inclusion of the book retrieval task. This required a full pick-and-place workflow. It is no coincidence that the target (a book) was larger and easier to grasp than a USB stick. It is worth noting that the book task required 2x as many training data points as any other task, despite being an easier object to manipulate.

Thoughts on Proprioception: VISK vs. MCR

In my previous review of Manipulation Centric Representations (MCR), I noted that proprioception was critical for performance. VISK claims the opposite: proprioception hurts performance. Why the contradiction?

It comes down to how the data is used:

1. In MCR: proprioception was used as a supervisor (loss function) during pre-training. It taught the vision model what to focus on (arm and object being manipulated) and what to ignore (background, lighting flicker, camera jitter).
2. In VISK: proprioception was used as an input feature to the policy.

When you feed joint angles directly into a behavior cloning policy (as VISK did), the robot can cheat. It memorizes arm position rather than learning object-relative relationships. This is a known issue in imitation learning, sometimes called causal confusion. VISK’s finding supports the idea that proprioception can be powerful for aligning representations, but it is not always beneficial as a direct policy input.

We should also consider that this situation is caused by imitation learning. If we were to use reinforcement learning, the model would learn that it receives a higher reward by avoiding overfitting to proprioceptive shortcuts. If we learn via world models, proprioception could help the robot determine its arm position in 3D space, a task that can also be estimated by tracking the gripper position (with or without proprioception).

Challenges with Tactile Sensors

In human tactile perception, mechanical tension across the dermal layers activates specialized ion channels, such as PIEZO-2, which translate physical force into neural signals. This biological system is a sophisticated network of specialized sensors: some are tuned to constant pressure and compression, others detect lateral skin stretching to perceive shear, and a third group responds to high-frequency vibrations to identify texture and the slippage of a failing grip. I’ll leave it at that for now, but plan to provide more detail in a coming blog post.

For robotics, the challenge lies in engineering tactile sensors that can mirror this versatility, integrating the detection of compression, shear, and slip into a single, cohesive feedback loop to achieve human-like dexterity.

Engineering this is a nightmare.

• Optical sensors (DIGIT) offer high resolution, but they are fragile, bulky, and require significant compute to decode.
• Magnetic sensors (AnySkin) are fast, which means they can detect the high-frequency signatures of slip or a fingertip moving over textured surfaces. They are not as strong for static texture mapping, but they are very sensitive to small forces (normal and shear), and they tend to be cheap and robust.

VISK's choice of magnetic skin is pragmatic. In the real world, skins rip. If a robot needs a new fingertip every week, we cannot afford to retrain the whole model or recalibrate a complex camera system. Magnetic skins that are plug-and-play align better with the industrial reality of scaling robot fleets.

Morpheus Pitch

VISK is another positive signal for the value of touch in robot data. Many of the researchers we have spoken with have noted that this is an issue: we need multimodal data.

The paper shows that adding touch improves performance. However, relying on small, lab-collected datasets (such as the ~100 demonstrations used here) is not a viable solution. We need to scale.

We are moving beyond simple imitation. We are building the infrastructure to collect large-scale, multimodal human data, capturing not only video but also tactile and proprioceptive signals, as papers such as VISK have shown are essential.

If you are a researcher looking to collaborate on the next generation of multimodal datasets, or if you want to join our waitlist to help collect this data, visit our website.