A tactile-guided online reinforcement learning framework

TORL-VLA: Tactile-Guided Online Reinforcement Learning for Contact-Rich Manipulation

Huaihang Zheng1, Yi Yang1,4, Kai Ma1, Shenglin Xu1,2, Tian Xie1, Guozheng Li1,5, Xiangyu Wang1,3, Yiren Ma1, Si Liu3, Yinian Mao1, Baoxu Liu1

1Meituan   2Beijing Institute of Technology   3Beihang University   4State Key Lab of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS   5China University of Mining and Technology (Beijing)

Project Video

0:00/0:00

Overview

Tactile-Guided Online Reinforcement Learning

Contact-rich manipulation depends on physical states often difficult to infer from images, such as slipping, jamming, and excessive contact force. Although tactile-aware VLAs improve contact awareness, their behavior remains limited by contact conditions covered in fixed offline data.

TORL-VLA addresses this limitation through tactile-guided online reinforcement learning. A tactile-derived wrench-aware VLA predicts reference actions and future wrench sequences, while a lightweight stage-specific actor-critic learns to refine them through real-world interaction. Training combines policy-generated and human-intervention data; an intervention-censored critic prevents post-intervention success from being wrongly credited to preceding policy actions.

Autonomous evaluation uses fixed model parameters with closed-loop tactile feedback.

Method

Method Overview

TORL-VLA trains a tactile-derived wrench-aware VLA offline to predict reference actions and future wrench sequences. With this reference frozen, stage-specific actor-critic modules learn action refinement through real-world online RL. Final autonomous evaluation uses fixed parameters and closed-loop tactile feedback.

TORL-VLA framework: offline reference training, stage-specific online RL refinement, and autonomous evaluation.
A tactile-derived wrench-aware VLA provides action-and-wrench references for stage-specific online RL refinement.
Stage I

Tactile-Aware Reference Training

Fuse tactile-derived wrench information with vision, language, and robot state. Train the reference model offline to jointly predict action chunks and future wrench sequences.

Stage II

Stage-Specific Online RL Refinement

Keep the VLA frozen and train lightweight stage-specific actor-critic modules through real-world interaction. Measured wrench feedback and predicted wrench cues guide action refinement at contact-critical stages.

Stage III

Autonomous Evaluation

Fix all model parameters and evaluate autonomous execution on long-horizon and standalone tasks. Tactile feedback remains available for closed-loop action refinement.

01 · Reference Model

Action-and-Wrench Reference Model

The reference model fuses tactile-derived wrench information with vision, language, and robot state to jointly predict reference actions and future wrench sequences.

Each fingertip tactile array is mapped to a 6D wrench. Left and right fingertips form a 12D observation.
Tactile encoding, attention and MoE fusion, and a zero-initialized bypass support joint action-and-wrench prediction.
Wrench Representation and Model Details

The calibrated vendor SDK maps each 6 × 8 fingertip tactile array to a 6D force–torque wrench. The temporal representation uses 10 observations from the most recent 2 seconds, including the current measurement. Each observation is expressed relative to the current wrench before encoding.

The tactile token is appended after visual-language-model encoding and fused through attention and soft-routed mixture-of-experts (MoE). A learnable bypass coefficient is initialized to zero. Future wrench targets are changes relative to the current measurement, wt+k − wt.

02 · Online Refinement

Stage-Specific Online Reference Refinement

With the VLA frozen, lightweight stage-specific actors learn to refine its reference actions using measured wrench feedback and predicted wrench cues.

01 · Refiner Inputs

Action-and-Wrench Reference

The VLA remains frozen during online RL.

The frozen VLA predicts H = 50 action steps and future wrench changes. The first K = 10 steps form the references used before replanning.

ArefReference action chunk
WrefPredicted wrench changes

Inputs for Action Refinement The refiner combines the references above with current and recent wrench measurements, visual-semantic features, and robot state.

02 · Stage Routing

Stage-Specific Execution

A frozen stage estimator identifies the current task stage.

Other stages

Execute the VLA reference

Use the frozen reference action chunk directly.

Aref
Contact-critical stages

Execute the selected actor

The stage-specific actor uses the reference signals and feedback to directly output a refined action chunk.

Aφ
Action Horizon and Stage Selection

The VLA predicts H = 50 action steps and future wrench changes. The first K = 10 steps define the action and wrench references used before replanning. Actor inputs include current measured wrench, recent measured wrench history, predicted wrench cues, reference actions, proprioception, and compact visual-semantic features.

The stage estimator is trained offline and remains frozen during online RL. A route change requires confidence above 0.9 for two consecutive chunks. Contact-critical stages include cup insertion, latch locking, egg grasping and placement, and the two standalone tasks.

Training Mechanism

Learning from Policy and Human-Intervention Data

Two critics separate task-return learning from intervention-aware action evaluation. A reference-relative hinge guides policy improvement.

01

Task Critic

Learns task returns from both policy and human-intervention data.

02

Intervention-Censored Critic

Stops post-intervention success from being wrongly credited to preceding policy actions by truncating continuation at takeover boundaries.

03

Reference-Relative IC-Value Hinge

Adds a training penalty when a refined action has a lower IC-critic score than the VLA reference action in the same state.

Critic Targets, IC-Value Hinge, and Behavior Regularization

Critic Targets

The task critic retains standard bootstrapping. At a detected takeover boundary, the IC target applies the intervention cost and sets its continuation term to zero.

Behavior Regularization

Policy-controlled steps
Follow the VLA reference.
Human-controlled steps
Imitate the executed correction.

Reference-Relative IC-Value Hinge

Compare the refined and reference actions in the same state.

ΔQic= [ miniQic,g(i)(ct,Atref) − miniQic,g(i)(ct,Atφ) ]+

Penalty activeThe refined action has a lower IC-critic score than the reference action.

No hinge penaltyThe refined action scores at least as well as the reference. A low absolute score alone does not activate the hinge.

Latch-Lock execution during real-world online RL training.

Experiments

Real-Robot Experiments

Experimental Setup

The latch-box task combines coffee-cup placement, latch locking, and egg handling in one sequence. Block Placement and Test-Tube Insertion are evaluated independently.

Experiments use a Piper arm with a Pika gripper, three camera views, and two Tacta fingertip sensors.

Real-robot latch-box setup with multi-view vision and dual-fingertip tactile sensing.
Three snapshots each of coffee-cup placement, latch locking, egg placement, test-tube insertion, and block placement.
Representative contact-rich tasks, including coffee-cup placement, latch locking, egg placement, test-tube insertion, and block placement.

Evaluation Protocol

Each subtask, full task, and standalone task is tested over 30 autonomous trials with all model parameters fixed. Full-task success requires completing the three latch-box subtasks in sequence without human intervention, reset, safety stop, object drop, or damage.

Task Outcome Criteria

Trial outcomes are judged after execution using the physical criteria below. These judgments are used for reporting only.

TaskSuccess conditionFailure condition
Coffee CupPick cup; insert into holder; release; cup remains upright and stable.Holder miss; rim hang-up; partial insertion; large tilt; drop; unsafe contact.
LatchGrip latch; flip toward locking direction; press into locked state; latch remains secured after release.Missed latch; slip during flipping; missed locking edge; incomplete lock; rebound; unsafe force.
EggGrasp egg; transfer to holder; release; egg remains stable without visible damage.Slip; drop; holder collision; rolling out after release; excessive grasping force; visible damage.
Full TaskComplete Coffee Cup, Latch, and Egg sequentially in one autonomous rollout.Any subtask failure; human intervention; reset; safety stop; object drop; object damage.

Performance on Contact-Rich Tasks

TORL-VLA improves full-task success and completion time over the offline and online refinement baselines.

28/30 Full-task success
165.45 s Full-task average time
Method Coffee Cup Latch Egg Full-task success Full-task average time
π0.518/3015/3020/3012/30199.65 s
TA-VLA19/3017/3020/3012/30204.45 s
ForceVLA21/3020/3022/3015/30195.34 s
TORL-VLA w/o RL25/3023/3025/3021/30191.91 s
RLT26/3025/3025/3023/30175.23 s
TORL-VLA30/3029/3030/3028/30165.45 s

Full-task average time is computed over successful full-task trials. TORL-VLA w/o RL directly executes the learned reference; RLT uses the same frozen reference-guided refinement setting but omits wrench input and the intervention-censored critic.

Full-task success

Successful autonomous rollouts out of 30 trials.

12π0.5
12TA-VLA
15ForceVLA
21Without RL
23RLT
28TORL-VLA

Full-task average time

Lower is better; computed over successful full-task trials.

π0.5199.65 s
TA-VLA204.45 s
ForceVLA195.34 s
Without RL191.91 s
RLT175.23 s
TORL-VLA165.45 s

RLT’s Egg failures mainly arise from overly aggressive grasps, whereas TORL-VLA w/o RL fails primarily from imprecise grasp positioning.

Independent Evaluation

Standalone Tasks

Block Placement and Test-Tube Insertion test contact-guided alignment beyond the latch-box task. Each method is evaluated over 30 autonomous trials per task.

Block Placement 30/30

π0.5: 18/30  ·  TORL-VLA w/o RL: 24/30

Test-Tube Insertion 29/30

π0.5: 17/30  ·  TORL-VLA w/o RL: 23/30

MethodBlock PlacementTest-Tube Insertion
π0.518/3017/30
TA-VLA20/3021/30
ForceVLA19/3020/30
TORL-VLA w/o RL24/3023/30
RLT24/3024/30
TORL-VLA30/3029/30

These standalone trials are excluded from latch-box full-task success and full-task average time. Representative videos appear in Standalone Task Demonstrations.

Ablation Studies

We remove individual reference-model components, then test online RL without wrench feedback or without the IC critic to measure their contributions to task success.

Reference ModelCupLatchEgg
Without wrench history token24/3022/3022/30
Without future wrench prediction25/3021/3024/30
Without MoE fusion18/3017/3019/30
Without physical bypass23/3020/3021/30
Full reference, without RL25/3023/3025/30
Online RLCupLatchEggFull Task
Without wrench context27/3027/3026/3026/30
Without IC critic27/3026/3028/3024/30
Full TORL-VLA30/3029/3030/3028/30

Removing MoE causes the largest drop in the reference-model ablation. Removing wrench context lowers full-task success from 28/30 to 26/30; removing the IC critic lowers it to 24/30.

Online RL Post-Training on Latch

The IC critic supports faster improvement early in Latch training. Policies are compared at the same accumulated interaction times using a shared frozen reference and matched interaction budgets.

Latch successes within a fixed 10-minute evaluation window at each online-data checkpoint.
Successful Latch completions during a fixed 10-minute autonomous evaluation. The horizontal axis is accumulated online interaction time in minutes; parameters are fixed during each evaluation.
Latch success rate at the same accumulated online-data checkpoints.
Latch success rate at the same interaction checkpoints. Each variant collects its own training rollouts and is evaluated with fixed parameters.

Intervention Learning

Intervention-Censored Learning Analysis

On Latch-Lock, we compare learning from human interventions and test how incorrect intervention-boundary labels affect final task success.

False Intervention Boundaries · Latch-Lock

MethodClean TrainingFalse BoundarySuccess Drop
RLIF-style25/3018/307/30
TORL-VLA29/3027/302/30

One false takeover boundary is added per training episode. Final evaluation uses fixed parameters over 30 autonomous trials, without injected boundaries.

Latch-Lock · Zero intervention cost 28/30

TORL-VLA retains 28 successes out of 30 when the scalar intervention cost is set to zero.

Intervention cost and boundary-threshold sensitivity

One originally non-boundary chunk per online-training episode is randomly relabeled as a boundary before replay updates. Logged control-source labels, actions, rewards, and other transition fields remain unchanged. Both methods use matched interaction budgets; final evaluation is frozen and perturbation-free.

The best tested RLIF-style intervention cost is cint = 0.5. TORL-VLA is evaluated across cint ∈ {0, 0.5, 1.0, 2.0} and ρint ∈ {0.3, 0.5, 0.7}. The chunk-level boundary rule uses ρint = 0.5.

MethodIntervention Costs cintLatch-Lock Successes / 30
RLIF-style0.1 / 0.5 / 1.0 / 2.023 / 25 / 21 / 19
TORL-VLA0 / 0.5 / 1.0 / 2.028 / 28 / 29 / 27
Boundary Threshold ρint0.30.50.7
TORL-VLA Latch-Lock28/3029/3029/30

The threshold is applied to the human-controlled fraction of adjacent action chunks during training. Control-source labels are used for learning and are not actor inputs.

Contact Prediction

Future-Wrench Prediction

Predicted changes in left-fingertip force follow the main trends and transients of the measured force over the next 50 steps.

Ground-truth and predicted changes in left-fingertip Fx, Fy, and Fz, in newtons relative to current force, over 50 future steps.
Force changes relative to the current measurement, in newtons. Blue solid lines show measured changes; red dashed lines show predictions.

Contact Generalization

Generalization to Changes in Friction and Compliance

All policies are trained under nominal contact conditions. We then change latch-hinge friction and use a softer cup, with model parameters fixed during testing and nearly unchanged visual appearance.

Latch-Hinge Friction

Low, nominal, and high hinge friction.

MethodLowNominalHigh
π0.510/3015/308/30
TORL-VLA w/o RL18/3023/3015/30
TORL-VLA27/3029/3026/30

Cup Material

Normal and softened cup material.

MethodNormalSoftened
π0.518/3010/30
TORL-VLA w/o RL25/3018/30
TORL-VLA30/3027/30

Entries are successes out of 30. TORL-VLA degrades substantially less than π0.5 and TORL-VLA w/o RL across both unseen physical-property shifts.

Tactile-Guided Correction

Tactile-Guided Correction under Visual Ambiguity

We compare a representative latch-flipping failure with a successful TORL-VLA rollout. Around the critical contact state, camera observations look similar, while fingertip wrench measurements reveal mechanical jamming.

Synchronized comparison of a base-policy latch jam and TORL-VLA success, with three camera views, Cartesian trajectories, and right-fingertip force traces.
Camera views, end-effector motion, and right-fingertip forces are aligned in time within each rollout. Fy points backward and Fx downward in the displayed sensor frame.
Base π0.5 · Jammed latch 6.50 N / 1.28 N

Peak / mean |Fy| in the illustrated failed rollout.

TORL-VLA · Successful correction 1.70 N / 0.27 N

Peak / mean |Fy| in the illustrated successful rollout.

These force statistics describe the two representative rollouts. Aggregate task performance is reported separately in the 30-trial evaluations.

Demonstrations

Contact-Rich Task Demonstrations

Seven tasks illustrate contact-rich execution: five compare the base VLA with the wrench-aware reference, and two additionally evaluate stage-specific online RL refinement.

Reference Model Demonstrations

These five tasks compare the base VLA with the wrench-aware reference model, without online RL refinement. They examine contact-aware execution using tactile-derived wrench information and future-wrench prediction.

Bottle Reorientation

Reorienting the bottle requires contact-aware direction control after the gripper establishes contact with the object.

Success Rate Bottle Reorientation
MethodSuccess Rate
π0.580.0%
Wrench-aware reference100.0%

π0.5

Base policy succeeds on many trials but remains sensitive to final contact direction.

Wrench-aware reference

Wrench-aware action references improve reorientation consistency.

Standalone Task Demonstrations

Block Placement and Test-Tube Insertion require fine alignment after initial contact. We compare the base VLA, TORL-VLA w/o RL, and TORL-VLA to show how online refinement adjusts contact during placement and insertion.

Block Placement

Block placement requires fine contact-guided alignment before the object can be released stably.

30-Trial Evaluation Block Placement
MethodSuccesses
π0.518/30
TORL-VLA w/o RL24/30
TORL-VLA30/30

Counts match the main standalone evaluation. Videos illustrate representative executions.

π0.5

Base policy can leave final alignment errors.

TORL-VLA w/o RL

Wrench-aware reference uses tactile-derived wrench cues for contact-aware placement.

TORL-VLA

Online refinement corrects contact alignment before release.

Analysis & Details

Analysis and Implementation

Explore the training setup, measured contact dynamics, wrench predictions, intervention learning, stage selection, and system runtime.

Training data and online interaction budgets

The reference uses 539 full-task latch-box demonstrations, 327 Block Placement demonstrations, and 338 Test-Tube Insertion demonstrations. Six independent stage-specific refiners are trained through real-world interaction.

RefinerOnline Interaction
Coffee Cup23.5 min
Latch-Lock55.3 min
Egg Grasp29.2 min
Egg Placement27.8 min
Block Placement26.9 min
Test-Tube Insertion35.7 min

Interaction time counts robot execution during online data collection, excluding resets and environment preparation. Observations are queried at 20 Hz and timestamp-aligned. Human stage-success judgments supply sparse binary rewards during training; final evaluation is autonomous.

Reference and Online RL Settings
Reference / ObservationValue
Reference horizon H50
Executed horizon K10
Action / wrench dimension7 / 12
Observation query frequency20 Hz
Wrench history samples / window10 / 2.0 s
Wrench prediction loss weight0.3
Stage-estimator held-out accuracy95.5% (chunk-level)
Online RLValue
Discount factor γ0.99
Update-to-data ratio5
Critic / actor update ratio2:1
Target-network EMA τ0.005
Hidden width256
Actor / critic layers3 / 3; Latch-Lock: 4 / 4
Boundary threshold ρint0.5

The VLA, token module, and stage estimator remain frozen during Stage II. All parameters remain fixed during final autonomous evaluation.

Execution Configuration

Execution settings for the real-robot system. Runtime measurements are listed under Throughput and Runtime.

Reference / executionValue
Robot command representationdelta chunk
Control frequency20 Hz
Replanning interval0.5 s
VLA token dimension2048
Online learning / ACValue
Actor snapshot / checkpoint interval100 / 1000 steps
Intervention cost / threshold1.0 / 0.5
Warmup replay / updates300 chunks / 5k
Measured contact evolution

The measured right-fingertip wrench changes continuously during latch and cup contact. In the cup task, the gripper maintains normal force fz to hold the cup while applying downward force fy during insertion.

Cup insertion: right-fingertip force traces show normal holding force and downward insertion force during placement into the holder.
Initial latch-grasping phase with synchronized camera views, right-fingertip wrench, and action traces.
Latch-flipping phase showing continuous contact variation while the latch is being manipulated.
Future-wrench prediction examples

The future-wrench head predicts contact trends over the next 50 action steps. The examples cover mechanism contact, environmental insertion contact, and delicate-object contact, pairing current and future observations with predicted and measured wrench norms.

Latch task: beginning of locking.
Latch task: end of locking.
Cup task: cup grasping.
Cup task: insertion into the holder.
Egg task: beginning of grasping.
Egg task: end of grasping.
Detailed six-dimensional future-wrench prediction on a latch-locking segment for the left-fingertip tactile sensor, complementing the norm-based examples above.
Intervention-Censored Learning Diagnostics

These training diagnostics compare critic scores and actor behavior at policy-to-human takeover boundaries in Latch-Lock. The IC critic supplies the scores used to compare refined actions with the VLA reference.

Boundary distributionValue
Episodes with replay162
Boundary contexts165
Episodes ≥ 1 boundary125 / 162 = 77.2%
Episodes ≥ 2 boundaries32 / 162 = 19.8%
Episodes ≥ 3 boundaries8 / 162 = 4.9%
Mean / median per episode1.02 / 1
90th percentile / maximum2 / 3
Actor behaviorValue
Logged Qic(c, Adata)-1.003
Reference Qic(c, Aref)-0.573
Final actor Qic(c, Aphi)-0.271
Delta Qic: actor vs. reference+0.301
Delta Qic: actor vs. logged data+0.731
IC-hinge-active boundary contexts21 / 165 = 12.73%
Critic / ActionClean MeanBoundary MeanClean-BoundaryAUC
Qic(c, Adata)0.073-1.0031.0760.999
Qic(c, Aref)-0.007-0.5730.5650.913
Qic(c, Aphi)0.091-0.2710.3630.806
Qtask(c, Adata)0.3330.2470.0860.621
Qtask(c, Aref)0.3160.2410.0740.626
Qtask(c, Aphi)0.3510.2720.0790.618
Actor training trendValue
IC-hinge-active contexts at 10k89 / 165 = 53.94%
IC-hinge-active contexts at 20k66 / 165 = 40.00%
IC-hinge-active contexts at final 33k21 / 165 = 12.73%
10k active contexts resolved by final actor71 / 89 = 79.8%
Mean margin improvement+0.421
Variant / WindowHuman-Controlled StepsAverage InterventionsIntervention-Free Episodes
Without IC critic, last 100 ep.27.78%13.4011.0%
Full model, last 100 ep.25.32%10.6829.0%
Without IC critic, last 50 ep.25.42%13.1012.0%
Full model, last 50 ep.18.12%7.4840.0%
Stage Selection and Routing

The stage estimator identifies the current task step and selects the corresponding actor or the VLA reference. It is trained offline and frozen during online RL; held-out full-task accuracy is 95.5% at the chunk level.

The frozen stage estimator uses visual-semantic, proprioceptive, and wrench context. A route change requires confidence above 0.9 for two consecutive chunks. Held-out full-task accuracy is 95.5% at the chunk level.
Full-task routing trace. Estimated routes align with ground-truth contact-window annotations; remaining mismatches concentrate near gradual phase boundaries rather than stable task stages.
Full-Task Throughput · 60-Minute Evaluation

This evaluation measures successful full-task completions within 60 minutes. The Latch training curves measure a single task within 10 minutes.

Reliability and throughput

Full-task success rate against successful full-task completions within a fixed 60-minute evaluation window.

Full-task success rate and throughput TORL-VLA reaches a 0.933 full-task success rate and 21 successful completions per 60 minutes, outperforming all baselines. 0 2.5 5 7.5 10 12.5 15 17.5 20 22.5 0.0 0.2 0.4 0.6 0.8 1.0 Throughput (Successes / 60 min) Full-task Success Rate TA-VLA: 0.40 success rate, 5 successes per 60 minutes. pi0.5: 0.40 success rate, 7 successes per 60 minutes. ForceVLA: 0.50 success rate, 9 successes per 60 minutes. TORL-VLA without RL: 0.70 success rate, 12 successes per 60 minutes. RLT: 0.767 success rate, 16 successes per 60 minutes. TORL-VLA: 0.933 success rate, 21 successes per 60 minutes. TA-VLA π0.5 ForceVLA TORL-VLA without RL RLT TORL-VLA
Runtime Measurements
Module / QueryTimed CallsMean Latency
Base π0.5 policy query (comparison)3094.7 ms
TORL-VLA wrench-aware reference query30100.2 ms
Stage estimator routing301.7 ms
Stage-specific online actor301.1 ms
End-to-end TORL-VLA query with actor refinement30103.0 ms

Latency is measured over warm-call policy queries on a single RTX 4090. The end-to-end TORL-VLA row is measured directly on the actor-refinement route, not obtained by summing separately timed modules.

Citation

Resources and BibTeX

TORL-VLA code will be released. Our open-source RLT reproduction, Yyshadow/openpi-RLT, is available for readers interested in reference-guided online RL.

@article{zheng2026torlvla,
  title   = {TORL-VLA: Tactile Guided Online Reinforcement Learning for Contact-Rich Manipulation},
  author  = {Zheng, Huaihang and Yang, Yi and Ma, Kai and Xu, Shenglin and Xie, Tian and Li, Guozheng and Wang, Xiangyu and Ma, Yiren and Liu, Si and Mao, Yinian and Liu, Baoxu},
  year    = {2026},
  journal = {arXiv preprint arXiv:2606.09337}
}

Figure detail

View PDF