Teleoperation & Haptics (Bilateral, Force-Reflective)
1. At a glance
Teleoperation is remote control of a robot (“slave” / follower) by a human operator through a “master” / leader device, typically with visual and force feedback. Haptics is the rendering of touch — kinesthetic (force/torque on the limb) and tactile (skin-level vibration, texture, slip).
Bilateral teleoperation closes the force loop in both directions: motion commands flow master to slave, and the contact force at the slave is reflected back to the operator’s hand. The operator feels remote contact as if their own hand were there — provided the controller is stable, transparent, and low-latency.
Mature application domains:
| Domain | Example system | Why teleop / haptics |
|---|---|---|
| Surgery | Intuitive da Vinci Xi, Medtronic Hugo, CMR Versius | Tremor filter, motion scaling, MIS through ports |
| Subsea | Schilling TITAN 4, Forum Perry XLX | Pressure + dark + slow ROV ops |
| Space | NASA Robonaut 2 (ISS), Astrobee, Mars rovers | EVA risk, vacuum, supervisory delay |
| Hazmat / nuclear | PaR Systems hot-cell manipulators, Brokk demolition | Radiation dose, ALARA |
| EOD | Northrop Andros Wolverine, QinetiQ TALON | Standoff from explosives |
| AR / VR | Apple Vision Pro, Microsoft HoloLens 2, HaptX Gloves G1 | Spatial computing, training |
| Training sim | Mimic dV-Trainer, Simbionix LapMentor | Surgical skill acquisition |
By 2024 over 12 million da Vinci procedures had been performed worldwide — the single largest commercial deployment of bilateral teleoperation.
2. First principles
2.1 Architectures (Lawrence 1993)
Four canonical channels can flow between master (subscript m) and slave (s):
| Channel | Signal | Direction |
|---|---|---|
| C1 | velocity / position | m → s |
| C2 | velocity / position | s → m |
| C3 | force / torque | m → s |
| C4 | force / torque | s → m |
- Position–Position (PP): C1 + C2. Cheap, no force sensor needed. Operator feels reflected force only via position-error spring. Poor transparency.
- Position–Force (PF): C1 + C4. Slave needs a 6-axis F/T sensor. Better transparency than PP; common in surgical robotics.
- Force–Force (FF): C3 + C4. Master needs a force sensor as well.
- Lawrence 4-channel: all four active. Ideal transparency theorem — with matched dynamics the operator’s perceived impedance Z_to equals the environment impedance Z_e, independent of slave dynamics.
2.2 Passivity-based stability (Hannaford 1989; Anderson–Spong 1989)
A teleoperator subsystem with input port (f, v) is passive iff ∫₀ᵗ f(τ)·v(τ) dτ + E(0) ≥ 0 for all t. The cascade of passive systems is passive; a passive system in feedback with any passive environment (and passive human arm) is stable. This decouples controller design from unknown human / environment dynamics — the dominant value of passivity for teleop.
2.3 Wave variables (Niemeyer–Slotine 1991)
Communication delay T_d destroys passivity of the raw position+force channels. The wave transform encodes power-flow into two scattering variables:
u = (b·v + f) / √(2b) — right-moving wave v_w = (b·v − f) / √(2b) — left-moving wave
with characteristic impedance b > 0. Any constant time delay applied to (u, v_w) preserves passivity because the delay simply translates the wave without amplifying energy. Trade-off: wave reflections cause sluggishness and “jelly” feel for large b; small b lets noise through.
2.4 Time Domain Passivity Approach — TDPA (Hannaford–Ryu 2002)
Modern practical method. A passivity observer (PO) monitors instantaneous energy flow:
E_obs(k) = E_obs(k−1) + T_s · f(k) · v(k)
When E_obs goes negative (active behavior), a passivity controller (PC) — a variable damper α(k) — dissipates exactly the surplus energy:
α(k) = −E_obs(k) / (T_s · v(k)²) if E_obs < 0, else 0
Unlike wave variables, TDPA gives ideal behavior whenever the channel is naturally passive and only intervenes when needed. Handles time-varying delay and jitter (UDP/IP).
2.5 Z-width (Colgate–Hogan 1997)
For a sampled-data haptic display rendering a virtual wall of stiffness K with intrinsic device damping B at sample period T_s:
K_max · T_s ≤ 2 B (necessary stability bound)
The Z-width is the range of impedances (stiffness × damping) the device can render passively. Increasing B (physical damping) or decreasing T_s (faster servo loop) both widen Z-width. This is why haptic devices target 1 kHz update rate while graphics is 60 Hz — kinesthetic stability demands it.
2.6 Human perceptual limits
| Modality | JND (Weber fraction) | Bandwidth |
|---|---|---|
| Force (hand) | 7 % to 15 % of reference | DC – 20 Hz kinesthetic |
| Position | ≈ 1 mm at fingertip | DC – 10 Hz |
| Orientation | ≈ 1° (wrist) | DC – 10 Hz |
| Vibration (tactile, Pacinian) | 17 % | 50 – 700 Hz, peak ~ 250 Hz |
| Visual | (Vernier) ~ 5 arcsec | flicker fusion ~ 60 Hz |
Practical implications: a teleop force loop running at 1 kHz with 8-bit force quantization wastes nothing; end-to-end latency budget is haptic < 10 ms, visual ≤ 50 ms before the operator notices and adapts negatively.
3. Worked examples
Example A — Bilateral teleop, PF architecture, 1 DOF
Master: 1-DOF haptic stick, mass m_m = 0.1 kg, damping b_m = 0.5 N·s/m. Slave: 1-DOF robot, mass m_s = 2.0 kg, damping b_s = 5 N·s/m, in contact with a soft environment of stiffness k_e = 800 N/m.
Master admittance control follows operator force: m_m·ẍ_m + b_m·ẋ_m = f_h − f_e_reflected
Slave impedance control tracks scaled master position: f_s = K_p · (n·x_m − x_s) + K_d · (n·ẋ_m − ẋ_s), n = 1 (no scaling)
Operator pushes x_m = 5 mm. Steady-state position error at slave with K_p = 5000 N/m against k_e = 800 N/m:
x_s = K_p / (K_p + k_e) · x_m = 5000/5800 · 5 mm = 4.31 mm f_e = k_e · x_s = 800 · 4.31×10⁻³ = 3.45 N
Reflected to operator (gain 1:1): operator feels 3.45 N. Transparency error = 800 / (800+5000) ≈ 14 % of environment impedance lost in the controller. To recover transparency, increase K_p or add feed-forward of f_e (Lawrence 4-ch).
Example B — Wave-variable teleop with 100 ms delay
T_d = 0.100 s (typical surgeon-console-to-patient-cart on adjacent operating suites via fibre; intercontinental satellite would be 0.5 s+). Characteristic impedance b = 30 N·s/m (chosen to match slave damping).
Operator commands step velocity v_m = 0.05 m/s, no slave force yet. Right-moving wave at master: u_m = (30 · 0.05 + 0) / √(60) = 1.5 / 7.746 = 0.194 W^½
After delay arrives at slave: u_s(t) = u_m(t − 0.1). Slave decodes: v_s = (√(2b) · u_s − f_s)/b. With f_s = 0 initially, v_s = √(2·30)/30 · u_s · = 7.746/30 · 0.194 = 0.050 m/s. Velocity arrives intact, delayed 100 ms. Energy is bounded — passive regardless of delay.
When slave hits wall, f_s rises, left-moving wave v_w returns to master 0.1 s later. Operator feels delayed (200 ms round-trip) but stable contact — no oscillation buildup as you would get with raw position+force feedback.
Example C — Z-width / virtual wall ceiling
A Force Dimension omega.7 with intrinsic damping B = 1 N·s/m, servo loop T_s = 1 ms (1 kHz).
K_max = 2 B / T_s = 2 · 1 / 0.001 = 2000 N/m
Programming a virtual wall stiffer than 2000 N/m at 1 kHz makes the device buzz / “kick” — added energy per cycle exceeds dissipation. Mitigations:
- Raise loop rate to 4 kHz → K_max = 8 000 N/m (Phantom Premium does this)
- Add software damping that scales with velocity
- TDPA: clamp commanded force when PO turns negative
- Add structural damping (eddy-current brake) — increases B, widens Z-width
4. Devices
4.1 Desktop / research haptic displays
| Device | Workspace | Peak force | DOF | Update | Price (2024) |
|---|---|---|---|---|---|
| 3D Systems Touch (was Phantom Omni) | 160×120×70 mm | 3.3 N | 3-DOF force | 1 kHz | ≈ $4 k |
| 3D Systems Touch X (was Phantom Desktop) | 160×120×120 mm | 7.9 N | 6-DOF in, 3-DOF out | 1 kHz | ≈ $13 k |
| 3D Systems Phantom Premium 1.5 | 381×267×191 mm | 8.5 N | 6-DOF in, 3- or 6-DOF out | 4 kHz | ≈ $50 k |
| Force Dimension omega.3 | Ø 160 × 110 mm | 12 N | 3-DOF | 4 kHz | ≈ $20 k |
| Force Dimension omega.7 | Ø 160 × 110 mm + grip | 12 N + 8 N grip | 7-DOF | 4 kHz | ≈ $40 k |
| Force Dimension sigma.7 | larger, 6-DOF active | 20 N | 7-DOF active | 4 kHz | ≈ $90 k |
| Haption Virtuose 6D | 700 mm sphere | 35 N peak | 6-DOF | 1 kHz | ≈ $80 k |
| Geomagic Touch (legacy Phantom Omni) | 160×120×70 mm | 3.3 N | 3-DOF force | 1 kHz | discontinued |
4.2 Wearable / glove haptics
- HaptX Gloves G1 (2023): 130 microfluidic tactile actuators per hand, 4 N per finger force feedback via pneumatic exotendon. Used for telerobotics (NASA Valkyrie hand control, Shadow Hand teleop).
- Meta Reality Labs prototype (research only): air-pocket pneumatic gloves.
- bHaptics TactSuit X40 / TactGlove: vibrotactile only (no kinesthetic).
- Apple Vision Pro (2024): no force feedback yet; eye tracking + pinch gestures + EyeSight + 23 M-pixel micro-OLED displays.
4.3 Surgical master consoles
- Intuitive Surgical da Vinci Xi / SP / 5 — 7-DOF EndoWrist instruments, motion scaling 1:1 to 5:1, voice-coil force-cue at console (not full bilateral; “haptic” feedback is largely visual + audio cues).
- Medtronic Hugo RAS (CE 2021, no US clearance as of 2024): modular arms, open console.
- CMR Surgical Versius (CE 2019): bedside arms, 3-DOF wrist console.
- Asensus Senhance: only major surgical platform with full kinesthetic haptic feedback (since 2017, CE-marked).
4.4 Industrial / field teleop
- Shadow Robot Dexterous Hand teleop with HaptX or CyberGrasp glove — research-grade humanoid hand telemanipulation.
- NASA / GM Robonaut 2 (ISS 2011–2018): bilateral teleop via custom glove
- immersive HMD, ground-controlled with up to 1 s delay.
- Boston Dynamics Spot Tablet (supervisory teleop, not bilateral) — click-to-go-here + manipulator joystick.
- Toyota T-HR3 (2017): bilateral humanoid with master suit; demo only.
4.5 Subsea ROV manipulators
- Schilling Robotics TITAN 4 / Conan 7P (resolved-rate or pilot-in-the-loop; rate teleop, not bilateral force).
- Forum Perry XLX series — 7-function ROV arms with master controller.
- Kraft Predator (HD7) — true bilateral master-slave with force feedback, used on US Navy work ROVs.
4.6 EOD / hazardous
- Northrop Grumman Andros series (Mark V-A1, Wolverine, F6A).
- Endeavor Robotics / FLIR TALON (acquired by Teledyne 2021).
- Brokk demolition robots — rate teleop, no force reflection.
5. Applications detail
5.1 Surgical
The dominant commercial case. Master console: open-architecture stereo endoscope view, two 7-DOF manipulanda gripping pencil-like handles, foot pedals for camera/energy/clutch. Slave (patient cart): 4 arms, each carrying an 8-mm EndoWrist instrument. Motion scaling 3:1 typical (1 cm at hand = 3 mm at tip). Tremor filtering: low-pass 6 Hz cutoff removes physiological tremor (8–12 Hz). Clutching: foot pedal disengages master to re-centre.
Haptics on da Vinci is limited — the company has long argued visual feedback is sufficient and full bilateral haptics complicates clearance. Independent academic studies (Massimino–Sheridan 1994; Wagner et al. 2007) and ASENSUS / Senhance commercial deployment dispute this.
5.2 Space teleoperation
- EVA / lunar Gemini-Apollo era: tethered direct control, no haptics.
- Robonaut 2 / Robonaut 5 Valkyrie: dexterous humanoid teleop from ground with 0.5–2 s lunar / cislunar delay → supervised autonomy + Niemeyer-style wave-variable bridge in research builds.
- Astrobee (free-flyer on ISS, 2019–): tablet supervisory teleop.
- Mars rovers (Spirit, Opportunity, Curiosity, Perseverance): no real-time teleop — 4 to 24 minute one-way delay → sequenced sols.
- Canadarm 2 / Dextre: rate teleop from ISS interior, no haptics.
5.3 Deep sea
Pressure (60 MPa at 6 km) precludes human divers below ~ 300 m. Work-class ROVs (HEAVY-WORK-class, e.g. Schilling UHD-III) carry one or two 7-function manipulators. Surface pilot uses spatially-correspondent master with resolved-motion-rate-control (RMRC) and optionally force overlay. Sensor fusion with sonar + HD cameras.
5.4 Nuclear / hot cell
Mechanical master-slave manipulators (Central Research Lab Model M, PaR Systems) date to Goertz 1949 — purely mechanical kinematic chain through shielding wall, no electronics. Modern variants (PaR Systems STT-series) use electric servo with 1:1 or scaled force reflection.
5.5 AR / VR / training simulators
- Mimic dV-Trainer: physical mock-up of da Vinci console + VR scenes, validated for surgical residency curriculum.
- Simbionix LapMentor (3D Systems): laparoscopic trainer with haptic forceps using 3D Systems Touch X engine.
- Apple Vision Pro / HoloLens: gesture and gaze input; no force loop yet.
- Unity XR Hands + Unreal OpenXR: standard interfaces; haptic plugins (Force Dimension Unity SDK, OpenHaptics, HaptX SDK) bridge to devices.
6. Edge cases & failure modes
| Hazard | Symptom | Mitigation |
|---|---|---|
| Delay > 100 ms | Pilot overshoots, sets up oscillation | Wave variables, TDPA, predictive display |
| Delay > 300 ms | Move-and-wait strategy; no continuous control | Supervisory autonomy + waypoint teleop |
| UDP packet loss | Spikes in wave variables | Forward-error correction, smoothing |
| TCP retransmit | Delay variance (jitter) | UDP + sequence numbers + TDPA |
| Single-fault tolerance (IEC 60601-1, medical) | Any single failure must not cause patient harm | Redundant encoders, brakes, watchdog |
| Stability ↔ transparency trade | High K_p → buzz; low K_p → mushy | TDPA, virtual fixtures, adaptive damping |
| Operator fatigue | Errors rise after 60–90 min | Gravity compensation, arm rest, motion scaling |
| Force scaling (1:10) | Tissue feel lost (haptic illusion below JND ≈ 10%) | Tactile augmentation (vibrotactile burst on touch) |
| Calibration drift | Master zero offset drifts in long session | Auto-zero before-engage, dual encoders |
| Workspace limit / virtual wall hit at master | Operator hits real device wall mid-task | Clutching (pedal disengages master to recentre) |
| Glove hygiene (surgical / VR shared) | Cross-contamination | Disposable inner liner, UV cabinet |
| Visual ↔ haptic asynchrony | ”Spongy” or “delayed” feel | Match latencies; cap visual to 50 ms, haptic 10 ms |
| Operator novice oscillation | High admittance gain triggers chatter | Adaptive gain — reduce gain when chatter detected |
7. Tools & software
| Tool | Purpose | License |
|---|---|---|
| CHAI 3D | C++ haptic rendering, supports most commercial devices | BSD-style |
| OpenHaptics (3D Systems) | Touch / Touch X / Phantom SDK | Proprietary, free academic |
| Force Dimension SDK | omega / sigma / delta devices | Proprietary, free with hardware |
| HaptX SDK | Glove G1 control | Proprietary |
| H3DAPI | Scene-graph haptics on top of CHAI 3D / OpenHaptics | GPL |
| ROS 2 + ros2_control | Slave-side admittance / impedance controllers | Apache 2.0 |
| ros2_teleop / leader_follower_controllers | Bilateral teleop reference impl | Apache 2.0 |
| Franka FCI + libfranka | Research arm with 1 kHz torque interface, common slave | Apache 2.0 |
| DLR Light-Weight Robot / KUKA iiwa research stack | High-fidelity research slave | Proprietary |
| MuJoCo / Isaac Sim | Simulated environments for haptic rendering | Apache 2.0 / proprietary |
| Unity XR Haptic + Unreal Haptic plugins | VR/AR integration | Engine licence |
| WebXR + Gamepad Haptic Actuator API | Browser-side vibrotactile | Web standard |
8. Case studies
8.1 Intuitive Surgical da Vinci (2000 – present)
Origin: SRI International “Green Telepresence Surgery System” (1990s, DARPA- funded for battlefield surgery). Commercialized by Intuitive Surgical; FDA-cleared 2000 (general laparoscopic). Architecture: position-position with audio + visual cues — no kinesthetic force feedback on the wristed instruments (cables + pulleys; force sensing through cables is noisy). EndoWrist 7-DOF: pitch, yaw, roll, grip + 3 master DOF mapped through a clutchable C-clamp. Motion scaling 1:1 to 5:1, 6 Hz tremor filter.
Result: > 12 M procedures by 2024, > 8 000 systems installed. Lessons: visual immersion + tool design > raw bilateral haptics for surgical acceptance; regulatory path easier without closed force loop.
8.2 NASA Robonaut 2 on ISS (2011 – 2018)
Humanoid upper body developed with GM. Flew to ISS Feb 2011 (STS-133). Telop architecture: ground-controlled with predictive virtual model (Niemeyer wave variables in research builds). Operator wears tracking glove + HMD with stereo cam from R2 head. Round-trip delay nominal 1.5–2 s through TDRSS Ku-band.
Strategy: not continuous bilateral — move-and-wait with a predictive visualization of where R2 would be after the operator’s intent arrives. Force feedback at operator’s fingertips (Phantom Premium-class device) only when slave reaches contact phase.
Result: demonstrated handrail grasping, pressing switches, soft-good manipulation. Returned to Earth 2018 for upgrades. Direct ancestor of Robonaut 5 / Valkyrie and Apptronik Apollo.
8.3 Apple Vision Pro (2024 – )
Released Feb 2024 (US), 2024-Jul international. Spatial computer running visionOS. Inputs: 12 cameras + LiDAR + IR illuminators tracking the hand at millimetre precision; eye tracking via internal IR cameras. Interaction model: gaze selects, pinch confirms, drag with maintained pinch.
Critically: no kinesthetic haptic feedback at hand. Tactile feedback is limited to the headset itself (face vibration absent in v1). Apple’s position matches da Vinci’s: rich visual + audio is sufficient for most consumer spatial tasks. Third-party teleop and surgical training applications layer HaptX or omega.7 over Vision Pro’s gestural input.
Significance: brings spatial UI to a mass-market device, expanding the operator-facing side of teleop infrastructure even before haptic gloves mature.
9. Cross-references
Internal links:
- impedance-control — slave-side compliant control fundamentals
- sensors-force-tactile — F/T sensors (ATI Mini40, Nano17), capacitive skin (Syntouch BioTac)
- end-effectors — gripper interfaces; surgical EndoWrist
- manipulator-design — series-elastic vs cable-driven; backdrivability
- safety-standards — IEC 60601-1 (medical), ISO 10218 / 15066 (industrial)
- comm-buses — EtherCAT 1 kHz cyclic, time-sensitive networking
- ros2-architecture — ros2_control 1 kHz hard real-time
- planned surgical-robotics (companion note, same batch)
- planned prosthetics (haptic feedback on bionic limbs)
- planned exoskeletons (full-arm haptic masters)
- classical-control — passivity, Lyapunov stability
- realtime-embedded — networked control over IP
10. Citations
Originator + year, peer-reviewed where applicable.
- Lawrence, D.A. “Stability and transparency in bilateral teleoperation.” IEEE Transactions on Robotics and Automation 9 (5), 1993, 624–637. — 4-channel architecture, transparency theorem.
- Hannaford, B. “A design framework for teleoperators with kinesthetic feedback.” IEEE TRA 5 (4), 1989, 426–434. — Hybrid matrix + passivity.
- Anderson, R.J. & Spong, M.W. “Bilateral control of teleoperators with time delay.” IEEE Trans. Automatic Control 34 (5), 1989, 494–501. — Scattering-theory delay compensation.
- Niemeyer, G. & Slotine, J.-J.E. “Stable adaptive teleoperation.” IEEE Journal of Oceanic Engineering 16 (1), 1991, 152–162. — Wave variables.
- Niemeyer, G. & Slotine, J.-J.E. “Telemanipulation with time delays.” International Journal of Robotics Research 23 (9), 2004, 873–890.
- Hannaford, B. & Ryu, J.-H. “Time-domain passivity control of haptic interfaces.” IEEE TRA 18 (1), 2002, 1–10. — TDPA.
- Colgate, J.E. & Hogan, N. “Robust control of dynamically interacting systems.” Int. J. Control 48 (1), 1988, 65–88; and Colgate & Schenkel, “Passivity of a class of sampled-data systems,” American Control Conf. 1994. — Z-width derivation.
- Sheridan, T.B. Telerobotics, Automation, and Human Supervisory Control. MIT Press, 1992. — canonical textbook.
- Salisbury, J.K., Brock, D., Massie, T., Swarup, N., Zilles, C. “Haptic rendering: programming touch interaction with virtual objects.” ACM Symposium on Interactive 3D Graphics 1995. — Phantom-era origin.
- Massie, T.H. & Salisbury, J.K. “The PHANToM haptic interface: a device for probing virtual objects.” ASME Dynamic Systems and Control Division 1994. — Phantom haptic device.
- Massimino, M.J. & Sheridan, T.B. “Teleoperator performance with varying force and visual feedback.” Human Factors 36 (1), 1994, 145–157.
- MacLachlan, R.A. & Riviere, C.N. “High-speed microscale optical tracking using digital frequency-domain multiplexing.” IEEE Trans. Instr. Meas. 58 (6), 2009, 1991–2001. — Microsurgical tracking.
- Hokayem, P.F. & Spong, M.W. “Bilateral teleoperation: an historical survey.” Automatica 42 (12), 2006, 2035–2057. — comprehensive review.
- Goertz, R.C. “Mechanical master-slave manipulator.” Nucleonics 1954. — first master-slave manipulator at Argonne 1949.
- Wagner, C.R., Stylopoulos, N., Jackson, P.G., Howe, R.D. “The benefit of force feedback in surgery: examination of blunt dissection.” Presence: Teleoperators and Virtual Environments 16 (3), 2007, 252–262.
- Lichiardopol, S. “A survey on teleoperation.” DCT report Eindhoven Univ. Tech. 2007. — practical engineering survey.
- Tachi, S., Komoriya, K., Sawada, K., Nishiyama, T., Itoko, T., Kobayashi, M., Inoue, K. “Telexistence cockpit for humanoid robot control.” Advanced Robotics 17 (3), 2003, 199–217. — telexistence framework.
- IEC 60601-1: General requirements for basic safety and essential performance of medical electrical equipment.
- ISO 10218-1/-2 (2011, rev. 2025) and ISO/TS 15066 (2016): Industrial robot safety and collaborative-robot speed-and-separation / power-and- force limits.
See also: _roadmap (Tier 2 control batch), _index (subdomain map). Last verified 2026-05-15 against manufacturer data sheets (3D Systems, Force Dimension, Intuitive Surgical, HaptX, Apple).