Sequential pulling policies to flatten and smooth fabrics have applications from surgery to manufacturing to home tasks such as bed making and folding clothes. Due to the complexity of fabric states and dynamics, we apply deep imitation learning to learn policies that, given color (RGB), depth (D), or combined color-depth (RGBD) images of a rectangular fabric sample, estimate pick points and pull vectors to spread the fabric to maximize coverage. To generate data, we develop a fabric simulator and an algorithmic supervisor that has access to complete state information. We train policies in simulation using domain randomization and dataset aggregation (DAgger) on three tiers of difficulty in the initial randomized configuration. We present results comparing five baseline policies to learned policies and report systematic comparisons of RGB vs D vs RGBD images as inputs. In simulation, learned policies achieve comparable or superior performance to analytic baselines. In 180 physical experiments with the da Vinci Research Kit (dVRK) surgical robot, RGBD policies trained in simulation attain coverage of 83% to 95% depending on difficulty tier, suggesting that effective fabric smoothing policies can be learned from an algorithmic supervisor and that depth sensing is a valuable addition to color alone. Supplementary material is available at https: //sites.google.com/view/fabric-smoothing.

In this letter, we present a method for resolving kinematic redundancy using a human motion database, with application to teleoperation of bimanual humanoid robots using low-cost devices. Handheld devices for virtual reality applications can realize low-cost interfaces for operating such robots but available information does not uniquely determine the arm configuration. The resulting arm motions may be unnatural and inconsistent due to the kinematic redundancy. The idea explored in this paper is to construct a human motion database in advance using an interface that can directly measure the whole arm configuration such as motion capture. During teleoperation, the database is used to infer the appropriate arm configuration, grasp forces, and object trajectory based on the end effector trajectories measured by low-cost devices. The database employs Bayesian Interaction Primitives that have been used for modeling human-robot interactions.

Robotic fabric manipulation has applications in home robotics, textiles, senior care and surgery. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We extend the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different fabric manipulation tasks with a single goal-conditioned policy. We introduce VisuoSpatial Foresight (VSF), which builds on prior work by learning visual dynamics on domain randomized RGB images and depth maps simultaneously and completely in simulation. We experimentally evaluate VSF on multi-step fabric smoothing and folding tasks against 5 baseline methods in simulation and on the da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train or test time. Furthermore, we find that leveraging depth significantly improves performance. RGBD data yields an 80% improvement in fabric folding success rate over pure RGB data. Code, data, videos, and supplementary material are available at https://sites.google.com/view/fabric-vsf/.

This paper presents a learning-from-demonstration (LfD) framework for teaching human-robot social interactions that involve whole-body haptic interaction, i.e. direct human-robot contact over the full robot body. The performance of existing LfD frameworks suffers in such interactions due to the high dimensionality and spatiotemporal sparsity of the demonstration data. We show that by leveraging this sparsity, we can reduce the data dimensionality without incurring a significant accuracy penalty, and introduce three strategies for doing so. By combining these techniques with an LfD framework for learning multimodal human-robot interactions, we can model the spatiotemporal relationship between the tactile and kinesthetic information during whole-body haptic interactions. Using a teleoperated bimanual robot equipped with 61 force sensors, we experimentally demonstrate that a model trained with 121 sample hugs from 4 participants generalizes well to unseen inputs and human partners.

In this paper, we present motion retargeting and control algorithms for teleoperated physical human-robot interaction (pHRI). We employ unilateral teleoperation in which a sensor-equipped operator interacts with a static object such as a mannequin to provide the motion and force references. The controller takes the references as well as current robot states and contact forces as input, and outputs the joint torques to track the operator's contact forces while preserving the expression and style of the motion. We develop a hierarchical optimization scheme combined with a motion retargeting algorithm that resolves the discrepancy between the contact states of the operator and robot due to different kinematic parameters and body shapes. We demonstrate the controller performance on a dual-arm robot with soft skin and contact force sensors using pre-recorded human demonstrations of hugging.