Robotics and
Representation Learning
Robotics and Representation Learning
I'm a Postdoctoral Researcher working with Danica Kragic, at the Robotics, Perception and Learning Lab (RPL), EECS, at KTH in Stockholm, Sweden. Where I also did my PhD under Danica Kragics supervission as well as co-supervision of Anastasiia Varava, and Hang Yin.
I'm working on learning representations for rigid and deformable objects manipulation. My work includes partial caging as well as working with highly deformable objects (clothing).
My CV can be found here (not neccesary up to date).
Representing clothing items for robotics tasks
Marco Moletta, Michael C. Welle, Alexander Kravchenko, Anastasia Varava and Danica Kragic
Abstract—
We study the problem of clothing item representation and build upon our recent work to perform a comparative study of the most commonly used representations of clothing items. We focus on visual and graph representations, both extracted from images. Visual representations follow the current trend of learning general representations rather than tailoring specific features relevant for the task. Our hypothesis is that graph representations may be more suitable for robotics tasks employing task and motion planning, while keeping the general properties and accuracy of visual representations. We rely on a subset of DeepFashion2 dataset and study performance of developed representations in an unsupervised, contrastive learning framework using a downstream classification task. We demonstrate the performance of graph representations in folding and flattening of different clothing items in a real robotic setup with a Baxter robot.
Elastic Context: Encoding Elasticity in Data-driven Models of Textiles
for Robotic Manipulation
Alberta Longhini, Marco Moletta, Alfredo Reichlin, Michael C. Welle, Alexander Kravchenko, Yufei Wang, David Held, Zackory Erickson, and Danica Kragic
Abstract—
Learning dynamic properties of elastic deformable objects such as textiles is relevant for applications such as assistive dressing, textile recycling, and automation of household tasks. Ongoing research in robotic manipulation has begun to explore interactions with textiles, where the textile properties are held constant or known a priori. We present an Elastic Context (EC) for data-driven models of textiles to leverage elastic-related information during robotic manipulation tasks. The definition of EC relies on force/displacement curves commonly used in textile engineering, which we reformulated for robotic applications. We evaluate EC using Graph Neural-Networks (GNN) to learn generalized elastic behaviours of textiles. In relation to the employed models, we discuss limitations of current robotic simulators in terms of learning data-driven elastic models for manipulation tasks.
We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a Latent Space Roadmap (LSR) for task planning which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of three parts: (1) a Mapping Module (MM) that maps observations given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states, (2) the LSR which builds and connects clusters containing similar states in order to find the latent plans between start and goal states extracted by MM, and (3) the Action Proposal Module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.
Martina Lippi*, Petra Poklukar*, Michael C. Welle*, Anastasia Varava, Hang Yin, Alessandro Marino, and Danica Kragic
Accepted in Transactions on Robotics (TRO), 2022
Caging grasps limit the mobility of an ob- ject to a bounded component of configuration space. We introduce a notion of partial cage quality based on maximal clearance of an escaping path. As comput- ing this is a computationally demanding task even in a two-dimensional scenario, we propose a deep learn- ing approach. We design two convolutional neural net- works and construct a pipeline for real-time planar par- tial cage quality estimation directly from 2D images of object models and planar caging tools. One neural network, CageMaskNN, is used to identify caging tool locations that can support partial cages, while a sec- ond network that we call CageClearanceNN is trained to predict the quality of those configurations. A partial caging dataset of 3811 images of objects and more than 19 million caging tool configurations is used to train and evaluate these networks on previously unseen ob- jects and caging tool configurations. Experiments show that evaluation of a given configuration on a GeForce GTX 1080 GPU takes less than 6 ms. Furthermore, an additional dataset focused on grasp-relevant configura- tions is curated and consists of 772 objects with 3.7 million configurations. We also use this dataset for 2D Cage acquisition on novel objects. We study how net- work performance depends on the datasets, as well as how to efficiently deal with unevenly distributed training data. In further analysis, we show that the evalua- tion pipeline can approximately identify connected re- gions of successful caging tool placements and we eval- uate the continuity of the cage quality score evaluation along caging tool trajectories. Influence of disturbances is investigated and quantitative results are provided.
Michael Welle, Anastasiia Varava, Jeffrey Mahler, Ken Goldberg, Danica Kragic, and Florian T. Pokorny
Published in Autonomous Robots, Special Issue Topological Methods in Robotics 2021
Cloth manipulation is a challenging task that, despite its importance, has received relatively little attention compared to rigid object manipulation. In this letter, we provide three benchmarks for evaluation and comparison of different approaches towards three basic tasks in cloth manipulation: spreading a tablecloth over a table, folding a towel, and dressing. The tasks can be executed on any bimanual robotic platform and the objects involved in the tasks are standardized and easy to acquire. We provide several complexity levels for each task, and describe the quality measures to evaluate task execution. Furthermore, we provide baseline solutions for all the tasks and evaluate them according to the proposed metrics.
Irene Garcia-Camacho*, Martina Lippi*, Michael C. Welle, Hang Yin, Rika Antonova, Anastasiia Varava, Júlia Borràs, Carme Torras, Alessandro Marino, Guillem Alenyà, Danica Kragic
Puplished in IEEE Robotics and Automation Letters 5.2 (2020)
Planning and executing object manipulation requires integrating multiple sensory and motor channels while acting under uncertainty and complying with task constraints. As the modern environment is tuned for human hands, designing robotic systems with similar manipulative capabilities is crucial. Research on robotic object manipulation is divided into smaller communities interested in, e.g., motion planning, grasp planning, sensorimotor learning, and tool use. However, few attempts have been made to combine these areas into holistic systems. In this review, we aim to unify the underlying mechanics of grasping and in-hand manipulation by focusing on the temporal aspects of manipulation, including visual perception, grasp planning and execution, and goal-directed manipulation. Inspired by human manipulation, we envision that an emphasis on the temporal integration of these processes opens the way for human-like object use by robots.
Judith Butepage, Silvia Cruciani, Mia Kokic, Michael Welle, and Danica Kragic
Published in Annual Review of Control, Robotics, and Autonomous Systems (2019)
Visual action planning particularly excels in applications where the state of the system cannot be computed explicitly, such as manipulation of deformable objects, as it enables planning directly from raw images. Even though the field has been significantly accelerated by deep learning techniques, a crucial requirement for their success is the availability of a large amount of data. In this work, we propose the Augment-Connect-Explore (ACE) paradigm to enable visual action planning in cases of data scarcity. We build upon the Latent Space Roadmap (LSR) framework which performs planning with a graph built in a low dimensional latent space. In particular, ACE is used to i) Augment the available training dataset by autonomously creating new pairs of datapoints, ii) create new unobserved Connections among representations of states in the latent graph, and iii) Explore new regions of the latent space in a targeted manner. We validate the proposed approach on both simulated box stacking and real-world folding task showing the applicability for rigid and deformable object manipulation tasks, respectively.
Martina Lippi*, Michael C. Welle*, Petra Poklukar, Alessandro Marino and Danica Kragic
Accepted in International Conference on Intelligent Robots and Systems (IROS2022)
Embedding an optimization process has been explored for imposing efficient and flexible policy structures. Existing work often build upon nonlinear optimization with explicitly iteration steps, making policy inference prohibitively expensive for online learning and real-time control. Our approach embeds a linear-quadratic-regulator (LQR) formulation with a Koopman representation, thus exhibiting the tractability from a closed-form solution and richness from a non-convex neural network. We use a few auxiliary objectives and reparameterization to enforce optimality conditions of the policy that can be easily integrated to standard gradient-based learning. Our approach is shown to be effective for learning policies rendering an optimality structure and efficient reinforcement learning, including simulated pendulum control, 2D and 3D walking, and manipulation for both rigid and deformable objects. We also demonstrate real world application in a robot pivoting task.
Hang Yin, Michael C. Welle and Danica Kragic
Accepted in International Conference on Intelligent Robots and Systems (IROS2022)
Learning state representations enables robotic planning directly from raw observations such as images. Most methods learn state representations by utilizing losses based on the reconstruction of the raw observations from a lower-dimensional latent space. The similarity between observations in the space of images is often assumed and used as a proxy for estimating similarity between the underlying states of the system. However, observations commonly contain task-irrelevant factors of variation which are nonetheless important for reconstruction, such as varying lighting and different camera viewpoints. In this work, we define relevant evaluation metrics and perform a thorough study of different loss functions for state representation learning. We show that models exploiting task priors, such as Siamese networks with a simple contrastive loss, outperform
Constantinos Chamzas*, Martina Lippi*, Michael C. Welle*, Anastasia Varava, Lydia E. Kavraki, and Danica Kragic
Accepted in International Conference on Intelligent Robots and Systems (IROS2022)
Identification of textile properties is an important milestone toward advanced robotic manipulation tasks that consider interaction with clothing items such as assisted dressing, laundry folding, automated sewing, textile recycling and reusing. Despite the abundance of work considering this class of deformable objects, many open problems remain. These relate to the choice and modelling of the sensory feedback as well as the control and planning of the interaction and manipulation strategies. Most importantly, there is no structured approach for studying and assessing different approaches that may bridge the gap between the robotics community and textile production industry. To this end, we outline a textile taxonomy considering fiber types and production methods, commonly used in textile industry. We devise datasets according to the taxonomy, and study how robotic actions, such as pulling and twisting of the textile samples, can be used for the classification. We also provide important insights from the perspective of visualization and interpretability of the gathered data.
Visual planning approaches have shown great success for decision making tasks with no explicit model of the state space. Learning a suitable representation and constructing a latent space where planning can be performed allows non-experts to setup and plan motions by just providing images. However, learned latent spaces are usually not semantically-interpretable, and thus it is difficult to integrate task constraints. We propose a novel framework to determine whether plans satisfy constraints given demonstrations of policies that satisfy or violate the constraints. The demonstrations are realizations of Linear Temporal Logic formulas which are employed to train Long Short-Term Memory (LSTM) networks directly in the latent space representation. We demonstrate that our architecture enables designers to easily specify, compose and integrate task constraints and achieves high performance in terms of accuracy. Furthermore, this visual planning framework enables human interaction, coping the environment changes that a human worker may involve. We show the flexibility of the method on a box pushing task in a simulated warehouse setting with different task constraints.
We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable objects. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the effectiveness of the method on a simulated box stacking task as well as a T-shirt folding task performed with a real robot.
Martina Lippi*, Petra Poklukar*, Michael C. Welle*, Anastasiia Varava, Hang Yin, Alessandro Marino, and Danica Kragic
Puplished in International Conference on Intelligent Robots and Systems (IROS2020)
Research on automatic identification of clothing categories and fashion landmarks based on images has recently gained significant interest in the research community due to its potential impact on various areas such as robotic clothing ma- nipulation, automated clothes sorting and recycling, and online shopping, to name a few. Several public large annotated fashion datasets have been created recently to facilitate research advances in this direction. In this work, we make the first step towards leveraging the data and techniques developed for fashion image analysis in vision-based robotic clothing manipulation tasks. We focus on techniques that can generalize from large-scale fashion datasets to less structured, small datasets collected in a robotic lab. Specifically, we propose training data augmentation methods such as elastic warping, and model adjustments such as rotation invariant convolutions to make the model generalize better. Our experiments demonstrate that our approach outperforms state- of-the art models with respect to clothing category classification and fashion landmark detection when tested on previously unseen datasets. Furthermore, we present experimental results on a new small dataset composed of images on which a robot holds different garments, collected in our lab.
Caging grasps limit the mobility of an object to a bounded component of configuration space. We introduce a notion of partial cage quality based on maximal clearance of an escaping path. As this is a computationally demanding task even in a two-dimensional scenario, we propose a deep learning approach. We design two convolutional neural networks and construct a pipeline for real-time partial cage quality estimation directly from 2D images of object models and planar caging tools. One neural network, CageMaskNN, is used to identify caging tool locations that can support partial cages, while a second network that we call CageClearanceNN is trained to predict the quality of those configurations. A dataset of 3811 images of objects and more than 19 million caging tool configurations is used to train and evaluate these networks on previously unseen objects and caging tool configurations. Furthermore, the networks are trained jointly on configurations for both 3 and 4 caging tool configurations whose shape varies along a 1-parameter family of increasing elongation. In experiments, we study how the networks’ performance depends on the size of the training dataset, as well as how to efficiently deal with unevenly distributed training data. In further analysis, we show that the evaluation pipeline can approximately identify connected regions of successful caging tool placements and we evaluate the continuity of the cage quality score evaluation along caging tool trajectories. Experiments show that evaluation of a given configuration on a GeForce GTX 1080 GPU takes less than 6 ms.
In this paper we present an end to end object modeling pipeline for an unmanned aerial vehicle (UAV). We contribute a UAV system which is able to autonomously plan a path, navigate, acquire views of an object in the environment from which a model is built. The UAV does collision checking of the path and navigates only to those areas deemed safe. The data acquired is sent to a registration system which segments out the object of interest and fuses the data. We also show a qualitative comparison of our results with previous work.
Michael Welle, Ludvig Ericson, Rares Ambrus, Patric Jensfelt
Published in European Conference on Mobile Robots (ECMR2017)
Deformable objects manipulation is a key component of a variety of everyday and specialized applications, ranging from domestic housework such as cloth folding or food handing, to medical scenarios such as surgery and suturing, up to industrial setups such as cable insertion. However, the large configuration space of deformable objects causes traditional modelling, planning and control approaches to fail when dealing with them. More specifically, unlike in the case of rigid objects, two main challenges arise: i) there is no clear and unified state representation and ii) the dynamics is complex and highly non-linear. This leads to the absence of current unified solutions and to highly domain-specific approaches emerging in the fields of perception, simulation, control and mechanics. In addition, the lack of scalable simulation environments limits possible developments in the above fields. We believe that robust progress can only be achieved by combining these complementary areas of robotics. Therefore, this workshop aims to start a discussion about the current state-of-the-art and possible research directions as well as connect people from different sub-fields to solve deformable objects challenges.
Martina Lippi*, Michael C. Welle*, Anastasiia Varava*, Hang Yin, Rika Antonova, Florian T. Pokorny, Danica Kragic, Yiannis Karayiannidis,
Ville Kyrki, Alessandro Marino, Julia Borras, Guillem Alenya, Carme Torras
Workshop held at ICRA2021
Clothes, food, cables, and body tissue are just a few examples of deformable objects involved in both everyday and specialized tasks. Although humans are able to reliably manipulate them, automating this process using robotic platforms remains challenging. Indeed, the high number of degrees of freedom involved undermines the effectiveness of traditional modelling, planning and control methods developed for rigid object manipulation. This paves the way for exciting questions from a research and application perspective. i) How to tractably represent the state of a deformable object? ii) How to model and simulate its highly complex and non-linear dynamics? iii) What hardware tools and platforms are best suited for grasping and manipulating? We aim to discuss these and more challenges that arise from handling deformable objects by connecting scientists from different subfields of robotics, including perception, simulation, control, and mechanics.
Martina Lippi*, Daniel Seita*, Michael C. Welle*, Hang Yin, Danica Kragic, David Held, Yiannis Karayiannidis
Workshop held at ICRA2022
The state-of-the-art unsupervised contrastive visual representation learning methods that have emerged recently (SimCLR, MoCo, SwAV) all make use of data augmentations in order to construct a pretext task of instant discrimination consisting of similar and dissimilar pairs of images. Similar pairs are constructed by randomly extracting patches from the same image and applying several other transformations such as color jittering or blurring, while transformed patches from different image instances in a given batch are regarded as dissimilar pairs. We argue that this approach can result similar pairs that are \textit{semantically} dissimilar. In this work, we address this problem by introducing a \textit{batch curation} scheme that selects batches during the training process that are more inline with the underlying contrastive objective. We provide insights into what constitutes beneficial similar and dissimilar pairs as well as validate \textit{batch curation} on CIFAR10 by integrating it in the SimCLR model.
Michael C. Welle*, Petra Poklukar*, and Danica Kragic
Workshop on Self-Supervised Learning for Reasoning and Perception, Workshop at International Conference on Machine Learning 2021
Representation learning allows planning actions directly from raw observations. Variational Autoencoders (VAEs) and their modifications are often used to learn latent state representations from high-dimensional observations such as images of the scene. This approach uses the similarity between observations in the space of images as a proxy for estimating similarity between the underlying states of the system. We argue that, despite some successful implementations, this approach is not applicable in the general case where observations contain task-irrelevant factors of variation. We compare different methods to learn latent representations for a box stacking task and show that models with weak supervision such as Siamese networks with a simple contrastive loss produce more useful representations than traditionally used autoencoders for the final downstream manipulation task.
Constantinos Chamzas*, Martina Lippi*, Michael C. Welle*, Anastasiia Varava, Lydia Kavraki, and Alessandro Marino, and Danica Kragic
NeurIPS 2020 Workshop on Robot Learning
We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable objects. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the effectiveness of the method on a simulated box stacking task as well as a T-shirt folding task performed with a real robot.
Martina Lippi*, Petra Poklukar*, Michael C. Welle*, Anastasiia Varava, Hang Yin, Alessandro Marino, and Danica Kragic
RSS 2020 Workshop - Visual Learning and Reasoning for Robotic Manipulation
Learning efficient and compact representations is crucial for various problems in artificial intelligence. Designing evaluation procedures that are independent from the task at hand is one of the key challenges of this area of research. Inspired by the theory of causal reasoning, we consider interventions for analysing data representation. Specifically, we leverage them to introduce Interventional Score for measuring disentanglement of the latent representations.
Petra Poklukar*, Michael C. Welle*, Anastasiia Varava and Danica Kragic
32nd annual workshop of the Swedish Artificial Intelligence Society (SAIS)
Summer Internship at HKUST, HongKong, Supervisors: Michael Wang, Hang Kaiyu
27.04.2022 Talk information
(As the official defense recording had technical difficulties it is a test recording of my thesis presentation)
Thesis title: Learning structured Representations for ridgid and deformable Object Manipulation
Co-supervising Alberta Longhini
Co-supervising Marco Moletta
Co-supervising Peiyang "Yonk" Shi
Tommy Walling; Structural Comparison of Data Representations Obtained from Deep Learning Models
David Norrman; Impact of Semantic Segmentation on OOD Detection Performance for VAEs and Normalizing Flow Models
Samuel Norling; Probabilistic Forecasting through Reformer Conditioned Normalizing Flows
Simon Westberg; Investigating the Learning Behavior of Generative Adversarial Networks
Joakim Dahl; Analysis of the effect of Latent Dimensions on Disentagement in Variational Autoencoders
Alberta Longhini; Fabric Material Classification by Combining Force and Vision;
Nik Vaessen; Training Multi-Task Deep Neural Networks with Disjoint Datasets
Georgios Deligiorgis; Context-Aware Graph Convolutional Network with Multi-Clusters Mini-Batch for Link Prediction
Ching-An Wu; Investigation of Different Observation and Action Spaces for Reinforcement Learning on Reaching Tasks
Fall 2021: TA in Introduction to Robotics (Msc)
Fall 2020: TA in Introduction to Robotics (Msc)
Fall 2020: Teacher in Project Course in Data Science (Msc)
Fall 2019: TA in Introduction to Robotics (Msc)
Fall 2019: Teacher in Project Course in Data Science (Msc)
Fall 2018: TA in Introduction to Robotics (Msc)
Fall 2018: TA in Artificial Intelligence (Msc)
Fall 2018: Teacher in Project Course in Data Science (Msc)
Fall 2017: TA in Artificial Intelligence (Msc)
Fall 2016: TA in Artificial Intelligence (Msc)