Interactive Robot Perception & Learning

Scope: Master Thesis
Start Date: ASAP
Advisors: Max Siebenborn, Georgia Chalvatzaki
Detailed Proposal: SE(3)-Equivariant Manipulation Thesis Call

^{Robot grasping differently rotated objects by jointly rotating end-effector. Task from [4]}

Equivariant policies [1,2,3] are increasingly used in robot manipulation because they respect geometric structure. Equivariance means that if the input (observations) is transformed—like rotated or translated—the output (robot actions) changes in the same predictable way. However, in real robots, factors like joint limits, collisions, end-effector design, and task goals can break this symmetry. For example, in certain scenarios, rotating an object may require a different grasp rather than just rotating the hand. In this thesis, you will explore when SE(3)-equivariance is useful—and when it is not—by pushing the limits of equivariant policies!

Research Questions

• When is SE(3)-equivariance a valid prior for manipulation?
• What assumptions or constraints cause symmetry to break?
• How can we use equivariance as a prior when full equivariance is broken, e.g., via soft priors?

What You Will Do

• Study equivariant imitation learning and reproduce existing results.
• Implement tasks that stress manipulability, collisions, and kinematic limits. Analyze symmetry-breaking conditions.
• Optionally, test on a real Tiago++ robot or develop architectures for symmetry-breaking cases.

Requirements

• Background in robotics and robot learning
• Strong Python and PyTorch skills
• Bonus: robot simulation experience (MuJoCo), imitation learning
• Bonus: knowledge in equivariant learning / group theory / differential geometry

Opportunities
Equivariance is a powerful prior that can improve robot manipulation, yet is rarely applied in practice. Your thesis could help bridge this gap. You will be given the opportunity to work with real robots and publish your results.

Interested?
Send a short (informal) motivation, CV, and grade sheets to: max.siebenborn@tu-darmstadt.de

References

[1] Wang, Dian, et al. “Equivariant diffusion policy.” 8th Annual Conference on Robot Learning, 2024.
[2] Tie, Chenrui, et al. “Et-seed: Efficient trajectory-level se (3) equivariant diffusion policy.” arXiv preprint arXiv:2411.03990 (2024).
[3] Yang, Jingyun, et al. “Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning.” 8th Annual Conference on Robot Learning, 2024.
[4] Mandlekar, Ajay, et al. “What matters in learning from offline human demonstrations for robot manipulation.” arXiv preprint arXiv:2108.03298 (2021).

Scope: Master Thesis
Advisors: Kumar Manas

The Action Chunking Transformer (ACT) architecture, proven successful in the ALOHA bimanual manipulation system, represents a breakthrough in learning complex coordinated movements. This thesis extends ACT to operate at the primitive level rather than raw actions, creating a hierarchical framework that combines the expressiveness of modern imitation learning with the interpretability of skill-based robotics.

This thesis targets real-world scenarios such as: coordinating dual robot arms to
restock supermarket shelves and handle airport baggage efficiently and safely.

Research Goal & Expected Outcomes

Primary Objective: Adapt the ACT architecture to predict chunks of manipulation primitives instead of raw joint actions, enabling temporal abstraction while preserving bimanual coordination capabilities. The modified ACT will use a CVAE (Conditional VAE) latent space to capture multimodal coordination strategies: when both arms should move together vs. sequentially, which arm leads in asymmetric tasks, and how to recover from perturbations.

Idea: Primitive chunking naturally provides temporal abstraction, where each chunk represents a coherent manipulation phase (e.g., approach-grasp-lift sequence) rather than arbitrary action windows.

Requirements:

Essential: Python/PyTorch skills, familiarity with VAE/CVAE architectures, Transformer implementation experience, understanding of imitation learning.

Preferred / willingness: knowledge of behavior cloning methods, real robot experiments, ROS/ROS2

We Provide: Real bimanual robot system, demonstration dataset of operations, computational resources, mentorship, project collaboration and opportunity to publish in top tier conferences.

Further Details are available here.

Apply:
Write an email to kumar.manas[at]tu-darmstadt.de. Please shortly state your motivation and experience (courses, projects) and if available, transcript of records.

Scope: Master Thesis
Advisors: Kumar Manas

Bimanual robot manipulation represents one of the most challenging frontiers in robotics, enabling robots to perform complex tasks that require coordinated dual-arm movements. This thesis focuses on developing intelligent bimanual robots for real-world applications including supermarket shelf restocking and airport baggage handling.

key research question: Given a library of manipulation primitives (e.g. reach, grasp, lift, place, push), how can a robot learn WHEN to execute WHICH primitive with WHICH arm, and HOW to coordinate both arms from human demonstrations?

Research Goal & Expected Outcomes:

Primary Objective: Develop a Transformer-based sequence model that learns to predict and coordinate bimanual manipulation primitive sequences from human demonstration data collected in real supermarket environments. The framework will take current robot state and task context as input, and output: (1) next primitive ID for each arm, (2) primitive parameters (target poses, forces), (3) execution duration, and (4) inter-arm synchronization signals.

Key Ideas: Cross-attention mechanism between left and right arm token streams to implicitly learn coordination patterns without explicit programming.

Requirements:

Essential: Python programming, PyTorch experience, understanding of Transformer architectures, basics of robot kinematics

Preferred / willingness: Experience with imitation learning, ROS/ROS2, robot experiments, sequence modeling (NLP or time-series).

We Provide: Real bimanual robot system, demonstration dataset of operations, computational resources, mentorship, project collaboration and opportunity to publish in top tier conferences.

Further Details are available here.

Apply:
Write an email to kumar.manas[at]tu-darmstadt.de. Please shortly state your motivation and experience (courses, projects) and if available, transcript of records.

Scope: Master Thesis
Advisors: Georgia Chalvatzaki, Xanthi Papageorgiou (external)

Keywords: Robotic Manipulation, Foundation Models, LLM, VLM, Active Perception, Physical Interaction, Uncertainty Resolution, Human-Robot Interaction

Topic: Recent advances in Vision and Language Models (VLMs) have enabled robots to understand high-level commands, but they often fail when faced with visual ambiguity. For example, is a bolt loose, or is that just a shadow? This project aims to move beyond simple “observe-then-act” pipelines by creating a robot that can actively investigate its environment to resolve such uncertainties.

The core idea is to develop a framework where the robot, when uncertain, can autonomously decide to take another look from a different angle or even perform a gentle physical “touch test” with its gripper. By fusing visual data with physical feedback, the system can build a more robust and reliable understanding of the world, making a more informed final decision. This project will be implemented on a Franka Emika robotic arm and will focus on a solar panel inspection task, directly contributing to the goals of the ARISE EU project.

Tasks: Your work will include the following tasks:
– Conduct a literature review on the latest developments in foundation models for robotic control and perception.
– Design and implement the proposed Investigative Robotic Agent Framework using Python and ROS.
– Develop a novel module to translate raw physical sensor data (e.g., force/torque) into concise natural language descriptions that an LLM can understand.
– Set up and calibrate the real-world hardware, including the Franka arm, a wrist-mounted camera, and the inspection mock-up.
– Rigorously evaluate the system’s performance against baseline model(s) that does not perform active investigation.
– Document your work and write your final Master’s thesis, and participate in a paper submission to a top-tier robotics conference.

Requirements:
– Strong programming skills in Python.
– Practical experience with robotics; familiarity with ROS is a significant plus.
– A good understanding of fundamental deep learning concepts; experience with frameworks like PyTorch is beneficial.
– A keen interest in working with real-world robotic hardware.
– Previous experience with LLM/VLM APIs is a bonus but not strictly required.

References:
– Ahn, M., Brohan, A., Brown, N., et al. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv preprint arXiv:2204.01691.
– Brohan, A., Brown, N., et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15818.
– Huang, W., et al. (2023). VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. arXiv preprint arXiv:2307.05973.

Contact: Interested students are invited to apply by emailing (georgia.chalvatzaki@tu-darmstadt.de) with their motivation letter, CV, and transcripts (if available).

Scope: Master Thesis

Start Date: ASAP / upon agreement

Advisors: Nils Dengler

Keywords: Robotic Manipulation, Active Perception, Contact-Rich Manipulation, Robot Learning, Generative Models, Physical Reasoning, Uncertainty Estimation, Pick-and-Place

Topic:

Robotic pick-and-place in realistic environments remains challenging when objects are only partially observed, target regions are cluttered, and physical properties such as center of mass, friction, or contact stability are unknown. While recent methods can predict stable placements from point clouds, most systems still follow an observe-then-act pipeline: the robot reconstructs the scene, selects a grasp and placement pose, and executes the action open-loop. However, in many real-world situations, the robot should not immediately commit to a placement. Instead, it should actively decide whether it needs another viewpoint, a gentle physical probe, or a closed-loop contact adjustment before releasing the object.

This thesis aims to develop an interactive placeability reasoning framework for object model-free robotic pick-and-place. The system will build on an existing placeability metric [1] that evaluates grasp-placement pairs from partial point clouds by considering object stability, grasp feasibility. The goal of this thesis is to extend this pipeline from passive open-loop placement to an interactive framework where the robot can actively reduce uncertainty in object placements through visual and physical investigation.

The core idea is that the robot maintains a belief over placement success. Given a partial object reconstruction and a target scene, the robot should reason about whether the current information is sufficient for a reliable placement. If not, it may choose to observe the object from another viewpoint, perform a gentle probing action to infer physical properties, or adapt the placement during contact using force or tactile feedback. Furthermore, a generative model such as a diffusion model can be used to propose diverse 6D placement candidates, which are then verified using the physical placeability metric.

Research Questions:

– Can active perception be guided by expected improvement in placement success rather than reconstruction quality?

– How can physical interaction, such as gentle probing or contact feedback during placement, improve estimates of center of mass, friction, and contact stability?

Your Tasks:
The thesis will begin with a literature review on active perception, contact-rich manipulation, physical-property estimation, and generative models for robotic placement. Based on this, the student will develop an active perception strategy that selects next-best views according to uncertainty in the final placement decision. In addition, a simple physical investigation module, such as gentle probing, lifting, or contact-based settling, will be implemented to refine estimates of stability-relevant object properties.

The student will further develop a closed-loop placement strategy that uses contact feedback to adapt the final release pose. A generative model, such as a diffusion or flow-based model, will be trained to propose diverse 6D placement candidates conditioned on object and scene point clouds. The resulting system will be evaluated in simulation and on a real robotic setup, and compared against fixed-view open-loop placement, reconstruction-based next-best-view, sampling-based placement generation, and placement without contact feedback.

Requirements:
The student should have strong Python skills and a background in robotics, robot perception, or robot learning. Experience with point clouds, 3D geometry, or manipulation planning is important, as the thesis involves reasoning about stable object placement from partial observations.

Experience with ROS/ROS 2, PyTorch, active perception, uncertainty estimation, force/torque sensing, contact-rich manipulation, or simulators such as Isaac Sim, MuJoCo, or PyBullet is beneficial.

References:

[1] Wingender, Dengler, Menon, Pan, Bennewitz. “A Robust Placeability Metric for Model-Free Unified Pick-and-Place Reasoning.”

[2] Shanthi and Hermans. “Pick and Place Planning is Better than Pick Planning then Place Planning.” IEEE Robotics and Automation Letters, 2024.

[3] Noh et al. “Learning To Place Unseen Objects Stably Using A Large-Scale Simulation.” IEEE Robotics and Automation Letters, 2024.

[4] Zhao et al. “AnyPlace: Learning Generalized Object Placement for Robot Manipulation.” arXiv preprint, 2025.

[5] Nadeau et al. “Generating Stable Placements via Physics-guided Diffusion Models.” arXiv preprint, 2025.

Interested?:
Students should send (nils.dengler@tu-darmstadt.de) a short informal motivation, CV, transcript if available, and a brief description of relevant experience in robotics, machine learning, simulation, or real robot systems.

Scope: Master Thesis
Start Date: ASAP / upon agreement
Advisors: Nils Dengler
Keywords: Robotic Manipulation, Reinforcement Learning, Contact-Rich Manipulation, Visual-Tactile Feedback, Object Pushing, Physical Reasoning, Stability Estimation, Sim-to-Real Transfer

Topic:

Robotic pushing is a powerful manipulation skill for moving objects in cluttered environments, especially when grasping is difficult, unreliable, or not immediately feasible. However, pushing is challenging when objects are tall, light, partially filled, or have unknown mass distributions. In such cases, an inappropriate push point, pushing direction, or applied force can easily cause the object to tilt, topple, or disturb the surrounding scene.

This thesis investigates how reinforcement learning can be used to develop adaptive 3D pushing strategies under toppling risk. The goal is to learn a policy that selects suitable push points, push directions, and pushing forces based on visual and tactile feedback. Instead of using a fixed pushing strategy, the robot should adapt its behavior to the observed object geometry, contact response, and stability-relevant physical cues.

The core idea is that a robot should learn when to push low and gently, when a stronger push is safe, and how to react if the object starts tilting or slipping. This requires reasoning about object stability, contact dynamics, and uncertainty in physical properties such as center of mass, friction, and mass distribution. The resulting system should enable robust object pushing in realistic tabletop or shelf-like environments while avoiding toppling, excessive forces, and unnecessary scene disturbance.

Research Questions:

– How can a robot learn stability-aware pushing policies that adapt the push point, direction, and force to different object geometries and physical properties?
– How can visual and tactile feedback be combined to estimate whether an object is at risk of toppling during pushing?

Your Tasks:

The thesis will begin with a literature review on robotic pushing, contact-rich manipulation, reinforcement learning for manipulation, visual-tactile feedback, and stability-aware object interaction. Based on this, the student will develop a simulation setup for 3D object pushing with objects of different shapes, heights, friction values, masses, and center-of-mass distributions. The goal is to create scenarios in which pushing at unsuitable contact points or with excessive force can lead to tilting or toppling.

The student will then design and train a reinforcement learning policy that selects adaptive push points, push directions, pushing heights, and pushing forces based on visual and tactile feedback. The policy should learn to move objects efficiently while avoiding toppling, excessive contact forces, and unnecessary disturbance of the surrounding scene. The resulting system will be evaluated in simulation and, if feasible, on a real robotic setup. It will be compared against fixed pushing strategies, heuristic low-contact pushing, stability-aware hand-crafted baselines, and policies without tactile feedback.

Requirements:
The student should have strong Python skills and a background in robotics, robot learning, or reinforcement learning. Experience with simulation-based robot learning is important, as the thesis involves training and evaluating policies for contact-rich object pushing under varying physical conditions.

Experience with PyTorch, reinforcement learning frameworks, ROS/ROS 2, RGB-D perception, force/torque sensing, tactile sensing, or simulators such as Isaac Sim, MuJoCo, or PyBullet is beneficial. Interest in real-world robot experiments, physical reasoning, and sim-to-real transfer is highly welcome.

References:

[1] Aoyama et al. “Poke and Strike: Learning Task-Informed Exploration Policies”, CORL 2025

[2] Ferrandis et al. “Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions”, CORL 2024

[3] Dengler et al. “Learning Goal-Directed Object Pushing in Cluttered Scenes with Location-Based Attention”, IROS, 2025

[4] Xu et al. “COCOI: Contact-aware Online Context Inference for Generalizable Non-planar Pushing”, IROS 2021

[5] Bui et al., “Push Anything: Single- and Multi-Object Pushing From First Sight with Contact-Implicit MPC”, ICRA, 2026

Contact:

Interested students should send (nils.dengler@tu-darmstadt.de) a short informal motivation, CV, transcript if available, and a brief description of relevant experience in robotics, machine learning, simulation, or real robot systems.