Interactive Robot Perception & Learning

Scope: Master Thesis
Start Date: ASAP
Advisors: Max Siebenborn

Reflection-equivariant manipulation: mirroring the trajectory across the robot’s body solves the mirrored box-pickup task. Since asymmetric robots or perception can make hard symmetry constraints harmful [7], we study soft symmetry priors that still allows symmetry breaking in robot manipulation.

Equivariant policies [1,2,3] are increasingly used in robot manipulation to impose geometric structure, enabling better sample efficiency and spatial generalization. Equivariance means that if the input (observations) to a model is transformed—e.g., rotated [1,2] or reflected [3]—the output (robot actions) changes in the same predictable way. However, in real robots, factors like joint limits, manufacturing errors, perception, and task goals can break this symmetry. For example, rotating an object may require a different grasp rather than just rotating the hand; or an asymmetry in the robot’s mass distribution make it not perfectly reflection-symmetric. In this thesis, we will explore whether therefore imposing equivariance as a soft inductive bias can be beneficial over hard constraint enforcement.

Research Questions

• What assumptions or constraints cause morphological or SE(3) symmetry to break in robot manipulation, and can a policy detect these factors during training or online?
• How can we use equivariance as a inductive bias when full equivariance is broken, through soft priors?
• Are there empirical benefits of soft equivariance over hard equivariance enforcement?

What You Will Do

• Study equivariant imitation learning and investigate symmetry breaking factors, including how relevant they are in existing benchmarks and robot tasks [1,7].
• Develop a mechanism for soft equivariance enforcement, inspired by [4,5,6], for robotic tasks and analyze how the learned representations detect and handle asymmetry.
• Evaluate the method under different symmetry breaking levels and conditions, analyzing whether soft priors retain structure but allow breaking it where necessary.

Requirements

• Background in robotics and robot learning
• Strong Python and PyTorch skills
• Bonus: robot simulation experience (MuJoCo), imitation learning
• Bonus: knowledge in equivariant learning, group theory, or differential geometry

Opportunities
Equivariance is a powerful prior that can improve robot manipulation, yet it remains unclear how to impose it at scale where perfect equivariance is usually broken. Your thesis will help bridge this gap. You will be given the opportunity to work with real robots and publish your results.

Interested?
Send a short (informal) motivation, CV, and grade sheets to: max.siebenborn[at]tu-darmstadt.de

References

[1] Wang, Dian, et al. “Equivariant diffusion policy.” 8th Annual Conference on Robot Learning, 2024.
[2] Yang, Jingyun, et al. “Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning.” 8th Annual Conference on Robot Learning, 2024.
[3] Apraez, Daniel Ordonez, et al. “Morphological symmetries in robotics.” The International Journal of Robotics Research 44.10-11 (2025): 1743-1766.
[4] Finzi, Marc, Gregory Benton, and Andrew G. Wilson. “Residual pathway priors for soft equivariance constraints.” Advances in Neural Information Processing Systems 34 (2021): 30037-30049.
[5] Wang, Rui, Robin Walters, and Rose Yu. “Approximately equivariant networks for imperfectly symmetric dynamics.” International Conference on Machine Learning. PMLR, 2022.
[6] Wilson, Andrew Gordon. “Deep learning is not so mysterious or different.” arXiv preprint arXiv:2503.02113 (2025).
[7] Wang, Dian, et al. “A general theory of correct, incorrect, and extrinsic equivariance.” Advances in Neural Information Processing Systems 36 (2023): 40006-40029.

Humans rely on vision, touch, and hearing to perceive contact and adapt how they manipulate objects. Most robot learning datasets, however, record only vision and proprioception. This leaves out the physical interaction signals that matter most in contact-rich tasks. Building on PolyUMI, our open-source wireless handheld gripper, this thesis will scale multisensory data collection beyond the lab. It will gather a large, diverse dataset of synchronized wrist-camera, optical tactile, contact-audio, and proprioceptive data in everyday environments such as kitchens, workshops, and homes. Using this data, the thesis will pretrain a shared visual-tactile-audio representation that captures how objects look, feel, and sound under interaction. The goal is to show that large-scale, in-the-wild multisensory pretraining yields representations that transfer to downstream robot policies. These representations should improve sample efficiency and robustness on contact-rich manipulation compared to vision-only or task-specific models.

Research Questions:

• How can useful visual-tactile-audio data be collected at scale in diverse, real-world environments?
• How can a representation be pretrained from such multisensory data so that it is useful for downstream task learning?
• How can the pretrained representation be combined with policy learning to achieve robust and efficient contact-rich manipulation?

What You Will Do

• Build a PolyUMI device with the support of the supervisor and use it to collect a diverse in-the-wild visual-tactile-audio dataset.
• Train a multimodal representation on the collected visual, tactile, and audio data.
• Integrate the pretrained representation into a robot policy and evaluate it on contact-rich manipulation tasks on a real robot.

Requirements

• Interest in hands-on robotics and hardware, and the motivation to collect data in real-world settings. Experience with ROS, robot manipulators, or sensor integration (tactile, audio, cameras) is a plus.
• Strong programming skills in Python and experience with PyTorch. Experience with large-scale data pipelines is a plus.
• Background in machine learning or deep learning, ideally including self-supervised and/or multimodal representation learning.

Opportunities
Your thesis could help robots perceive the world through touch and sound, not just vision. You will build your own PolyUMI device, work with real robots, and have the opportunity to publish your results to a international conference.

Interested?
Send a short (informal) motivation, CV, and grade sheets to: rickmer.krohn@tu-darmstadt.de

References

• PolyUMI: https://arxiv.org/abs/2609.29760
• Touch in the Wild: Learning Fine-Grained Manipulation with a – Portable Visuo-Tactile Gripper: https://arxiv.org/abs/2507.15062
• Universal Manipulation Interface (UMI): https://arxiv.org/abs/2402.10329

Scope: Master Thesis
Start Date: ASAP
Advisors: Vignesh Prasad

From manufacturing to assistive care, many real-world applications require humans and robots to carry objects collaboratively. Whole-body control offers a robust interface for robot learning approaches to generalize collaborative behaviours across dynamic environments. This thesis explores adaptive whole-body control that lets a robot dynamically synchronize its entire kinematic chain with a human partner during collaborative carrying. The goal is to learn safe, efficient human-robot collaboration through intuitive full-body coordination, rather than treating the end-effector as the only point of interaction.

Research Questions:

• How can a mobile manipulator adapt its whole-body impedance or admittance behaviour online to stay synchronized with a human partner during joint carrying?
• How much of this coordination can be learned from demonstrations or reinforcement learning rather than hand-tuned, and how does that affect safety and user experience?

What You Will Do

• Study and implement adaptive impedance and admittance controllers on a mobile manipulator
• Explore how imitation learning or reinforcement learning can be integrated into this whole-body control loop for human-robot interaction
• Evaluate the resulting controllers and learned policies through user studies with a real robot

Requirements

• Strong background in robotics and experience with robot learning
• Strong Python and potentially C++ skills
• Bonus: familiarity with imitation learning or reinforcement learning

Opportunities
Your thesis could help bridge the gap between learning-based approaches and classical controllers for Human-Robot Collaboration. You will be given the opportunity to work with real robots and publish your results.

Interested?
Send a short (informal) motivation, CV, and grade sheets to: vignesh.prasad@tu-darmstadt.de

References

https://yushi-du.github.io/COLA
https://h2compact.github.io/h2compact/

Scope: Master Thesis
Start Date: ASAP
Advisors: Jiayun Li, Georgia Chalvatzaki
Keywords: Reactive Robotic Manipulation, Computational Geometry, Model Predictive Control

Topic:

Ergodic planning is highly valuable for applications involving surface interaction, including industrial automation, surgical robotics, and other surface manipulation tasks. This thesis builds upon our previous work on surface ergodic coverage and extends it to more complex surface manipulation tasks. Since existing task-space ergodic control methods are limited by the reachable workspace of a single manipulator, this thesis explores bimanual systems to enable broader and more flexible surface interaction. The main challenge is to develop a reactive bimanual planning policy that can adapt to visual feedback.

Research Questions:

Formulating Time-Varying Variational Inference for Reactive Ergodic Motion Planning from Visual Feedback

Your Tasks:

• Learn about our current methods and help shape a new approach
• Participate in the development of experimental equipment
• Contribute to a journal submission with advisor

Requirements:
Good Python skills, particularly in numerical computing, are preferred. Familiarity with model predictive control (MPC), optimization algorithms, or camera systems would be helpful. Experience working with real robotic platforms is an advantage. Candidates should also be motivated to conduct research and work toward academic publications.

References:

[1] Li, Jiayun, et al. “Stein variational ergodic surface coverage with se (3) constraints.” ICRA (2026).

[2] Bilaloglu, Cem, Tobias Löw, and Sylvain Calinon. “Tactile ergodic coverage on curved surfaces.” IEEE Transactions on Robotics 41 (2025): 1421-1435.

Contact:

Interested students are invited to send a short, informal statement of motivation, a CV, an academic transcript if available, and a brief description of relevant experience to jiayun.li@tu-darmstadt.de.

Scope: Master Thesis
Advisors: Franziska Herbert

Learning to sequence manipulation primitives for supermarket replenishment is a challenging problem: it requires adapting to a large number of objects in a dynamically changing environment and reasoning over long-horizon task progressions. Scene graphs are a natural representations of such environments. Previous work has investigated how to predict actions from graph representations, but the graphs have remained largely static and do not account for the dynamic nature of the problem. An effective method has to handle object changes, react to failed actions, and adapt the graph structure based on object importance and task state.

In this thesis you will develop a method that learns dynamic scene graph representations to sequence manipulation primitives under these conditions.

Research Goal & Expected Outcomes:

• Study existing methods on sequencing robotic primitives based on dynamic scene graph representations
• Develop a pipeline for our bimanual mobile manipulator Tiago++
• Work toward a research publication

Requirements: Python (Advanced), Machine Learning (Advanced), Robotics & Control (Basics), Motivation & Willingness to learn (Essential)

We Provide: Real bimanual robot system, computational resources, mentorship, project collaboration and opportunity to publish in top tier conferences.

Related Work:

Herbert et al. Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning. IEEE Robotics and Automation Letters, vol. 11, no. 8, 2026.

Apply:
Write an email to franziska.herbert[at]tu-darmstadt.de. Please shortly state your motivation and experience (courses, projects) and include your transcript of records.

Scope: Bachelor or Master Thesis
Advisors: Sophie Lueth, Theo Gruner

Accurate trajectory tracking in robotic systems using (soft) torque control is often complicated by significant unmodeled friction (and other dissipative effects). The absence of a link-side torque sensor in most robots makes it particularly challenging to accurately estimate the required dynamics-feedforward torque, which is crucial for precise motion.

This thesis investigates whether data-driven methods can effectively compensate the unmodeled dynamic phenomena to improve the accuracy of dynamic motion tracking. Our long-termerm goal is to investigate whether this approach benefits from neural scaling-laws by learning a unified model for a multitude of actuators and kinematic chains.

Requirements:

• Genuine Interest in the topic
• Foundational Machine Learning Course(s)
• Basic Mechanics Understanding

Apply:
Write an email to sophie.lueth[at]tu-darmstadt.de & theo.gruner[tu]tu-darmstadt.de. Please shortly state your motivation and experience (courses, projects) and if available, transcript of records.

Related Work:
Sorrentino, Ines, et al. “Physics-informed learning for the friction modeling of high-ratio harmonic drives.” 2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids). IEEE, 2024.
Schulze, Lucas, et al. “Floating-Base Deep Lagrangian Networks.” arXiv preprint arXiv:2510.17270 (2025).
Luo, Di, et al. “Research on Parameter Identification Method for Robotic Manipulators Joint Friction Model Based on PINN.” 2024 IEEE International Conference on Advanced Intelligent Mechatronics (AIM). IEEE, 2024.

Scope: Master Thesis
Advisor: Christina Kassab

Can a robot use its own interactions to discover and correct errors in a scene graph?

Open-vocabulary 3D scene graphs are a widely-used representation for task planning [1, 2]. However, because they are often built from foundation-model predictions errors such as over- or under-segmentation are carried directly into the graph leading to failures in downstream tasks. Interactive perception has shown that observed motion can correct object segmentation [3], and failure-reasoning methods can identify why a task failed [4]; neither signal however is used to correct the scene graph itself.

This thesis addresses this gap by assigning each node and edge a confidence that is updated from observation, interaction and task outcome, and by repairing elements of the graph whose confidence falls.

Research Questions:

• How far can repeated observation alone correct a scene graph, and which errors does it leave unresolved?
• Can interaction and task outcome resolve the remaining errors, and with how few interactions?

What You Will Do:

• Observation: quantify the errors of existing scene graphs in simulated and real cluttered scenes; develop the confidence update and the corresponding graph repairs (node split and merge, object segmentation refinement, relation correction)
• Interaction: extend the update with object motion observed during manipulation and the task outcome, and compare against the observation-only graph
• Extension: transfer verified corrections to semantically similar objects in the scene

Requirements: Python, 3D Perception & Mapping, Robotics & Control (Basics)

Opportunities: This project offers hands-on work with vision foundation models, 3D scene graphs and robot manipulation, with the possibility of real-robot experiments. You will also receive mentorship and the opportunity to publish in top tier conferences.

Apply: Please write an email to christina.kassab[at]tu-darmstadt.de. Please state your motivation and experience (courses, projects) and include your transcript of records.

[1] Q. Gu et al., “ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning,” ICRA 2024.

[2] K. Rana et al., “SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning,” CoRL 2023.

[3] H. Qian et al., “RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features,” ICRA, 2024.

[4] Z. Liu et al, “REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction,” CoRL 2023.

Scope: Master Thesis
Advisor: Christina Kassab & Nils Dengler

Many manipulation instructions depend on properties a robot has not previously observed or cannot be observed from vision alone (for example, “find me a ripe avocado”).    Recent work uses LLMs to propose hypotheses about such properties and lets the robot verify them through observation and interaction [1, 2]. However, these hypotheses are usually tracked with discrete labels (true, false, unknown).

This thesis explores how a robot can maintain continuous confidences over its hypotheses and use them to guide interaction in tabletop manipulation. An LLM proposes hypotheses about object properties, each paired with a verification check. Their confidence is then updated as evidence arrives from visual inspection and simple physical checks such as lifting or squeezing. Confidence can be estimated in several ways: from the LLM itself, from agreement across samples or models, through probabilistic belief updates, or with a learned model that combines prior knowledge with interaction evidence. The goal is for the robot to choose informative checks, avoid unnecessary interactions and act once it is confident enough.

Research Questions:

• How can a robot represent and update continuous confidence over LLM-generated hypotheses as evidence accumulates?
• How well do different confidence estimates reflect actual hypothesis correctness, and how can calibration be improved?
• Do calibrated confidences lead to fewer unnecessary interactions and higher task success than discrete labels?

What You Will Do:

• Develop and compare approaches for estimating and updating hypothesis confidence, and evaluate how well-calibrated they are.
• Use the confidences to select checks and decide when to act, and compare against a discrete-label baseline.
• Extension: after a failed action, use the confidences to identify which assumptions most likely caused the failure, and revise those

Requirements:

• Python skills and experience with PyTorch,
• Background in machine learning; familiarity with uncertainty estimation is a plus
• Bonus: experience with LLMs or VLMs and simulation

Opportunities: You will work at the intersection of foundation models, uncertainty and manipulation, with the possibility of real-robot experiments and the opportunity to publish your results.

Apply: Please write an email to christina.kassab[at]tu-darmstadt.de and nils.dengler[at]tu-darmstadt.de. Please state your motivation and experience (courses, projects) and include your transcript of records.

[1] Xiao et al., “Hypothesis-driven Model Expansion under Uncertainty for Open-World Robot Planning,” arXiv 2026


[2] Lin et al., “PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration,” arXiv 2026


[3] A. Z. Ren et al., “Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners,” CoRL 2023.


[4] Z. Liu et al., “REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction,” CoRL 2023..