Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki
TLDR: Semantic–geometric task graph-representations learned from human demonstrations enable generalizable bimanual robot manipulation.
Bimanual manipulation requires understanding how actions, objects, and their geometric relations unfold over time, a structure that varies significantly across task executions. We present a semantic–geometric graph-based task representation that jointly encodes object identities, inter-object relations, and motion histories using an MPNN encoder and Transformer-based decoder. The encoder operates on temporal scene graphs without action labels, learning task-agnostic representations that transfer across embodiments via decoder-only finetuning. Evaluated on eleven bimanual tasks, our structured semantic–geometric representations consistently outperform sequence-based models, with gains growing with task variability. At deployment, a planner couples action and motion predictions with Probabilistic Movement Primitives, achieving full task success on two real-robot bimanual tasks and outperforming baselines.
Check our website for more details: https://frherbert.github.io/bimanual-task-graphs/