Hejia Zhang

I am a PhD candidate in computer science (robotics & AI) at the Interactive and Collaborative Autonomous Robotic Systems (ICAROS) Lab advised by Prof. Stefanos Nikolaidis at the University of Southern California (USC). I am also working closely with Prof. Daniel Seita. Before starting my PhD, I received my Master's degree from USC, where I worked as a research assistant at the Robotic Embedded Systems Laboratory (RESL) and was advised by Prof. Gaurav Sukhatme and Prof. Stefan Schaal. I got my bachelor's degree from Zhejiang University (ZJU), where I started my study on robotics.

Job Market



My research goal is to enable robots to intelligently perform long-horizon, complex (e.g., deformable object, collaborative...) manipulation tasks in unstructured human environments (e.g., kitchens, underground mines...).

To achieve my research goal, I am interested in developing fundamental learning and planning algorithms for robot manipulation tasks.

Learning and Planning for Robot Manipulation in Human Environments

GPT-Fabric: Folding and Smoothing Fabric by Leveraging Pre-Trained Foundation Models
Vedant Raval*, Enyu Zhao*, Hejia Zhang, Stefanos Nikolaidis, Daniel Seita
Under Review
[PDF] [Project]

We utilize pre-trained large language models to generate robot actions for fabric smoothing and folding tasks.

Multi-Robot Geometric Task-and-Motion Planning for Collaborative Manipulation Tasks
Hejia Zhang, Shao-Hung Chan, Jie Zhong, Jiaoyang Li, Peter Kolapo, Sven Koenig, Zach Agioutantis, Steven Schafrik, Stefanos Nikolaidis
Autonomous Robots (AURO), 2023
The design of the autonomous roof bolting system: Technical Report for Alpha Foundation for the Improvement of Mine Safety and Health

This work is an extended version of our previous work on multi-robot geometric task-and-motion planning (MR-GTAMP). We conduct an application study on the roof-bolting task, which is an essential operation within the underground mining cycle. We conduct additional scalability evaluation experiments. We substantially expand the description of the previous MR-GTAMP framework.

PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection
Shivin Dass*, Karl Pertsch*, Hejia Zhang, Youngwoon Lee, Joseph J. Lim, Stefanos Nikolaidis
Robotics: Science and Systems (RSS), 2023
Also at Pre-training for Robot Learning @ CoRL 2022.
[Code] [PDF] [Project]

We enable scalable robot data collection by assisting human teleoperators with a learned policy. Our approach estimates its uncertainty over future actions to determine when to request user input. In real world user studies we demonstrate that our system enables more efficient teleoperation with reduced mental load and up to four robots in parallel.

A MIP-Based Approach for Multi-Robot Geoemtric Task-and-Motion Planning
Hejia Zhang, Shao-Hung Chan, Jie Zhong, Jiaoyang Li, Sven Koenig, Stefanos Nikolaidis
IEEE International Conference on Automation Science and Engineering (CASE), 2022
Also at SCR 2022.
[PDF] [Slides]

We address multi-robot geometric task-and-motion planning (MR-GTAMP) problems in synchronous, monotone setups. The key insight is that pre-computed manipulation capabilities of individual robots can be used to guide multi-robot planning.

Zero-Shot Imitating Collaborative Manipulation Plans from YouTube Cooking Videos
Hejia Zhang, Jie Zhong, Stefanos Nikolaidis
Emergent Behaviors in Human-Robot Systems @ RSS, 2020
Featured as Paper of the Month by Kinova Robotics.
[PDF] [Slides] [Talk]

Previous work has shown that the space of human manipulation actions has a linguistic, hierarchical structure that relates actions to manipulated objects and tools. Building upon this theory of language for action, we propose a system for understanding and executing demonstrated action sequences from full-length, real-world cooking videos on the web.

Learning Collaborative Action Plans from YouTube Videos
Hejia Zhang, Po-Jen Lai, Sayan Paul, Suraj Kothawade and Stefanos Nikolaidis
International Symposium on Robotics Research (ISRR), 2019
[BibTeX] [PDF]

We present a system for knowledge acquisition of collaborative manipulation action plans that outputs commands to the robot in the form of visual sentence.

Scaling Simulation-to-Real Transfer by Learning a Latent Space of Robot Skills
Ryan Julian, Eric Heiden, Zhanpeng He, Hejia Zhang, Stefan Schaal, Joseph J. Lim, Gaurav S. Sukhatme, Karol Hausman
International Journal of Robotics Research (IJRR), 2020
Also at Deep RL Workshop @ NeurIPS 2018.

This is an extended verion of our previous work on sim-to-real transfer. We show an algorithm which allows our sim-to-real method to perform long-horizon tasks never seen in simulation, by intelligently sequencing short-horizon latent skills.

Scaling simulation-to-real transfer by learning composable robot skills
Ryan Julian*, Eric Heiden*, Zhanpeng He, Hejia Zhang, Stefan Schaal, Joseph J. Lim, Gaurav S. Sukhatme, Karol Hausman
International Symposium on Experimental Robotics (ISER), 2018
[BibTeX] [PDF]

We present a novel solution to the problem of simulation-to-real transfer, which builds on recent advances in robot skill decomposition.

Evaluating Human-Robot Interaction Systems

Surrogate Assisted Generation of Human-Robot Interaction Scenarios
Varun Bhatt, Heramb Nemlekar, Matthew C. Fontaine, Bryon Tjanaka, Hejia Zhang, Ya-Chuan Hsu, Stefanos Nikolaidis
Conference on Robot Learning (CoRL), 2023 (Oral Presentation; 6.6% acceptance rate)
Also at EGG @ RSS 2023.

We propose augmenting scenario generation systems with surrogate models that predict both human and robot behaviors. In the shared control teleoperation domain and a more complex shared workspace collaboration task, we show that surrogate assisted scenario generation efficiently synthesizes diverse datasets of challenging scenarios.

Video Game Level Repair via Mixed Integer Linear Programming
Hejia Zhang*, Matthew Fontaine*, Amy Hoover, Julian Togelius, Bistra Dilkina, Stefanos Nikolaidis
AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), 2020 (Oral Presentation; 25% acceptance rate)
[BibTeX] [Code] [PDF]

We propose a “generate-then-repair” framework for automatic generation of playable levels adhering to specific styles.


I developed this learning from YouTube videos system and presented it [here]. It was featured as Paper of the Month by Kinova Robotics.


I contributed to the development of a robot hair brushing demo and presented the demo in NeurIPS 2019! It was highlighted at the Fortune.

I developed this fun feeding demo for Thanksgiving 2019!