6th Robot Learning Workshop: Pretraining, Fine-Tuning, and Generalization with Large Scale Models

Abstract

Large pre-trained models have accelerated progress in many domains of machine learning research, such as text generation, chatbots, and image generation. In the 6th iteration of the Robot Learning workshop at NeurIPS, we will create a space for researchers from diverse backgrounds to gather and discuss the opportunities, challenges, and risks associated with large models in robotics research. Robotics is one of the most exciting and diverse applications for machine learning. It is both a hard challenge and a fruitful source of problems for machine learning approaches and our workshop is a space for members of both communities to meet.

The topic is chosen purposefully to be broad in terms of modalities and data sources as we are interested in different ideas of how pre-training can be applied to robotics. The combination of pre-trained models for vision and language for example has recently led to rapid progress in robotic tasks such as high-level planning or scene understanding. While pre-training on large-scale datasets usually comes with the benefit of generalization capabilities, it poses novel challenges that need to be addressed. The pre-training dataset can come from a wide range of sources with different perception systems in a range of environments. Therefore fine-tuning is an essential step in order to use large-scale models for a specific task. How to efficiently perform this fine-tuning, typically with limited hardware, while also ensuring a safe deployment remains an open research question.

Invited Speakers

• Deepak Pathak (CMU)
• Jesse Thomason (USC)
• Masha Itkina (Toyota Research Institute)
• Dhruv Batra (Georgia Tech and Meta AI)
• Matt Barnes (Google Research)
• Suraj Nair (Stanford)
• Keerthana Gopalakrishnan (Google Brain)
• Montserrat González Arenas (Google Brain)
• Fei Xia (Google DeepMind)
• Andrey Kolobov (Microsoft Research)
• Arjun Majumdar (Georgia Tech)

Schedule

Contributed talks and posters, invited talks, and a debate.

Live Demos

• Knolling bot 2.0: Enhancing Object Organization with Self-supervised Graspability Estimation, Yuhang Hu video
• On Bringing Robots Home: Dobb·E, Nur Muhammad “Mahi” Shafiullah, Haritheja Etukuru video
• Pupper: An Open-Source Quadruped Robot and Curriculum for AI Robotics Education, Gabrael Levine, Stuart Bowers, Teresa Nguyen video
• CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controller, Yuxiang Yang, Guanya Shi video
• ViNT: A Foundation Model for Visual Navigation, Dhruv Shah video
• Dexterous In-hand Object Rototation, Haozhi Qi video
• hello robot video

Accepted Papers

• A Statistical Guarantee for Representation Transfer in Multitask Imitation Learning
• Sample-Efficient Online Imitation Learning using Pretrained Behavioural Cloning Policies
• LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation
• Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning
• T3GDT: Three-Tier Tokens to Guide Decision Transformer for Offline Meta Reinforcement Learning
• World Model Based Sim2Real Transfer for Visual Navigation
• Learning to Act from Actionless Videos through Dense Correspondences
• LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion
• Reinforcement-learning robotic sailboats: simulator and preliminary results
• Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks
• Policy-Guided Diffusion
• Trajeglish: Learning the Language of Driving Scenarios
• D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation
• Exploitation-Guided Exploration for Semantic Embodied Navigation
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
• EvIL: Evolution Strategies for Generalisable Imitation Learning
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Vision-Language Models Provide Promptable Representations for Reinforcement Learning
• Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
• $\texttt{PREMIER-TACO}$ is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Loss
• Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment
• Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
• A Survey on Generalization in Deep Reinforcement Learning
• Pre-Trained Binocular ViTs for Image-Goal Navigation
• Human Scene Transformer
• CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controller
• IG-Net: Image-Goal Network for Offline Visual Navigation on A Large-Scale Game Map
• DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models
• Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models
• A$^2$Nav: Action-Aware Zero-Shot Robot Navigation Using Vision-Language Ability of Foundation Models
• Reasoning with Latent Diffusion in Offline Reinforcement Learning
• Robotic Task Generalization via Hindsight Trajectory Sketches
• RoboAgent: Towards Sample Efficient Robot Manipulation with Semantic Augmentations and Action Chunking
• LLM Augmented Hierarchical Agents
• Hybrid Inverse Reinforcement Learning
• Robotic Offline RL from Internet Videos via Value-Function Pre-Training
• Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design
• MultiReAct: Multimodal Tools Augmented Reasoning-Acting Traces for Embodied Agent Planning
• How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies
• Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild
• Language Models as Zero-Shot Trajectory Generators
• Causal Influence Aware Counterfactual Data Augmentation
• RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
• Knolling bot 2.0: Enhancing Object Organization with Self-supervised Graspability Estimation

Organizers

• Dhruv Shah (UC Berkeley)
• Claas Voelcker (University of Toronto)
• Paula Wulkop (ETH Zurich)
• Georgia Chalvatzaki (TU Darmstadt, Germany)
• Ransalu Senanayake (Stanford University and Arizona State University)
• Julien Perez (Naver Labs Europe)
• Jonathan Tompson (Google DeepMind, Mountain View)
• Hamidreza Kasaei (University of Groningen, Netherlands)
• Alex Bewley (Google DeepMind, Zurich)

Advisory Board

• Roberto Calandra (TU Dresden)
• Markus Wulfmeier (Google DeepMind, London)
• Ingmar Posner (Oxford University)
• Danica Kragic (KTH)
• Fabio Ramos (NVIDIA, University of Sydney)
• Vincent Vanhoucke (Google DeepMind, Mountain View)
• Igor Gilitschenski (University of Toronto)

Important dates

• Submission deadline: 06 October 2023 (Anywhere on Earth)
• Notification: 27 October 2023 (Anywhere on Earth)
• Camera-ready due: 1 December 2023 (Anywhere on Earth)
• Workshop: 16 December 2023

Call for Papers

The workshop aims to highlight both favorable and critical voices with regard to the emerging trend of large scale pre-training to encourage a lively debate and meaningful exchange among the presenters and attendees. We encourage the submissions of original research as workshop papers as well as the submission of videos of robot demonstrations which could be shown live during the workshop.

Specific areas of interest include, but are not limited to:

• the role of pre-training from offline data, self-play, imitation, or other source in robotics pipelines;
• generalization of pre-trained models to novel tasks and environments;
• combination of different data modalities for training large models in robotics;
• finetuning, or other modular adaptation mechanisms for deploying pre-trained models on a new environment;
• combining large models and multimodal training for robotics,
• safe real-world deployment of pre-trained models;
• opportunities and challenges arising from embodiments and data collection;
• datasets and method proposals for collecting, curating, and sharing pre-training data for robotics

Call for Demos

We encourage the submission of demos of robotics setups in our workshop. All submitted demos must include a live component, either via bringing the system to the physical conference, or by setting up a livestream to a lab environment. In the latter case, we would like to encourage that the demonstration be interactive, i.e. by allowing audience submitted tasks. The demonstrations will be held during the poster session.

Please submit demo proposals by filling out the form here.

Submission Instructions

Submissions should use the NeurIPS Workshop template available here and be 4 pages (plus as many pages as necessary for references). The reviewing process will be double blind, so please submit as anonymous by using ‘\usepackage{neurips_wrl2023}’ in your main tex file.

Accepted papers and eventual supplementary material will be made available on the workshop website. However, this does not constitute an archival publication and no formal workshop proceedings will be made available, meaning contributors are free to publish their work in archival journals or conference.

Submissions can be made at openreview.

FAQ

1. Can supplementary material be added beyond the 4-page limit and are there any restrictions on it?

   Yes, you may include additional supplementary material, but we ask that it be limited to a reasonable amount (max 10 pages in addition to the main submission) and that it follow the same NeurIPS format as the paper. References do not count towards the limit of 4 pages.

2. Can a submission to this workshop be submitted to another NeurIPS workshop in parallel?

   We discourage this, as it leads to more work for reviewers across multiple workshops and it will be hard to attend workshops in parallel. Our suggestion is to pick one workshop to submit to.

3. Can a paper be submitted to the workshop that has already appeared at a previous conference with published proceedings?

   We will not be accepting such submissions unless they have been adapted to contain significantly new results (where novelty is one of the qualities reviewers will be asked to evaluate). However, we will accept submissions that are under review at the time of submission to our workshop. For instance, papers that have been submitted to the International Conference on Robotics and Automation (ICRA) 2024 or the International Conference on Learning Representations (ICLR) 2024 can be submitted to our workshop.

4. Do demos also have to have a paper?

   We encourage authors to submit papers with their demos if there is valuable scientific content for the workshop audience, but we do not require it. The acceptance of the demonstration will be decided independent of any submitted paper and you may submit a demo proposal that builds on published work from this year.

Contacts

For any further questions, you can contact us at neuripswrl2023@robot-learning.ml

Sponsors

We are very thankful to our corporate sponsors for enabling us to provide best paper awards and student registration fees.

If you would like to sponsor the workshop, please contact neuripswrl2023@robot-learning.ml.

Financial Support

We have funding from our sponsors for financial aid of authors or attendees from under-represented groups.

Application form for financial aid to attend the conference (deadline: Nov 22nd): Click Here!