GROOT: Learning to Follow Instructions by Watching Gameplay Videos

Shaofei Cai1,2, Bowei Zhang3, Zihao Wang1,2, Xiaojian Ma5, Anji Liu4, Yitao Liang1*
Team CraftJarvis
1Institute for Artificial Intelligence, Peking University
2School of Intelligence Science and Technology, Peking University
3School of Electronics Engineering and Computer Science, Peking University
4Computer Science Department, University of California, Los Angeles
5Beijing Institute for General Artificial Intelligence (BIGAI)
*Indicates Corresponding Author
Paper Code arXiv

Our GROOT can solve open-ended tasks specified by reference videos in the open-world Minecraft.

Abstract

We study the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space. We implement our agent GROOT in a simple yet effective encoder-decoder architecture based on causal transformers. We evaluate GROOT against open-world counterparts and human players on a proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the human-machine gap as well as exhibiting a 70% winning rate over the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis.

Architecture Design

MY ALT TEXT


Left: In the training stage, a video encoder (non-causal transformer) learns to extract the semantic meaning and transfer the video (state sequence) into the goal embedding space. A goal-conditioned policy (causal transformer) is learned to predict actions following the given instructions. We learn the agent using behavior cloning under a KL constraint.
Right: During the inference, any reference video is passed into the video encoder to generate the goal embeddings that drives the policy to interact with the environment.

Experimental Results

Minecraft SkillForge Benchmark

In order to comprehensively evaluate the mastery of tasks by agents in Minecraft, we created a diverse benchmark called Minecraft SkillForge. It covers 30 tasks from 6 major categories of representative skills in Minecraft, including collect, explore, craft, tool, survive, and build.

The CraftJarvis Series

BibTeX


@misc{cai2023groot,
    title={GROOT: Learning to Follow Instructions by Watching Gameplay Videos}, 
    author={Shaofei Cai and Bowei Zhang and Zihao Wang and Xiaojian Ma and Anji Liu and Yitao Liang},
    year={2023},
    eprint={2310.08235},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}