Recently developed foundation models, such as large language models and multi-modal models, open great opportunities to build generally capable agents, combined with reinforcement learning. This project focuses on learning skills and foundation models and connecting them to build generalist agents. In the following, we introduce some of our studies. For more details, please refer to the papers. Plan4MC We study building a multi-task agent in Minecraft. Without human demonstrations, solving long-horizon tasks in this open-ended environment with reinforcement learning (RL) is extremely sample inefficient. [Read More…]