LangSuit·E: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments

Zixia Jia, Mengmeng Wang, Baichen Tong, Song-Chun Zhu, Zilong Zheng


Abstract
Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely onlanguage descriptions as inputs. However, it remains unclear how well LLMs can function as few-shot or zero-shot embodied agents in dynamic interactive environments. To address this gap, we introduce LangSuit·E, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuit·E (i) offers adaptability to diverse environments without multiple simulation engines, (ii) evaluates agents’ capacity to develop “internalized world knowledge” with embodied observations, and (iii) allows easy customization of communication and action strategies. To address the embodiment challenge, we devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information. Comprehensive benchmark results illustrate challenges and insights of embodied planning. LangSuit·E represents a significant step toward building embodied generalists in the context of language models.
Anthology ID:
2024.findings-acl.879
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
14778–14814
Language:
URL:
https://aclanthology.org/2024.findings-acl.879
DOI:
10.18653/v1/2024.findings-acl.879
Bibkey:
Cite (ACL):
Zixia Jia, Mengmeng Wang, Baichen Tong, Song-Chun Zhu, and Zilong Zheng. 2024. LangSuit·E: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments. In Findings of the Association for Computational Linguistics ACL 2024, pages 14778–14814, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
LangSuit·E: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments (Jia et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.879.pdf