Uncovering Limitations of Large Language Models in Information Seeking from Tables

Chaoxu Pang, Yixuan Cao, Chunhao Yang, Ping Luo


Abstract
Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4-turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.
Anthology ID:
2024.findings-acl.82
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1388–1409
Language:
URL:
https://aclanthology.org/2024.findings-acl.82
DOI:
10.18653/v1/2024.findings-acl.82
Bibkey:
Cite (ACL):
Chaoxu Pang, Yixuan Cao, Chunhao Yang, and Ping Luo. 2024. Uncovering Limitations of Large Language Models in Information Seeking from Tables. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1388–1409, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Uncovering Limitations of Large Language Models in Information Seeking from Tables (Pang et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.82.pdf