AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations

Zhicheng Yang, Yinya Huang, Jing Xiong, Liang Feng, Xiaodan Liang, Yiwei Wang, Jing Tang


Abstract
Large Language Models prompting, such as using in-context demonstrations, is a mainstream technique for invoking LLMs to perform high-performance and solid complex reasoning (e.g., mathematical reasoning, commonsense reasoning), and has the potential for further human-machine collaborative scientific findings. However, current LLMs are delicate and elusive in prompt words and styles. And there is an unseen gap between LLM understanding and human-written prompts. This paper introduces AlignedCoT, an LLM-acquainted prompting technique that includes proficient “native-speaking” in in-context learning for the LLMs. Specifically, it achieves consistent and correct step-wise prompts in zero-shot scenarios by progressively probing, refining, and formatting the LLM chain of thoughts so that free from handcrafted few-shot demonstrations while maintaining the prompt quality. We conduct experiments on mathematical reasoning and commonsense reasoning. We find that LLMs with AlignedCoT perform significantly superior to them with human-crafted demonstrations. We further apply AlignedCoT for rewriting the GSM8k training set, resulting in a GSM8k-Align dataset. We observe its benefits for retrieval augmented generation.
Anthology ID:
2024.findings-emnlp.163
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2857–2896
Language:
URL:
https://aclanthology.org/2024.findings-emnlp.163
DOI:
10.18653/v1/2024.findings-emnlp.163
Bibkey:
Cite (ACL):
Zhicheng Yang, Yinya Huang, Jing Xiong, Liang Feng, Xiaodan Liang, Yiwei Wang, and Jing Tang. 2024. AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2857–2896, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations (Yang et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-emnlp.163.pdf
Data:
 2024.findings-emnlp.163.data.zip