Minji Jung
2024
Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering
Minji Jung
|
Soyeon Park
|
Jeewoo Sul
|
Yong Suk Choi
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)
Prompt tuning, which freezes all parameters of a pre-trained model and only trains a soft prompt, has emerged as a parameter-efficient approach. For the reason that the prompt initialization becomes sensitive when the model size is small, the prompt transfer that uses the trained prompt as an initialization for the target task has recently been introduced. Since previous works have compared tasks in large categories (e.g., summarization, sentiment analysis), the factors that influence prompt transfer have not been sufficiently explored. In this paper, we characterize the question answering task based on features such as answer format and empirically investigate the transferability of soft prompts for the first time. We analyze the impact of initialization during prompt transfer and find that the train dataset size of source and target tasks have the influence significantly. Furthermore, we propose a novel approach for measuring catastrophic forgetting and investigate how it occurs in terms of the amount of evidence. Our findings can help deeply understand transfer learning in prompt tuning.
2021
VUS at IWSLT 2021: A Finetuned Pipeline for Offline Speech Translation
Yong Rae Jo
|
Youngki Moon
|
Minji Jung
|
Jungyoon Choi
|
Jihyung Moon
|
Won Ik Cho
Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)
In this technical report, we describe the fine-tuned ASR-MT pipeline used for the IWSLT shared task. We remove less useful speech samples by checking WER with an ASR model, and further train a wav2vec and Transformers-based ASR module based on the filtered data. In addition, we cleanse the errata that can interfere with the machine translation process and use it for Transformer-based MT module training. Finally, in the actual inference phase, we use a sentence boundary detection model trained with constrained data to properly merge fragment ASR outputs into full sentences. The merged sentences are post-processed using part of speech. The final result is yielded by the trained MT module. The performance using the dev set displays BLEU 20.37, and this model records the performance of BLEU 20.9 with the test set.
Search
Co-authors
- Soyeon Park 1
- Jeewoo Sul 1
- Yong Suk Choi 1
- Yong Rae Jo 1
- Youngki Moon 1
- show all...