Arafat Ahsan
2023
Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Language
Vandan Mujadia
|
Pruthwik Mishra
|
Arafat Ahsan
|
Dipti M. Sharma
Proceedings of the 20th International Conference on Natural Language Processing (ICON)
With the primary focus on evaluating the effectiveness of large language models for automatic reference-less translation assessment, this work presents our experiments on mimicking human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evaluation task where we performed zero-shot learning, in-context example-driven learning, and fine-tuning of large language models to provide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA2-13B) achieves a comparable or higher overall correlation with human judgments for the considered Indian language pairs (Refer figure 1).
2021
Assessing Post-editing Effort in the English-Hindi Direction
Arafat Ahsan
|
Vandan Mujadia
|
Dipti Misra Sharma
Proceedings of the 18th International Conference on Natural Language Processing (ICON)
We present findings from a first in-depth post-editing effort estimation study in the English-Hindi direction along multiple effort indicators. We conduct a controlled experiment involving professional translators, who complete assigned tasks alternately, in a translation from scratch and a post-edit condition. We find that post-editing reduces translation time (by 63%), utilizes fewer keystrokes (by 59%), and decreases the number of pauses (by 63%) when compared to translating from scratch. We further verify the quality of translations thus produced via a human evaluation task in which we do not detect any discernible quality differences.
2010
Coupling Statistical Machine Translation with Rule-based Transfer and Generation
Arafat Ahsan
|
Prasanth Kolachina
|
Sudheer Kolachina
|
Dipti Misra
|
Rajeev Sangal
Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers
In this paper, we present the insights gained from a detailed study of coupling a highly modular English-Hindi RBMT system with a standard phrase-based SMT system. Coupling the RBMT and SMT systems at various stages in the RBMT pipeline, we observe the effects of the source transformations at each stage on the performance of the coupled MT system. We propose an architecture that systematically exploits the structural transfer and robust generation capabilities of the RBMT system. Working with the English-Hindi language pair, we show that the coupling configurations explored in our experiments help address different aspects of the typological divergence between these languages. In spite of working with very small datasets, we report significant improvements both in terms of BLEU (7.14 and 0.87 over the RBMT and the SMT baselines respectively) and subjective evaluation (relative decrease of 17% in SSER).
Search
Co-authors
- Dipti Misra Sharma 2
- Vandan Mujadia 2
- Prasanth Kolachina 1
- Sudheer Kolachina 1
- Rajeev Sangal 1
- show all...