Jeremy Gwinnup


2024

pdf bib
Adding multimodal capabilities to a text-only translation model
Vipin Vijayan | Braeden Bowen | Scott Grigsby | Timothy Anderson | Jeremy Gwinnup
Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)

While most current work in multimodal machine translation (MMT) uses the Multi30k dataset for training and evaluation, we find that the resulting models overfit to the Multi30k dataset to an extreme degree. Consequently, these models perform very badly when evaluated against typical text-only testing sets such as the newstest datasets. In order to perform well on both Multi30k and typical text-only datasets, we use a performant text-only machine translation (MT) model as the starting point of our MMT model. We add vision-text adapter layers connected via gating mechanisms to the MT model, and incrementally transform the MT model into an MMT model by 1) pre-training using vision-based masking of the source text and 2) fine-tuning on Multi30k. We achieve a state-of-the-art performance on the Multi30k 2016 en-de test set of 46.5 BLEU4 score and 0.61 CoMMuTE score via this approach while retaining the performance of the original text-only MT model against the newstest dataset.

pdf bib
Detecting concrete visual tokens for Multimodal Machine Translation
Braeden Bowen | Vipin Vijayan | Scott Grigsby | Timothy Anderson | Jeremy Gwinnup
Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)

The challenge of visual grounding and masking in multimodal machine translation (MMT) systems has encouraged varying approaches to the detection and selection of visually-grounded text tokens for masking. We introduce new methods for detection of visually and contextually relevant (concrete) tokens from source sentences, including detection with natural language processing (NLP), detection with object detection, and a joint detection-verification technique. We also introduce new methods for selection of detected tokens, including shortest n tokens, longest n tokens, and all detected concrete tokens. We utilize the GRAM MMT architecture to train models against synthetically collated multimodal datasets of source images with masked sentences, showing performance improvements and improved usage of visual context during translation tasks over the baseline model.

2023

pdf bib
Enhancing Video Translation Context with Object Labels
Jeremy Gwinnup | Tim Anderson | Brian Ore | Eric Hansen | Kevin Duh
Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)

We present a simple yet efficient method to enhance the quality of machine translation models trained on multimodal corpora by augmenting the training text with labels of detected objects in the corresponding video segments. We then test the effects of label augmentation in both baseline and two automatic speech recognition (ASR) conditions. In contrast with multimodal techniques that merge visual and textual features, our modular method is easy to implement and the results are more interpretable. Comparisons are made with Transformer translation architectures trained with baseline and augmented labels, showing improvements of up to +1.0 BLEU on the How2 dataset.

pdf bib
Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)
Liling Tan | Dmitrijs Milajevs | Geeticka Chauhan | Jeremy Gwinnup | Elijah Rippeth
Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)

2021

pdf bib
Tune in: The AFRL WMT21 News-Translation Systems
Grant Erdmann | Jeremy Gwinnup | Tim Anderson
Proceedings of the Sixth Conference on Machine Translation

This paper describes the Air Force Research Laboratory (AFRL) machine translation sys- tems and the improvements that were developed during the WMT21 evaluation campaign. This year, we explore various methods of adapting our baseline models from WMT20 and again measure improvements in performance on the Russian–English language pair.

2020

pdf bib
The AFRL WMT20 News Translation Systems
Jeremy Gwinnup | Tim Anderson
Proceedings of the Fifth Conference on Machine Translation

This report summarizes the Air Force Research Laboratory (AFRL) machine translation (MT) systems submitted to the news-translation task as part of the 2020 Conference on Machine Translation (WMT20) evaluation campaign. This year we largely repurpose strategies from previous years’ efforts with larger datasets and also train models with precomputed word alignments under various settings in an effort to improve translation quality.

pdf bib
The AFRL IWSLT 2020 Systems: Work-From-Home Edition
Brian Ore | Eric Hansen | Tim Anderson | Jeremy Gwinnup
Proceedings of the 17th International Conference on Spoken Language Translation

This report summarizes the Air Force Research Laboratory (AFRL) submission to the offline spoken language translation (SLT) task as part of the IWSLT 2020 evaluation campaign. As in previous years, we chose to adopt the cascade approach of using separate systems to perform speech activity detection, automatic speech recognition, sentence segmentation, and machine translation. All systems were neural based, including a fully-connected neural network for speech activity detection, a Kaldi factorized time delay neural network with recurrent neural network (RNN) language model rescoring for speech recognition, a bidirectional RNN with attention mechanism for sentence segmentation, and transformer networks trained with OpenNMT and Marian for machine translation. Our primary submission yielded BLEU scores of 21.28 on tst2019 and 23.33 on tst2020.

2019

pdf bib
Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation
Brian Thompson | Jeremy Gwinnup | Huda Khayrallah | Kevin Duh | Philipp Koehn
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)

Continued training is an effective method for domain adaptation in neural machine translation. However, in-domain gains from adaptation come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catastrophic forgetting of general-domain knowledge. To mitigate it, we adapt Elastic Weight Consolidation (EWC)—a machine learning method for learning a new task without forgetting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading in-domain performance, outperforming the previous state-of-the-art. We also explore the full range of general-domain performance available when some in-domain degradation is acceptable.

pdf bib
The AFRL WMT19 Systems: Old Favorites and New Tricks
Jeremy Gwinnup | Grant Erdmann | Tim Anderson
Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)

This paper describes the Air Force Research Laboratory (AFRL) machine translation systems and the improvements that were developed during the WMT19 evaluation campaign. This year, we refine our approach to training popular neural machine translation toolkits, experiment with a new domain adaptation technique and again measure improvements in performance on the Russian–English language pair.

pdf bib
Quality and Coverage: The AFRL Submission to the WMT19 Parallel Corpus Filtering for Low-Resource Conditions Task
Grant Erdmann | Jeremy Gwinnup
Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)

The WMT19 Parallel Corpus Filtering For Low-Resource Conditions Task aims to test various methods of filtering a noisy parallel corpora, to make them useful for training machine translation systems. This year the noisy corpora are the relatively low-resource language pairs of Nepali-English and Sinhala-English. This papers describes the Air Force Research Laboratory (AFRL) submissions, including preprocessing methods and scoring metrics. Numerical results indicate a benefit over baseline and the relative benefits of different options.

2018

pdf bib
The AFRL IWSLT 2018 Systems: What Worked, What Didn’t
Brian Ore | Eric Hansen | Katherine Young | Grant Erdmann | Jeremy Gwinnup
Proceedings of the 15th International Conference on Spoken Language Translation

This report summarizes the Air Force Research Laboratory (AFRL) machine translation (MT) and automatic speech recognition (ASR) systems submitted to the spoken language translation (SLT) and low-resource MT tasks as part of the IWSLT18 evaluation campaign.

pdf bib
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation
Brian Thompson | Huda Khayrallah | Antonios Anastasopoulos | Arya D. McCarthy | Kevin Duh | Rebecca Marvin | Paul McNamee | Jeremy Gwinnup | Tim Anderson | Philipp Koehn
Proceedings of the Third Conference on Machine Translation: Research Papers

To better understand the effectiveness of continued training, we analyze the major components of a neural machine translation system (the encoder, decoder, and each embedding space) and consider each component’s contribution to, and capacity for, domain adaptation. We find that freezing any single component during continued training has minimal impact on performance, and that performance is surprisingly good when a single component is adapted while holding the rest of the model fixed. We also find that continued training does not move the model very far from the out-of-domain model, compared to a sensitivity analysis metric, suggesting that the out-of-domain model can provide a good generic initialization for the new domain.

pdf bib
The AFRL WMT18 Systems: Ensembling, Continuation and Combination
Jeremy Gwinnup | Tim Anderson | Grant Erdmann | Katherine Young
Proceedings of the Third Conference on Machine Translation: Shared Task Papers

This paper describes the Air Force Research Laboratory (AFRL) machine translation systems and the improvements that were developed during the WMT18 evaluation campaign. This year, we examined the developments and additions to popular neural machine translation toolkits and measure improvements in performance on the Russian–English language pair.

pdf bib
The AFRL-Ohio State WMT18 Multimodal System: Combining Visual with Traditional
Jeremy Gwinnup | Joshua Sandvick | Michael Hutt | Grant Erdmann | John Duselis | James Davis
Proceedings of the Third Conference on Machine Translation: Shared Task Papers

AFRL-Ohio State extends its usage of visual domain-driven machine translation for use as a peer with traditional machine translation systems. As a peer, it is enveloped into a system combination of neural and statistical MT systems to present a composite translation.

pdf bib
Coverage and Cynicism: The AFRL Submission to the WMT 2018 Parallel Corpus Filtering Task
Grant Erdmann | Jeremy Gwinnup
Proceedings of the Third Conference on Machine Translation: Shared Task Papers

The WMT 2018 Parallel Corpus Filtering Task aims to test various methods of filtering a noisy parallel corpus, to make it useful for training machine translation systems. We describe the AFRL submissions, including their preprocessing methods and quality metrics. Numerical results indicate relative benefits of different options and show where our methods are competitive.

2017

pdf bib
The AFRL-MITLL WMT17 Systems: Old, New, Borrowed, BLEU
Jeremy Gwinnup | Timothy Anderson | Grant Erdmann | Katherine Young | Michaeel Kazi | Elizabeth Salesky | Brian Thompson | Jonathan Taylor
Proceedings of the Second Conference on Machine Translation

pdf bib
The AFRL-OSU WMT17 Multimodal Translation System: An Image Processing Approach
John Duselis | Michael Hutt | Jeremy Gwinnup | James Davis | Joshua Sandvick
Proceedings of the Second Conference on Machine Translation

pdf bib
The AFRL WMT17 Neural Machine Translation Training Task Submission
Grant Erdmann | Katherine Young | Jeremy Gwinnup
Proceedings of the Second Conference on Machine Translation

2016

pdf bib
A Taxonomy of Weeds: A Field Guide for Corpus Curators to Winnowing the Parallel Text Harvest
Katherine M. Young | Jeremy Gwinnup | Lane O.B. Schwartz
Conferences of the Association for Machine Translation in the Americas: MT Users' Track

pdf bib
The AFRL-MITLL WMT16 News-Translation Task Systems
Jeremy Gwinnup | Tim Anderson | Grant Erdmann | Katherine Young | Michaeel Kazi | Elizabeth Salesky | Brian Thompson
Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers

pdf bib
The MITLL-AFRL IWSLT 2016 Systems
Michaeel Kazi | Elizabeth Salesky | Brian Thompson | Jonathan Taylor | Jeremy Gwinnup | Timothy Anderson | Grant Erdmann | Eric Hansen | Brian Ore | Katherine Young | Michael Hutt
Proceedings of the 13th International Conference on Spoken Language Translation

This report summarizes the MITLL-AFRL MT and ASR systems and the experiments run during the 2016 IWSLT evaluation campaign. Building on lessons learned from previous years’ results, we refine our ASR systems and examine the explosion of neural machine translation systems and techniques developed in the past year. We experiment with a variety of phrase-based, hierarchical and neural-network approaches in machine translation and utilize system combination to create a composite system with the best characteristics of all attempted MT approaches.

2015

pdf bib
The AFRL-MITLL WMT15 System: There’s More than One Way to Decode It!
Jeremy Gwinnup | Tim Anderson | Grant Erdmann | Katherine Young | Christina May | Michaeel Kazi | Elizabeth Salesky | Brian Thompson
Proceedings of the Tenth Workshop on Statistical Machine Translation

pdf bib
Drem: The AFRL Submission to the WMT15 Tuning Task
Grant Erdmann | Jeremy Gwinnup
Proceedings of the Tenth Workshop on Statistical Machine Translation

pdf bib
The MITLL-AFRL IWSLT 2015 MT system
Michaeel Kazi | Brian Thompson | Elizabeth Salesky | Timothy Anderson | Grant Erdmann | Eric Hansen | Brian Ore | Katherine Young | Jeremy Gwinnup | Michael Hutt | Christina May
Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign

2014

pdf bib
The MITLL-AFRL IWSLT 2014 MT system
Michaeel Kazi | Elizabeth Salesky | Brian Thompson | Jessica Ray | Michael Coury | Tim Anderson | Grant Erdmann | Jeremy Gwinnup | Katherine Young | Brian Ore | Michael Hutt
Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign

This report summarizes the MITLL-AFRL MT and ASR systems and the experiments run using them during the 2014 IWSLT evaluation campaign. Our MT system is much improved over last year, owing to integration of techniques such as PRO and DREM optimization, factored language models, neural network joint model rescoring, multiple phrase tables, and development set creation. We focused our eforts this year on the tasks of translating from Arabic, Russian, Chinese, and Farsi into English, as well as translating from English to French. ASR performance also improved, partly due to increased eforts with deep neural networks for hybrid and tandem systems. Work focused on both the English and Italian ASR tasks.

pdf bib
Machine Translation and Monolingual Postediting: The AFRL WMT-14 System
Lane Schwartz | Timothy Anderson | Jeremy Gwinnup | Katherine Young
Proceedings of the Ninth Workshop on Statistical Machine Translation

2013

pdf bib
The MIT-LL/AFRL IWSLT-2013 MT system
Michaeel Kazi | Michael Coury | Elizabeth Salesky | Jessica Ray | Wade Shen | Terry Gleason | Tim Anderson | Grant Erdmann | Lane Schwartz | Brian Ore | Raymond Slyh | Jeremy Gwinnup | Katherine Young | Michael Hutt
Proceedings of the 10th International Workshop on Spoken Language Translation: Evaluation Campaign

This paper describes the MIT-LL/AFRL statistical MT system and the improvements that were developed during the IWSLT 2013 evaluation campaign [1]. As part of these efforts, we experimented with a number of extensions to the standard phrase-based model that improve performance on the Russian to English, Chinese to English, Arabic to English, and English to French TED-talk translation task. We also applied our existing ASR system to the TED-talk lecture ASR task. We discuss the architecture of the MIT-LL/AFRL MT system, improvements over our 2012 system, and experiments we ran during the IWSLT-2013 evaluation. Specifically, we focus on 1) cross-entropy filtering of MT training data, and 2) improved optimization techniques, 3) language modeling, and 4) approximation of out-of-vocabulary words.