John Makhoul

Also published as: J. Makhoul


2020

pdf bib
Reformulating Information Retrieval from Speech and Text as a Detection Problem
Damianos Karakos | Rabih Zbib | William Hartmann | Richard Schwartz | John Makhoul
Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS2020)

In the IARPA MATERIAL program, information retrieval (IR) is treated as a hard detection problem; the system has to output a single global ranking over all queries, and apply a hard threshold on this global list to come up with all the hypothesized relevant documents. This means that how queries are ranked relative to each other can have a dramatic impact on performance. In this paper, we study such a performance measure, the Average Query Weighted Value (AQWV), which is a combination of miss and false alarm rates. AQWV requires that the same detection threshold is applied to all queries. Hence, detection scores of different queries should be comparable, and, to do that, a score normalization technique (commonly used in keyword spotting from speech) should be used. We describe unsupervised methods for score normalization, which are borrowed from the speech field and adapted accordingly for IR, and demonstrate that they greatly improve AQWV on the task of cross-language information retrieval (CLIR), on three low-resource languages used in MATERIAL. We also present a novel supervised score normalization approach which gives additional gains.

pdf bib
The 2019 BBN Cross-lingual Information Retrieval System
Le Zhang | Damianos Karakos | William Hartmann | Manaj Srivastava | Lee Tarlin | David Akodes | Sanjay Krishna Gouda | Numra Bathool | Lingjun Zhao | Zhuolin Jiang | Richard Schwartz | John Makhoul
Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS2020)

In this paper, we describe a cross-lingual information retrieval (CLIR) system that, given a query in English, and a set of audio and text documents in a foreign language, can return a scored list of relevant documents, and present findings in a summary form in English. Foreign audio documents are first transcribed by a state-of-the-art pretrained multilingual speech recognition model that is finetuned to the target language. For text documents, we use multiple multilingual neural machine translation (MT) models to achieve good translation results, especially for low/medium resource languages. The processed documents and queries are then scored using a probabilistic CLIR model that makes use of the probability of translation from GIZA translation tables and scores from a Neural Network Lexical Translation Model (NNLTM). Additionally, advanced score normalization, combination, and thresholding schemes are employed to maximize the Average Query Weighted Value (AQWV) scores. The CLIR output, together with multiple translation renderings, are selected and translated into English snippets via a summarization model. Our turnkey system is language agnostic and can be quickly trained for a new low-resource language in few days.

pdf bib
What Set of Documents to Present to an Analyst?
Richard Schwartz | John Makhoul | Lee Tarlin | Damianos Karakos
Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS2020)

We describe the human triage scenario envisioned in the Cross-Lingual Information Retrieval (CLIR) problem of the [REDUCT] Program. The overall goal is to maximize the quality of the set of documents that is given to a bilingual analyst, as measured by the AQWV score. The initial set of source documents that are retrieved by the CLIR system is summarized in English and presented to human judges who attempt to remove the irrelevant documents (false alarms); the resulting documents are then presented to the analyst. First, we describe the AQWV performance measure and show that, in our experience, if the acceptance threshold of the CLIR component has been optimized to maximize AQWV, the loss in AQWV due to false alarms is relatively constant across many conditions, which also limits the possible gain that can be achieved by any post filter (such as human judgments) that removes false alarms. Second, we analyze the likely benefits for the triage operation as a function of the initial CLIR AQWV score and the ability of the human judges to remove false alarms without removing relevant documents. Third, we demonstrate that we can increase the benefit for human judgments by combining the human judgment scores with the original document scores returned by the automatic CLIR system.

2015

pdf bib
Statistical Machine Translation Features with Multitask Tensor Networks
Hendra Setiawan | Zhongqiang Huang | Jacob Devlin | Thomas Lamar | Rabih Zbib | Richard Schwartz | John Makhoul
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

2014

pdf bib
Fast and Robust Neural Network Joint Models for Statistical Machine Translation
Jacob Devlin | Rabih Zbib | Zhongqiang Huang | Thomas Lamar | Richard Schwartz | John Makhoul
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf bib
Anticipatory translation model adaptation for bilingual conversations
Sanjika Hewavitharana | Dennis Mehay | Sankaranarayanan Ananthakrishnan | Rohit Kumar | John Makhoul
Proceedings of the 11th International Workshop on Spoken Language Translation: Papers

Conversational spoken language translation (CSLT) systems facilitate bilingual conversations in which the two participants speak different languages. Bilingual conversations provide additional contextual information that can be used to improve the underlying machine translation system. In this paper, we describe a novel translation model adaptation method that anticipates a participant’s response in the target language, based on his counterpart’s prior turn in the source language. Our proposed strategy uses the source language utterance to perform cross-language retrieval on a large corpus of bilingual conversations in order to obtain a set of potentially relevant target responses. The responses retrieved are used to bias translation choices towards anticipated responses. On an Iraqi-to-English CSLT task, our method achieves a significant improvement over the baseline system in terms of BLEU, TER and METEOR metrics.

2013

pdf bib
Systematic Comparison of Professional and Crowdsourced Reference Translations for Machine Translation
Rabih Zbib | Gretchen Markiewicz | Spyros Matsoukas | Richard Schwartz | John Makhoul
Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

2012

pdf bib
Machine Translation of Arabic Dialects
Rabih Zbib | Erika Malchiodi | Jacob Devlin | David Stallard | Spyros Matsoukas | Richard Schwartz | John Makhoul | Omar F. Zaidan | Chris Callison-Burch
Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

2010

pdf bib
Decision Trees for Lexical Smoothing in Statistical Machine Translation
Rabih Zbib | Spyros Matsoukas | Richard Schwartz | John Makhoul
Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR

2006

pdf bib
A Study of Translation Edit Rate with Targeted Human Annotation
Matthew Snover | Bonnie Dorr | Rich Schwartz | Linnea Micciulla | John Makhoul
Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers

We examine a new, intuitive measure for evaluating machine-translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments. Translation Edit Rate (TER) measures the amount of editing that a human would have to perform to change a system output so it exactly matches a reference translation. We show that the single-reference variant of TER correlates as well with human judgments of MT quality as the four-reference variant of BLEU. We also define a human-targeted TER (or HTER) and show that it yields higher correlations with human judgments than BLEU—even when BLEU is given human-targeted references. Our results indicate that HTER correlates with human judgments better than HMETEOR and that the four-reference variants of TER and HTER correlate with human judgments as well as—or better than—a second human judgment does.

1994

pdf bib
On Using Written Language Training Data for Spoken Language Modeling
R. Schwartz | L. Nguyen | F. Kubala | G. Chou | G. Zavaliagkos | J. Makhoul
Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994

pdf bib
Adaptation to New Microphones Using Tied-Mixture Normalization
Anastasios Anastasakos | Francis Kubala | John Makhoul | Richard Schwartz
Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994

pdf bib
On-Line Cursive Handwriting Recognition Using Hidden Markov Models and Statistical Grammars
John Makhoul | Thad Starner | Richard Schwartz | George Chou
Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994

pdf bib
Robust Continuous Speech Recognition
John Makhoul | Richard Schwartz
Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994

pdf bib
Usable, Real-Time, Interactive Spoken Language Systems
John Makhoul | Madeleine Bates
Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994

1993

pdf bib
Comparative Experiments on Large Vocabulary Speech Recognition
Richard Schwartz | Tasos Anastasakos | Francis Kubala | John Makhoul | Long Nguyen | George Zavaliagkos
Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993

pdf bib
Robust Continuous Speech Recognition
John Makhoul | Richard Schwartz
Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993

pdf bib
Usable, Real-Time, Interactive Spoken Language Systems
John Makhoul | Madeleine Bates
Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993

1992

pdf bib
Session 3: Spoken Language Systems III
John Makhoul
Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992

pdf bib
BBN BYBLOS and HARC February 1992 ATIS Benchmark Results
Francis Kubala | Chris Barry | Madeleine Bates | Robert Bobrow | Pascale Fung | Robert Ingria | John Makhoul | Long Nguyen | Richard Schwartz | David Stallard
Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992

pdf bib
Improving State-of-the-Art Continuous Speech Recognition Systems Using the N-Best Paradigm with Neural Networks
S. Austin | G. Zavaliagkos | J. Makhoul | R. Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992

pdf bib
BBN Real-Time Speech Recognition Demonstrations
Steve Austin | Rusty Bobrow | Dan Ellard | Robert Ingria | John Makhoul | Long Nguyen | Pat Peterson | Paul Placeway | Richard Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992

pdf bib
Development of a Spoken Language System
John Makhoul | Madeleine Bates
Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992

pdf bib
Robust Continuous Speech Recognition
John Makhoul | Richard Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992

1991

pdf bib
BYBLOS Speech Recognition Benchmark Results
F. Kubala | S. Austin | C. Barry | J. Makhoul | P. Placeway | R. Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991

pdf bib
BBN HARC and DELPHI Results on the ATIS Benchmarks - February 1991
S. Austin | D. Ayuso | M. Bates | R. Bobrow | R. Ingria | J. Makhoul | P. Placeway | R. Schwartz | D. Stallard
Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991

pdf bib
Continuous Speech Recognition Using Segmental Neural Nets
S. Austin | J. Makhoul | R. Schwartz | G. Zavaliagkos
Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991

pdf bib
Research in Continuous Speech Recognition
John Makhoul | Richard Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991

pdf bib
Spoken Language Systems
John Makhoul
Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991

1990

pdf bib
Developing an Evaluation Methodology for Spoken Language Systems
Madeleine Bates | Sean Boisen | John Makhoul
Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990

pdf bib
Research in Continuous Speech Recognition
John Makhoul | Richard Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990

pdf bib
Spoken Language Systems
John Makhoul
Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990

1989

pdf bib
Research in Continuous Speech Recognition
John Makhoul | Richard Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Philadelphia, Pennsylvania, February 21-23, 1989

pdf bib
The BBN BYBLOS Continuous Speech Recognition System
Richard Schwartz | Chris Barry | Yen-Lu Chow | Alan Deft | Ming-Whei Feng | Owen Kimball | Francis Kubala | John Makhoul | Jeffrey Vandegrift
Speech and Natural Language: Proceedings of a Workshop Held at Philadelphia, Pennsylvania, February 21-23, 1989

pdf bib
Speaker Adaptation from Limited Training in the BBN BYBLOS Speech Recognition System
Francis Kubala | Ming-Whei Feng | John Makhoul | Richard Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Philadelphia, Pennsylvania, February 21-23, 1989

pdf bib
Improved HMM Models for High Performance Speech Recognition
Steve Austin | Chris Barry | Yen-Lu Chow | Man Derr | Owen Kimball | Francis Kubala | John Makhoul | Paul Placeway | William Russell | Richard Schwartz | George Yu
Speech and Natural Language: Proceedings of a Workshop Held at Cape Cod, Massachusetts, October 15-18, 1989

pdf bib
Automatic Detection Of New Words In A Large Vocabulary Continuous Speech Recognition System
Ayman Asadi | Richard Schwartz | John Makhoul
Speech and Natural Language: Proceedings of a Workshop Held at Cape Cod, Massachusetts, October 15-18, 1989

pdf bib
Research in Continuous Speech Recognition
John Makhoul | Richard Schwartz
Speech and Natural Language: Proceedings of a Workshop Held at Cape Cod, Massachusetts, October 15-18, 1989

pdf bib
Spoken Language Systems
John Makhoul
Speech and Natural Language: Proceedings of a Workshop Held at Cape Cod, Massachusetts, October 15-18, 1989

pdf bib
White Paper on Spoken Language Systems
John Makhoul | Fred Jelinek | Larry Rabiner | Clifford Weinstein | Victor Zue
Speech and Natural Language: Proceedings of a Workshop Held at Cape Cod, Massachusetts, October 15-18, 1989