[PDF][PDF] A lightweight evaluation framework for machine translation reordering

D Talbot, H Kazawa, H Ichikawa… - Proceedings of the …, 2011 - aclanthology.org
D Talbot, H Kazawa, H Ichikawa, J Katz-Brown, M Seno, FJ Och
Proceedings of the Sixth Workshop on Statistical Machine Translation, 2011aclanthology.org
Reordering is a major challenge for machine translation between distant languages. Recent
work has shown that evaluation metrics that explicitly account for target language word order
correlate better with human judgments of translation quality. Here we present a simple
framework for evaluating word order independently of lexical choice by comparing the
system's reordering of a source sentence to reference reordering data generated from
manually word-aligned translations. When used to evaluate a system that performs …
Abstract
Reordering is a major challenge for machine translation between distant languages. Recent work has shown that evaluation metrics that explicitly account for target language word order correlate better with human judgments of translation quality. Here we present a simple framework for evaluating word order independently of lexical choice by comparing the system’s reordering of a source sentence to reference reordering data generated from manually word-aligned translations. When used to evaluate a system that performs reordering as a preprocessing step our framework allows the parser and reordering rules to be evaluated extremely quickly without time-consuming endto-end machine translation experiments. A novelty of our approach is that the translations used to generate the reordering reference data are generated in an alignment-oriented fashion. We show that how the alignments are generated can significantly effect the robustness of the evaluation. We also outline some ways in which this framework has allowed our group to analyze reordering errors for English to Japanese machine translation.
aclanthology.org
Showing the best result for this search. See all results