Evaluating the translation quality and post-editing efficiency of targoman translator

Abstract

With the increasing reliance on machine translation (MT) systems, assessing their linguistic accuracy and practical utility has become critical. In Iran, Targoman Translator is a significant local MT project that requires a critical evaluation of its effectiveness in comparison to emerging AI tools such as ChatGPT. The purpose of this study was to critically evaluate the output quality of Targoman Translator, a domestic MT tool, using Wilss’ (1982) matrix, with a focus on syntax, semantics and pragmatics. ChatGPT was also used to establish whether Targoman Translator’s translations needed post-editing and whether it might serve as a dependable AI tool for end-users, perhaps replacing human editors. The study included a translation test of 80 statements from Mistrík’s (1997) taxonomy of text types, narrative, descriptive, and argumentative, along with idiomatic expressions added by the researcher. Targoman Translator initially translated the statements from English to Persian. Evaluators were provided both the original English and Persian translations, and they assessed the translation quality using a Likert-scale questionnaire. The findings revealed that Targoman performed well grammatically on most statements and was moderately successful semantically. However, it regularly mistranslated idiomatic statements, demonstrating poor performance because it was unable to identify functional dependencies, Additionally, the study found that using ChatGPT for post-editing greatly increased translation accuracy. ChatGPT produced translations that were more in line with the intended meaning by fixing grammatical mistakes and clarifying meaning. Because of this, ChatGPT can be regarded as a strong and trustworthy AI editing tool that is advantageous to end users.

Keywords: Translation Quality, Post-Editing, Targoman Translator, Wilss’ matrix of transformation/evaluation