Experimental setup: task design and implementation

130

the experiment since the nature of the errors is not important. The key point is that the results can be used to diagnose whether avoiding an NTI can improve accuracy and/or fluency.

131

Table 5.2 38 NTIs tested in this experiment

No. Category NTI

01 Lexical - Vocabulary Usage 支援 02 Lexical - Vocabulary Usage 反映 03 Lexical - Vocabulary Usage 発信 04 Lexical - Vocabulary Usage 評価 05 Lexical - Vocabulary Usage 貢献 06 Lexical - Vocabulary Usage 活用 07 Lexical - Vocabulary Usage 追求 08 Lexical - Vocabulary Usage 配信 09 Lexical - Vocabulary Usage 決定 10 Lexical - Vocabulary Usage 独自 11 Lexical - Vocabulary Usage 最適 12 Lexical - Vocabulary Usage 確実 13 Lexical - Vocabulary Usage 現場 14 Lexical - Vocabulary Usage 考え 15 Lexical - Vocabulary Usage イメージ 16 Lexical - Vocabulary Usage ダイバーシティ 17 Lexical - Vocabulary Usage モビリティ 18 Lexical - Vocabulary Usage ニーズ 19 Lexical - Vocabulary Usage トータル 20 Lexical - Prefix/Suffix Usage Use suffix “化”

21 Syntactic - Coordination N1やN2、N3 22 Syntactic - Coordination N1やN2などのN3 23 Syntactic pattern …し、…するため 24 Syntactic pattern NのVsahen-stemを行う 25 Compound particle usage N1に向けたN2

26 Syntactic pattern N1、N2、N3のNo.+Counter (*No.>3) 27 Syntactic pattern Use [Vinf+N] to indicate potential 28 Syntactic pattern Vsahen-stemを実現する

29 Syntactic pattern Vsahen-stemが進められる

30 Syntactic pattern NのV1sahen-stem・V2sahen-stem 31 Compound particle usage に向けて

32 Syntactic pattern Vsahen-stemを検討する 33 Syntactic - Punctuation centered dot (・)

34 Syntactic - Punctuation full-width slash (／)

35 Syntactic - Ellipsis Use [N+『』/「」] to indicate the reference 36 Textual - Information Load Phrase beforeとしてis too long (Limit: 20) 37 Textual - Information Load Phrase beforeを目的とするis too long (Limit: 20) 38 Textual - Information Load Phrase beforeにおいてis too long (Limit: 20)

132

Testing the effectiveness of eliminating the NTIs requires comparing the translations of sentences containing NTIs (ST-original) with the translations of the corresponding sentences in which the NTIs have been rewritten (ST-rewritten). The question then arises whether the two versions of the same sentence should be produced by the same or different translators. If it is the same translator, then there is a risk of sensitization (Saldanha & O’Brien, 2014, p. 31) in that the previous translation may be recalled consciously or unconsciously. If it is a different translator, then—given the acknowledged multiplicity of valid translations for a given source—there is a risk that the idiosyncracies of individual translators may cloud the issue. It was decided that the first option presented the lesser risk. An interval of one month was left between the two rounds of translation, admittedly less than the three month gap generally suggested, although Saldanha & O’Brien (2014, p. 160) observe “we should also bear in mind that opinions can actually change over such time frames.” In fact, in WMT (Workshop on Statistical Machine Translation) events held over only a few days it was observed that

“intra-annotator agreement (i.e., the agreement of raters with themselves when faced with the same assessment task multiple times) did not exceed 0.65” (Bojar et al., 2013, pp. 6-8). Using the corpus described in Section 5.1, the following steps were taken to establish what type of translation errors can be induced by the NTIs (RQ5), and whether rewriting the NTIs improves translation quality (RQ6).

Step 1. Extract 20 random samples for each NTI in Table 5.2 using a CQL or a regular expression formulated in Section 5.1, and pick the first four valid samples.

Step 2. Divide the samples into two sets (Set A and Set B, each containing two different samples of the same NTI), send Set A to Translators A and B, and Set B to Translators C and D for translation into ZH-CN. This yields the set of TTs derived from the original STs (with NTIs), henceforth “TT-original.”

133

Step 3. Work with the rewriter o rewrite the samples applying the rewrite rules developed in Chapter 4. The principle of rewriting/pre-editing is to disambiguate the ST by substituting the NTI with an alternative expression while preserving the intended meaning as much as possible. Since only the NTI for testing is to be modified, any other NTI contained in the same sample is left as is.

Step 4. Send the rewritten samples in Set A to Translators A and B and those in Set B to Translators C and D for translation. This yields the set of TTs derived from the rewritten STs (without NTIs), henceforth “TT-rewritten.” Table 5.5 illustrates the translation assignment in the text-based experiment: the 1^st and 2^nd samples (both ST-original and ST-rewritten) go to Translators A and B, and the 3^rd and 4^th samples (both ST-original and ST-rewritten) go to Translators C and D. The use of CAT tools is not allowed to prevent translators from referring to TT-original when translating ST-rewritten, and specific instructions not to refer to prior work are also given. In the case that translators decide to go against the instructions and refer to TT-original when translating ST-written, they will have to identify where the change is for each sample, which is extremely difficult because the order of the samples is randomized and aligning original and ST-written would be too time-consuming.

Step 5. Perform accuracy evaluation by asking two ZH-CN native speakers—

professional translators previously employed in Chapter 3 as assessors—to evaluate separately whether the translation of the NTI and the translation of the pre-edited text deliver the intended meaning. As shown in Table 5.3, three options were available to the evaluators: “1” if the translation is accurate with respect to the highlighted source span; “2” if the translation is not accurate with respect to the highlighted source span; and “3” if it is not possible to decide whether the translation is accurate because the meaning of the source span is not clear. The evaluators were given ST-original or ST-rewritten in the “Japanese”

134

column, TT-original or TT-rewritten in the “Chinese translation” column, and evaluation instructions in ZH-CN in an Excel spreadsheet. The evaluators were requested to enter their scores into the spreadsheets. See Table 5.3 for sample results.

Table 5.3 Accuracy evaluation interface

Japanese Chinese translation

根据上下文判断划线部分 能否传达原文想要表达的 意思

1.能传达 2.不能传达

3.无法判断，因为原文的 意思不明确或无法理解 教育格差是正を支援する教

材提供プロジェクト

支援缩小教育差距的 免费提供教材项目

2 当社は、公益財団法人日本

経済研究センターが主催す る日経 GSR 研究会のプログ ラムの一つである「日経 GSR 学生アイデア・コンテ スト」を第 1 回から支援し ています。

该公司从第一届开始 就一直支援“日经 GSR 学生创意大赛”活 动，这是由公益财团 法人日本经济研究中 心举办的日经 GSR 研 究会的项目之一。

2

Step 6. Perform fluency ranking evaluation (as shown in Table 5.4) by asking three ZH-CN native speakers—who majored in ZH-CN or English and are currently working as high school teachers in China—to compare TT-original and TT-rewritten, focusing on NTI related spans. Four options were given to the evaluators: “1” if version 1 is more fluent than version 2; “2” if version 2 is more fluent than version 1; “3” if version 1 and version 2 are equally fluent; and “4”

if version 1 and version 2 are equally disfluent/unnatural. As shown in Table 5.4, the evaluators were given “Version 1,” “Version 2,” and evaluation instructions in ZH-CN in an Excel spreadsheet. The evaluators were blind as to whether a given translation was TT-original or TT-rewritten as both “Version 1” and

“Version 2” were a mix of TT-original and TT- rewritten, the key to their distribution being known only to the author of the present study. The evaluators

135

were requested to enter their scores into the spreadsheets (see Table 5.4). After the evaluation, the key was used to reassign these scores such that “1” means TT-original is more fluent than TT-rewritten, “2” means TT-rewritten is more fluent than TT-original, “3” means the two are equally fluent, and “4” means the two are equally disfluent/unnatural.

Step 7. Compare the fluency evaluation and accuracy evaluation results to ascertain the impact of each NTI and the effectiveness of the rewrite rules and thus their readiness for adoption as pre-editing rules.

Table 5.4 Fluency ranking evaluation interface

Version 1 Version 2

判断划线部分哪个版本自然流畅，

并从 1 至 4 中进行选择。

1. Ver.1 比 Ver.2 更自然流畅 2. Ver.2 比 Ver.1 更自然流畅 3. Ver.1 和 Ver.2 同样自然流畅 4. Ver.1 和 Ver.2 都不自然 支援缩小教育差距的

免费提供教材项目

改 善 教 育 差 距 的 教 材免费提供项目

4

该公司从第一届开始 就一直支援“日经 GSR 学生创意大赛”活动，

这是由公益财团法人 日本经济研究中心举 办的日经 GSR 研究会 的项目之一。

本公司自第一届“日 经 GSR 学生创意大 赛 ” 就 开 始 提 供 赞 助，这是由公益财团 法 人 日 本 经 济 研 究 中心主办的日经 GSR 研究会项目之一。

2