Chapter 6 Text-Based Rewriting/Pre-editing Experiment
6.3 Results and analysis
154
Table 6.4 Fluency ranking example in the text-based experiment
Version 1 Version 2
判断划线部分哪个版本更自然流 畅,并从 1 至 4 中进行选择 1. Ver.1 比 Ver.2 更自然流畅 2. Ver.2 比 Ver.1 更自然流畅 3. Ver.1 和 Ver.2 同样自然流畅 4. Ver.1 和 Ver.2 都不自然 结果发现，部分商品中，
有的触犯了电器用品安全 法，有的则错误地标上了
“生态”的标记。
进行深度验证后，结果 发现部分商品中存在与 电器用品安全法相抵触 的商品及被误标环保标 志的商品。
4 结果发现，部分商品中，
有的触犯了电器用品安全 法，有的则错误地标上了
“生态”的标记。
结果发现，部分商品 中，有的触犯了电器用 品安全法，有的则错误 地标上了“生态”的标 记。
-
Step 9. Compare the fluency evaluation and accuracy evaluation results to ascertain the impact of each NTI and the effectiveness of the rewrite rules and thus their readiness for adoption as pre-editing rules. Where there is no change in the translation, enter the evaluation result as “-.” Where there is no translation, therefore, no evaluation, enter the result as “x.”1
155
5.3: a change from “2” to “1” indicates that TT-original was judged to have failed to deliver the intended meaning of the NTI while TT-rewritten was judged to have accurately delivered the meaning of substitution of NTI; a change from “3” to “1” or
“2” indicates that an evaluator who cannot understand the meaning of ST-original can understand that of ST-rewritten and is able to make a sensible judgment; On the other hand, a change from “1” to “2” is regarded as a negative result since TT-original is judged to deliver the intended meaning whereas TT-rewritten is not. Also, where “1”
remains “1” and “2” remains “2,” the result is regarded as neutral, since eliminating an NTI had no impact on the translation quality. Where there is no evaluation result due to the translator not translating the ST because they could not understand the meaning of the ST, it is marked as “x.” Thus, a change from “x” to “1” or “2” indicates a positive result.
As for fluency evaluation, “1,” “2,” and “3” have the same indications as explained in Section 5.3: “1” positive, “2” neutral, and “3”/”4” negative. In addition, where there is no change between TT-original and TT-rewritten, it is excluded from the evaluation, and the result is marked as “-.” Where there is no evaluation result, it is marked as “x.”
156
Table 6.5 Accuracy evaluation results of TT1-o and TT1-r
No. NTI
Translator A Translator B TT-o
Eva1 TT-r Eva1
TT-o Eva2
TT-r Eva2
TT-o Eva1
TT-r Eva1
TT-o Eva2
TT-r Eva2
1 [N1やN2、N3] 1 1 3 1 1 2 2 1
[N2やN2、N3] 2 2 3 1 1 2 1 2
2 [N2やN2] 1 1 1 1 2 1 1 1
[N2やN2] 2 1 3 1 1 2 1 2
[N2やN2] 2 1 3 1 2 1 1 2
3 [N+が+V] 2 1 1 1 2 2 1 2
4 [N1abbreviated to [N1に関するN2] is の N2]
2 1 3 1 2 2 3 2 [N1に関するN2] is
abbreviated to [N1の N2]
2 1 2 1 2 2 2 1 [N1に関するN2] is
abbreviated to [N1の
N2] 2 2 2 1 2 2 1 2
5 [N1abbreviated to [N1+N2] についてのN2] is 2 2 2 1 2 2 2 1 6 [N1stem+N2] のVsahen- 1 2 2 1 1 1 2 1 7 [N1
を含むN2] is abbreviated into [N1の N2]
2 1 3 1 2 2 1 1
8 [NをV1・V2] 2 2 3 1 2 1 1 1
9 [Vinf+N] 2 1 1 1 1 2 1 2
10 [Vsahenを進める] 2 1 1 1 2 2 1 1
11 [によるN] 2 1 2 1 2 2 3 2
12 Centered dot 1 2 3 2 2 2 3 1
Centered dot 1 2 3 1 1 2 3 2
Centered dot 1 1 1 1 2 2 1 1
Centered dot 1 1 3 2 2 1 1 1
Centered dot 2 1 3 1 1 1 3 1
13 End a sentence with
Vsahen 1 2 2 1 1 1 2 2
End a sentence with
Vsahen 2 2 1 1 2 1 3 1
14 Inappropriate style of
acronym 1 1 1 1 2 1 1 1
15 Modifier is too long 2 2 1 1 2 2 2 2 Modifier is too long 2 1 2 1 2 2 1 1 16 Phrase beforeにおいて
is too long 2 1 3 1 2 2 3 2
17 Sentence is too long 2 1 3 1 2 1 3 1 Sentence is too long 2 2 2 1 2 2 3 2 Sentence is too long 2 2 2 1 1 1 1 2 18 Unclear loan word 2 2 1 2 2 2 3 2
19 生かす 2 1 3 1 2 2 2 2
20 イメージ 2 2 3 1 1 1 3 1
21 こうした 1 2 2 1 1 2 1 2
157
22 その 2 1 3 1 2 1 3 1
その 2 1 2 1 2 2 2 1
23 その結果 2 1 1 2 2 2 1 2
24 それ 2 1 2 1 2 2 2 1
25 トータル 2 2 1 2 1 1 1 2
26 取り組む 1 1 2 1 2 1 1 1
27 において 1 1 2 1 2 1 3 1
28 に向けて 2 2 1 1 2 1 1 1
29 考える 2 1 2 1 2 1 1 1
30 取り組み 2 2 1 1 2 1 1 2
31 生かす 2 2 2 1 2 1 1 1
32 対応 2 2 3 1 2 1 3 2
33 地域 2 1 3 1 1 1 3 2
34 適切 2 1 3 1 2 1 3 1
35 努める 1 1 2 1 1 2 1 2
158
Table 6.6 Fluency ranking results of TT1-o and TT1-r
No. NTI Translator A Translator B Eva1 Eva2 Eva3 Eva1 Eva2 Eva3 1 [N2やN2、N3] 2 1 1 1 2 1
[N2やN2、N3] 1 4 2 2 2 2
2 [N2やN2] 1 2 2 - - -
[N2やN2] 1 2 1 2 2 2
[N2やN2] 2 1 2 1 2 2
3 [N+が+V] 2 2 1 2 2 1
4 [N1に関するN2] is
abbreviated to [N1のN2] 2 4 1 1 1 1 [N1に関するN2] is
abbreviated to [N1のN2] 1 2 1 1 2 1 [N1に関するN2] is
abbreviated to [N1のN2] 2 2 2 2 1 1 5 [N1についてのN2] is
abbreviated to [N1+N2] - - - 2 2 1 6 [N1のVsahen-stem+N2] 1 1 1 - - - 7 [N1を含むN2] is
abbreviated into [N1の N2]
2 1 1 - - -
8 [NをV1・V2] 1 1 1 1 1 1
9 [Vinf+N] 1 2 2 - - -
10 [Vsahenを進める] 2 2 2 2 2 1
11 による 1 4 1 1 1 2
12 Centered dot 1 1 1 2 2 1
Centered dot 2 2 1 2 2 1
Centered dot 1 1 1 1 2 1
Centered dot 1 1 2 2 1 1
Centered dot 2 1 1 1 2
13 End a sentence with
Vsahen 2 1 1 2 2 1
End a sentence with
Vsahen 2 2 2 - - -
14 Inappropriate style of
acronym 1 2 1 - - -
15 Modifier is too long 1 2 2 - - - Modifier is too long 1 2 2 - - - 16 Phrase beforeにおいてis
too long 1 1 1 1 2 2
17 Sentence is too long 2 2 1 2 1 2 Sentence is too long 2 2 2 1 1 2 Sentence is too long 1 1 2 1 2 1 18 Unclear loan word 2 1 1 2 1 1
19 生かす 2 2 2 2 1 1
20 イメージ 1 1 1 2 2 1
21 こうした 2 2 1 - - -
22 その 2 2 1 - - -
その 1 1 1 - - -
23 その結果 1 1 1 - - -
24 それ 2 1 1 - - -
25 トータル 1 2 2 2 1 2
159
26 取り組む 2 2 1 1 2 2
27 において - - - -
28 に向けて 2 2 2 - - -
29 考える 2 1 2 - - -
30 取り組み 2 2 2 - - -
31 生かす 1 1 2 - - -
32 対応 2 2 2 - - -
33 地域 1 2 1 1 1 2
34 適切 2 2 1 - - -
35 努める 1 1 1 2 2 2
160
Table 6.7 Accuracy evaluation results of TT2-o and TT2-r
No. NTI
Translator C Translator D TT-o
Eva1
TT-r Eva1
TT-o Eva2
TT-r Eva2
TT-o Eva1
TT-r Eva1
TT-o Eva2
TT-r Eva2
1 [N2やN2、N3] 2 2 3 1 - - - -
2 [N2やN2] 1 1 2 1 1 2 2 2
[N2やN2] 1 1 3 2 2 1 3 2
[N2やN2] 2 1 3 1 1 1 1 2
3 A definition is included
in the modifier 2 2 2 1 1 1 3 1
4 Centered dot 2 2 2 1 1 2 1 1
Centered dot 1 1 3 2 1 2 2 2
Centered dot 1 1 3 1 2 2 3 1
Centered dot 2 2 3 2 2 2 3 2
5 End a sentence with
Vsahen 1 1 1 1 1 2 3 2
End a sentence with
Vsahen 2 2 2 1 2 2 3 1
6 Modifier is too long 2 2 3 1 2 2 3 1 Modifier is too long 2 1 1 2 1 1 2 1 Modifier is too long 2 2 2 1 1 1 3 1 7 Sentence is too long 2 2 2 2 2 2 2 1 Sentence is too long 2 1 3 1 2 2 2 2 Sentence is too long 2 1 1 1 1 1 3 1 Sentence is too long 1 1 2 1 1 2 3 1 8 Unclear loan word x 1 x 1 1 2 3 1 Unclear loan word 2 2 3 1 2 2 3 2
9 活性化 2 1 3 1 1 2 3 2
10 活動 2 2 3 1 2 2 3 1
活動 2 1 2 1 2 2 2 1
11 活用 1 1 3 1 1 1 3 1
12 からの 2 2 3 1 2 2 1 1
13 こうした 2 1 3 1 1 1 3 2
14 対応 1 1 3 1 2 1 3 1
15 地域 2 1 2 1 1 1 1 1
地域 1 1 3 1 1 1 1 1
地域 1 1 3 1 2 2 3 1
地域 2 1 3 1 1 2 3 2
地域 2 2 3 1 2 2 3 1
16 提案 2 2 3 1 2 2 3 2
提案 2 2 3 1 1 2 2 1
17 において 2 2 2 2 1 2 3 2
18 による 2 1 3 1 2 2 3 2
161
Table 6.8 Fluency ranking results of TT2-o and TT2-r
No. NTI Translator C Translator D Eva1 Eva2 Eva3 Eva1 Eva2 Eva3 1 [N2やN2、N3] 1 2 1 - - -
2 [N2やN2] 1 1 2 2 2 2
[N2やN2] 2 2 1 2 2 2
[N2やN2] 2 2 1 2 2 1
3 A definition is included in
the modifier 2 2 2 2 2 1
4 Centered dot - - - 2 2 2
Centered dot - - - 1 2 1
Centered dot 1 2 2 2 2 1
Centered dot 1 1 2 2 2 1
5 End a sentence with Vsahen 2 2 1 - - - End a sentence with Vsahen 1 1 2 1 2 2 6 Modifier is too long 1 1 2 2 1 1 Modifier is too long 2 2 2 2 2 1 Modifier is too long 2 2 2 2 2 2 7 Sentence is too long 2 2 2 2 2 2 Sentence is too long 2 2 2 1 1 2 Sentence is too long 2 1 2 1 2 2 Sentence is too long 2 2 1 2 2 2 8 Unclear loan word - - - 3 2 2
Unclear loan word 2 2 2 1 2 2
9 活性化 2 2 2 - - -
10 活動 2 1 1 2 2 2
活動 2 3 1 2 2 2
11 活用 2 2 2 - - -
12 からの 2 1 1 - - -
13 こうした 2 2 2 - - -
14 対応 1 1 1 2 1 2
15 地域 2 2 2 2 2 2
地域 2 2 2 - - -
地域 1 2 1 3 2 2
地域 2 1 1 - - -
地域 1 2 1 - - -
16 提案 2 2 2 1 2 2
提案 1 2 2 2 2 2
17 において 2 1 2 - - -
18 による 1 1 1 3 2 2
In an attempt to normalize the scores of the accuracy and fluency results and thereby make it possible to compare the impact of pre-editing on a text level, the scores of positive, neutral, and negative results from both accuracy and fluency evaluations are set to “3,” “2,” and “1” respectively, as shown in Table 6.9. Table 6.10 and Table 6.11 show the normalized comparative scores for accuracy and fluency by text.
162
Table 6.9 Normalized score of the accuracy and fluency evaluation results Accuracy result Fluency result Impact Score
2 to 1 3 to 1 3 to 2 x to 1 or 2
2 Positive 3
1 to 1 2 to 2 - to -
3 4 -
Neutral 2
1 to 2 1 Negative 1
Table 6.10 Normalized comparative scores for accuracy and fluency (Text 1)
No. NTI
Accuracy score Fluency score
Translator A Translator B Translator A Translator B Eva1 Eva2 Eva1 Eva2 Eva1 Eva2 Eva3 Eva1 Eva2 Eva3
1 [N2やN2、N3] 2 3 1 3 3 1 1 1 3 1
[N2やN2、N3] 2 3 1 1 1 2 3 3 3 3
2 [N2やN2] 2 2 3 2 1 3 3 2 2 2
[N2やN2] 3 3 1 1 1 3 1 3 3 3
[N2やN2] 3 3 3 1 3 1 3 1 3 3
3 [N+が+V] 3 2 2 1 3 3 1 3 3 1
4
[N1に関するN2] is abbreviated to [N1の N2]
3 3 2 3 3 2 1 1 1 1 [N1に関するN2] is
abbreviated to [N1の N2]
3 3 2 3 1 3 1 1 3 1 [N1に関するN2] is
abbreviated to [N1の N2]
2 3 2 1 3 3 3 3 1 1 5 [N1についてのN2] is
abbreviated to [N1+N2] 2 3 2 3 2 2 2 3 3 1 6 [N1の
Vsahen-stem+N2] 1 3 2 3 1 1 1 2 2 2
7
[N1を含むN2] is abbreviated into [N1の N2]
3 3 2 2 3 1 1 2 2 2
8 [NをV1・V2] 2 3 3 2 1 1 1 1 1 1
9 [Vinf+N] 3 2 1 1 1 3 3 2 2 2
10 [Vsahenを進める] 3 2 2 2 3 3 3 3 3 1
11 による 3 3 2 3 1 2 1 1 1 3
12 Centered dot 1 3 2 3 1 1 1 3 3 1
Centered dot 1 3 1 3 3 3 1 3 3 1
Centered dot 2 2 2 2 1 1 1 1 3 1
Centered dot 2 3 3 2 1 1 3 3 1 1
Centered dot 3 3 2 3 3 1 1 1 2 3
13 End a sentence with
Vsahen 1 3 2 2 3 1 1 3 3 1
End a sentence with
Vsahen 2 2 3 3 3 3 3 2 2 2
163 14 Inappropriate style of
acronym 2 2 3 2 1 3 1 2 2 2
15 Modifier is too long 2 2 2 2 1 3 3 2 2 2 Modifier is too long 3 3 2 2 1 3 3 2 2 2 16 Phrase beforeにおいて
is too long 3 3 2 3 1 1 1 1 3 3
17 Sentence is too long 3 3 3 3 3 3 1 3 1 3 Sentence is too long 2 3 2 3 3 3 3 1 1 3 Sentence is too long 2 3 2 1 1 1 3 1 3 1 18 Unclear loan word 2 1 2 3 3 1 1 3 1 1
19 生かす 3 3 2 2 3 3 3 3 1 1
20 イメージ 2 3 2 3 1 1 1 3 3 1
21 こうした 1 3 1 1 3 3 1 2 2 2
22 その 3 3 3 3 3 3 1 2 2 2
その 3 3 2 3 1 1 1 2 2 2
23 その結果 3 1 2 1 1 1 1 2 2 2
24 それ 3 3 2 3 3 1 1 2 2 2
25 トータル 2 1 2 1 1 3 3 3 1 3
26 取り組む 2 3 3 2 3 3 1 1 3 3
27 において 2 3 3 3 2 2 2 2 2 2
28 に向けて 2 2 3 2 3 3 3 2 2 2
29 考える 3 3 3 2 3 1 3 2 2 2
30 取り組み 2 2 3 1 3 3 3 2 2 2
31 生かす 2 3 3 2 1 1 3 2 2 2
32 対応 2 3 3 3 3 3 3 2 2 2
33 地域 3 3 2 3 1 3 1 1 1 3
34 適切 3 3 3 3 3 3 1 2 2 2
35 努める 2 3 1 1 1 1 1 3 3 3
164
Table 6.11 Normalized comparative scores for accuracy and fluency (Text 2)
No. NTI
Accuracy score Fluency score
Translator C Translator D Translator C Translator D Eva1 Eva2 Eva1 Eva2 Eva1 Eva2 Eva3 Eva1 Eva2 Eva3 1 [N2やN2、
N3]
2 3 2 2 1 3 1 2 2 2
2 [N2やN2] 2 3 1 2 1 1 3 3 3 3
[N2やN2] 2 3 3 3 3 3 1 3 3 3
[N2やN2] 3 3 2 1 3 3 1 3 3 1
3 A definition is included in the modifier
2 3 2 3 3 3 3 3 3 1
4 Centered dot 2 3 1 2 2 2 2 3 3 3
Centered dot 2 3 1 2 2 2 2 1 3 1
Centered dot 2 3 2 3 1 3 3 3 3 1
Centered dot 2 3 2 3 1 1 3 3 3 1
5 End a sentence with Vsahen
2 2 1 3 3 3 1 2 2 2 End a sentence
with Vsahen
2 3 2 3 1 1 3 1 3 3 6 Modifier is too
long
2 3 2 3 1 1 3 3 1 1 Modifier is too
long
3 1 2 3 3 3 3 3 3 1 Modifier is too
long
2 3 2 3 3 3 3 3 3 3 7 Sentence is too
long
2 2 2 3 3 3 3 3 3 3 Sentence is too
long
3 3 2 2 3 3 3 1 1 3 Sentence is too
long
3 2 2 3 3 1 3 1 3 3 Sentence is too
long
2 3 1 3 3 3 1 3 3 3 8 Unclear loan
word
3 3 1 3 2 2 2 2 3 3 Unclear loan
word
2 3 2 3 3 3 3 1 3 3
9 活性化 3 3 1 3 3 3 3 2 2 2
10 活動 2 3 2 3 3 1 1 3 3 3
活動 3 3 2 3 3 2 1 3 3 3
11 活用 2 3 2 3 3 3 3 2 2 2
12 からの 2 3 2 2 3 1 1 2 2 2
13 こうした 3 3 2 3 3 3 3 2 2 2
14 対応 2 3 3 3 1 1 1 3 1 3
15 地域 3 3 2 2 3 3 3 3 3 3
地域 2 3 2 2 3 3 3 2 2 2
地域 2 3 2 3 1 3 1 2 3 3
地域 3 3 1 3 3 1 1 2 2 2
地域 2 3 2 3 1 3 1 2 2 2
16 提案 2 3 2 3 3 3 3 1 3 3
提案 2 3 1 3 1 3 3 3 3 3
17 において 2 2 1 3 3 1 3 2 2 2
18 による 3 3 2 3 1 1 1 2 3 3
165
The scores acquired from Table 6.10 and Table 6.11 are further computed to show the followings in Table 6.12:
the number of “3,” “2,” and “1” by the evaluator
the number of NTI occurrences (N)
the percentage of “3,” “2,” and “1” by the evaluator
Table 6.12 Number and percentage of positive, neutral, and negative results
Text 1 Accuracy Fluency
Translator A Translator B Translator A Translator B Eva1 Eva2 Eva1 Eva2 Eva1 Eva2 Eva3 Eva1 Eva2 Eva3
# of 3 21 36 16 22 24 24 19 16 17 12
# of 2 23 10 26 15 2 5 2 20 21 20
# of 1 5 3 7 12 23 20 28 13 11 17 N 49 49 49 49 49 49 49 49 49 49
% of 3 42.86 73.47 32.65 44.90 48.98 48.98 38.78 32.65 34.69 24.49
% of 2 46.94 20.41 53.06 30.61 4.08 10.20 4.08 40.82 42.86 40.82
% of 1 10.20 6.12 14.29 24.49 46.94 40.82 57.14 26.53 22.45 34.69
Text 2 Accuracy Fluency
Translator C Translator D Translator C Translator D Eva1 Eva2 Eva1 Eva2 Eva1 Eva2 Eva3 Eva1 Eva2 Eva3
# of 3 11 31 2 27 22 21 20 17 23 19
# of 2 25 4 24 8 3 4 3 13 10 10
# of 1 0 1 10 1 11 11 13 6 3 7 N 36 36 36 36 36 36 36 36 36 36
% of 3 30.56 86.11 5.56 75.00 61.11 58.33 55.56 47.22 63.89 52.78
% of 2 69.44 11.11 66.67 22.22 8.33 11.11 8.33 36.11 27.78 27.78
% of 1 0.00 2.78 27.78 2.78 30.56 30.56 36.11 16.67 8.33 19.44
Table 6.13 was produced by averaging the percentage of “3,” “2,” and “1” by the translator. It is observable that substituting NTIs according to pre-defined rewrite rules improves accuracy in 48.89% of cases on average while degrading it in 11.06 % of cases, which is an encouraging result. On the other hand, substituting NTIs according to pre-defined rewrite rules improves fluency in 47.29% of cases on average, while degrading it in 30.85 % of cases, thus showing no significant improvement in fluency.
It is worth noting that the differences in the evaluations of individual translators are consistent.
166
Table 6.13 Average percentage of positive, neutral, and negative results
Type Ave. % of scores
Text 1 Text 2 Ave.
Translator A Translator B Translator C Translator D Accuracy Ave. % of 3 58.16 38.78 58.33 40.28 48.89
Ave. % of 2 33.67 41.84 40.28 44.44 40.06 Ave. % of 1 8.16 19.39 1.39 15.28 11.06 Fluency Ave. % of 3 45.58 30.61 58.33 54.63 47.29 Ave. % of 2 6.12 41.50 9.26 30.56 21.86 Ave. % of 1 48.30 27.89 32.41 14.81 30.85
Looking back to the results of the corpus-based experiment in Chapter 5, Table 5.9 indicates that the test NTI set improves accuracy in 39.64% of cases while degrading it in only 9.87% of cases. It also shows that the test NTI set improves and degrades fluency to a very similar extent, i.e., it has no clear positive or negative impact.
The results of the corpus-based experiment and those of the text-based experiment are consistent.