Writings
journal or
publication title
Hoshi journal of general education
number 29
page range 1‑9
year 2011
URL http://id.nii.ac.jp/1240/00000254/
Analyzing Multiple Reviewers’ Comments on EFL Writings
Sayuri Yoshizawa
(Hoshi University, School of Pharmacy and Pharmaceutical Sciences)
1. Introduction
Many researchers in the United States and some Asian countries have stated in their papers that the peer review method is quite useful for improving writings. [1]-[12]. The application of the peer review method to the second language acquisition for foreigners has been increasing in Japan [13],[14]. Although the method is validated as a useful training tool to acquire the skills to create documents easily comprehensible for readers, many Japanese teachers seem reluctant to adopt this method when they teach English to their Japanese students. The author has been doing research on the effectiveness of the peer review method as well as the method using the normalized compression distance (NCD) [15]-[22]. The NCD, a compression-based classification algorithm, was first formulated by Cilibrasi and Vitányi [23] on the basis of the normalized information distance (NID) proposed by Li et al [24] and has been used to cluster many kinds of similar objects.
In this experiment, the author examined how the number of reviewer comments affects the decision of writers upon their essay revision by comparing the comments made by one, two or three reviewers with similar English proficiency levels. The reviewers were asked to indicate surface errors, such as spelling, grammar, and syntax errors. They were also told to make some comments on the organization of the essay to see their logical thinking skills. This paper shows that the educational effect the multiple reviewers mutually give during the peer review activity resulted in reaching a higher achievement.
The following section shows the method of this experiment using the
peer review method. The results with the different number of reviewers are presented and discussed in the third section. The conclusions of this experiment and the future research are stated in the final section.
2. Method of the experiment
The effects of the peer review method with two or more reviewers were examined in English as a foreign language (EFL) composition classes.
The peer evaluations on the development and the organization of the essay were also examined in this research.
2.1 Subjects
The subjects were 61 third year students (26 men, 35 women) who enrolled in the six-year course at a pharmaceutical university in Tokyo, Japan. They were all educated in Japan and had received at least eight years of English instruction in Japan. Only two of them had experience living abroad in their earliest days. All subjects were first instructed on how to make comments on the peer’s essay by using a sample essay in which the instructor intentionally placed some grammatical errors as well as illogical development. They were told not to criticize the peer’s opinion itself. They had undergone peer review activities at least three times during the first half of the course.
Two female students did not attend the second half of the peer review activity, and one male student did not agree to use his data for the purpose of this research. Thus, the number of data used for this analysis came out to be 58.
2.2 Procedures
At the beginning of the class the students were divided into two groups
and given different subjects to write a persuasive essay. One week later
they underwent the peer review activity and revised the essay using their
peers’ comments after another week. They were told to make comments
on any peer suggestion that they did not refer to including why they
did not accept the peer suggestion. The author examined what kind of
comments or indication had influence on the essay revision, and whether
or not the comments made by multiple reviewers would have more impact on their decision than those made by a single reviewer when revising the essays.
2.2.1 Procedure 1
The students were divided into two groups (Group A and Group B), and different composition subjects were provided to each group so that the reviewers would not have any preliminary knowledge or reflect their own opinions when they make comments on the essays. The subject given to Group A related to gastric ulcer; if your patient, who has a strong possibility of having gastric ulcer, is reluctant to be examined because this patient believes any examination is meaningless, how would you persuade him/her to undergo the Helicobacter pylori examination? The subject for Group B related to high blood pressure; if your patient, who has a high blood pressure, refuses to take medicines because this patient believes that taking any medicine shortens his/her life, how would you persuade him/her to take medications to lower the blood pressure? The activity was done in a computer room. They were allowed to use both Japanese- English and English-Japanese electronic dictionaries. However, they were not permitted to use any online translation tools. The provided time for writing an essay was 65 minutes, and the expected number of words was set to 200 words. The instructor advised them to use the Microsoft Word spell checker function before submitting the essay.
Moreover, in order to eliminate any advantage or disadvantage caused by the difference in their knowledge on the subjects, a set of data on each subject was distributed to them. Gathering information on the subjects using the web search engines was prohibited so as to eliminate any inequity caused by the difference in ability to find information helpful for essay composition.
2.2.2 Procedure 2
One week later they underwent the peer review activity in the same
computer room so that the activity could be done anonymously. Each
student made comments on the three essays that were distributed by the
instructor. They received essays on the different subject from that of his/
her own essay; that is, Group A commented on the Group B essays, and vice versa. The reviewers were advised to read the essay in a position of the very patient upon deciding whether the essay was persuasive enough to change the patient’s mind. The Microsoft Word track change function was used to make comments and suggestions. The instructor collected all the essays with peer comments at the end of the class. Assuming that it would take about 20 minutes for the reviewer to comment on each essay, 60 minutes was provided for this activity.
2.2.3 Procedure 3
After one week from the procedure 2, they revised the essay using the set of comments made by the anonymous three reviewers. This activity was conducted in a regular classroom and the provided time for this revision was 50 minutes.
Furthermore, they were told to make comments on any peer suggestion that they did not refer to when revising the essay, such as the reasons why they did not accept the peer suggestion. This opportunity was provided because the previous participants asked for some kind of activity to express their thoughts and to bring forth their counterargument to the reviewers’ comments [18]. The author expected that this opportunity would help to find out whether the writer had unconditionally accepted the peer suggestion and rejected the peer comments for decent reasons, etc.
3. Results and discussion
The analysis was conducted from the perspective of the necessity for the actual revision that the instructor assessed to see the educational effects the peer would give to each other. It was expected that this activity would raise their awareness and develop the ability to critically read their own essays.
The average number of words in the first draft was 126 for Group A
(gastric ulcer) and 114 for Group B (high blood pressure). The average
number of words in the second draft was 164 for Group A and 131 for
Group B. As for Group A, the total number of comments and suggestions
the reviewers gave to the first draft was 188. The number of comments
on surface errors such as spelling, grammar, and syntax, was 116, and the number of suggestions referring to the contents was 72. The suggestions included comments on the logicality of the contents and helpful advice upon improving the essay. As for Group B, the total number of comments and suggestions the reviewers gave to the first draft was 284. The number of comments on surface errors was 198, and the number of suggestions referring to the contents was 86. The minimum number of total comments that all three reviewers indicated to one composition was 7, the maximum number was 34, and the average number was 13.
As for Group A, the number of comments on surface errors one writer received ranged from 2 to 24. The number of suggestions on the adequacy and organization of the content ranged from 2 to 12. As for Group B, the number of comments on surface errors ranged from 3 to 32, and the number of suggestions on content ranged from 2 to 13.
Table 1 shows that multiple reviewers’ comments were more easily adopted by the writers in revising the essays. Although the number of the cases that the multiple reviewers made similar comments was limited, it shows that the authors tend to adopt more comments when multiple reviewers made much the same comments.
Table 2 shows the number of improper comments or suggestions by the number of reviewers.
Table 2 # of reviewers and improper comments or suggestions Number of reviewers
Type of comments 1 2 3
Surface (spelling, grammar, syntax errors) 8.6%
(20/233)
7.4%
(2/27)
8.3%
(1/12) Content (logical development, suggestions
for improvement)
9.2%
(9/98)
0.0%
(0/18)
0.0%
(0/5) Table 1 # of reviewers and comments adopted upon revision
Number of reviewers
Type of comments 1 2 3
Surface (spelling, grammar, syntax errors) 75.1%
(175/233)
92.6%
(25/27)
91.7%
(11/12) Content (logical development, suggestions
for improvement)
68.4%
(67/98)
94.4%
(17/18)
100%
(5/5)
Because commenting on surface errors is easier for the student reviewers, they provided a great number of comments, but more improper such comments were recognized than the comments focusing on contents. The lack of reviewer's grammar knowledge brought about this result. Those reviewers were the ones that did not do well on a sample grammar test at the beginning of the course. Although there were some improper comments on contents, all suggestions shared by multiple reviewers were proper ones.
The author of this paper assessed the adequacy of each comment. Four writers adopted improper surface comments and another three revised the essay using improper suggestions on the organization of the content.
Although there was one case that all three reviewers gave improper comments on the same grammatical point, the writer did not accept it. This student had a high score on a sample grammar test. However, comments and suggestions were mostly appropriate when the multiple reviewers commented on the same thing. Providing the writers with an opportunity to state the reasons why they accepted or ignored the suggestions made them rethink about their own statements. Overall, comments and suggestions turned out more adequate than those by a single reviewer.
4. Conclusions and future research
In this experiment, the author examined how the number of reviewer comments affects the decision of writers upon their essay revision. By performing reviews on essays whose subject was different from the one the reviewer had written on, the reviewers consequently provided unbiased objective comments that became very useful for the writers in revising their essays.
Although about ten percent of indication was found inappropriate due
to the lack of the reviewers’ English knowledge, it did not really affect
the writer’s decision for essay revision because of the help of other
reviewers’ appropriate comments. It was also found that when two or
more reviewers point out the same or similar error regarding contents,
there is no inappropriateness in the indication. Moreover, reviewing
anonymously eliminated diffidence and stimulated the reviewers to write candid opinions. The opportunity to state the reasons for the decisions on the suggestions, which the writers did not accept, helped them to critically read their own statements. Therefore, conducting the anonymous peer review with multiple reviewers who have not written an essay on the same subject is a very effective classroom activity.
The author is now analyzing the results of the experiment conducted with a new peer review method using a problem checklist that is expected to assist the composition revision, which will help establish an objective measurement for more effective peer review activities.
References
[1] Berg, E.C. (1999). The effects of trained peer response on ESL students’ revision types and writing quality. Journal of Second Language Writing, 8(3), 215-241.
[2] Carlson, J.G. & Nelson, G. L. (1996). Chinese students’ perceptions of ESL peer response group interaction. Journal of Second Language Writing, 5(1), 1-19.
[3] Chaudron, C. (1984). The effects of feedback on students’
composition revisions. RELC Journal, 15(2), 1-14.
[4] Connor, U. & Asenavage, K. (1994). Peer response groups in ESL writing classes: How much impact on revision? Journal of Second Language Writing, 3 (3), 257-276
[5] Liu, J. & Hansen, J.G. (2002). Peer Response in Second Language Writing Classrooms. Ann Arbor, MI: The University of Michigan Press.
[6] Mendonça, C. & Johnson, K. E. (1994). Peer review negotiations:
Revision activities in ESL writing instruction.” TESOL Quarterly, 28 (4), 745-69.
[7] Min, H.T. (2006). The effects of trained review on EFL students’
revision types and writing quality. Journal of Second Language Writing, 15, 118-141.
[8] Mittan, R. (1989). The peer review process: harnessing students’
communicative power. In D. Johnson & Roen, D. (Eds.), Richness in
Writing: Empowering ESL Students, 207-219, New York: Longman.
[9] Nelson, G. & Murphy, J. P. (1993). Do L2 writers use peer comments in revising their drafts?” TESOL Quarterly, 27(1), 135-142.
[10] Rollinson, P. (2005). Using peer feedback in the ESL writing class.
ELT Journal, 59(1), 23-30.
[11] Tsui, A.B.M. & Ng, M. (2000). Do secondary L2 writers benefit from peer comments? Journal of Second Language Writing, 9(2), 147-170.
[12] Zhang, S. (1995). Re-examining the affective advantage of peer feedback in the ESL writing class. Journal of Second Language Writing, 4(3), 209-222.
[13]
池田玲子・舘岡洋子(2007)
．『ピア・ラーニング入門－創造的な学びのデザインのために』東京：ひつじ書房．
[14]
田中信行(2009)
．自律的な書き手を育成する活動としてのピア・レスポンス－学習者のピア・レスポンスへの適応過程の分析を通して
－，アカデミック・ジャパニーズ・ジャーナル
1, 25-36.
[15] Yoshizawa, S. & Kambayashi, Y. (2009). “Dialogue and monologue:
A practice of producing a coherent document,” IEEE International Professional Communication Conference Proceedings, 1A, 1-4.
[16] Yoshizawa, S. (2009). Effects of peer review activities in the EFL writing classrooms. Hoshi Journal of General Education, 27, 1-8.
[17]
吉澤小百合・寺野隆雄・吉川厚(2010)
．正規化圧縮距離の利用におけるピアリヴュー効果の予測可能性, 人工知能学会第
59
回先進的学 習科学と工学研究会資料,SIG-ALST/B001, 7-12.
[18]
吉澤小百合・吉川厚・寺野隆雄(2010)
．第二言語習得におけるピアリヴュー効果の分析, 日本認知科学会第
27
回大会論文集,457-462.
[19] Yoshizawa,S., Terano, T. & Yoshikawa, A. (2010). Analyzing the effects of peer review activities in the EFL writings, Proceedings of the 18
thInternational Conference in Computers in Education, 738- 742.
[20] Yoshizawa, S. (2010). Developing logical thinking skills through peer review activities, Hoshi Journal of General Education, 28, 1-10.
[21]
吉澤小百合・吉川厚・寺野隆雄(2011)
．英作文ピアリヴュー学習における複数評価者の影響に関する考察, 第
61
回先進的学習科学と工 学研究会資料,SIG-ALST/B003, 19-24.
[22]
吉澤小百合・吉川厚・寺野隆雄(2011)
．複数評価者による学習効果に関する一考察, 日本認知科学会第
