JACET
1
2013
3 :
? B L :
$/RRNDW:ULWWHQ&RUUHFWLYH)HHGEDFNIURPD'LVFRXUVH3HUVSHFWLYH: 7KH7HDFKHUDVD5HDGHU:KR,QIHUVLQWR/HDUQHU¶V,QWHQVLRQV
࣮ࣝࣈࣜࢵࢡ ࡢጇᙜᛶࡘ࠸࡚ࡢ᳨ド㸸ᵓᡂࢆ୰ᚰ
C1= R201–2012 4 S ? B + - G K 9 E - # RJACETS O I 8 M
! 7 / ? B #
1I 6
+2 Q, 㔠ᚿ ె௦Ꮚ
ஂ␃ 友紀Ꮚ ṇᮌ ⨾知Ꮚ ᒣす ༤அ
1
13
{ z 1N I
Ꮫⱥㄒᩍ⫱Ꮫ㸦-$&(7㸧㛵すᨭ㒊ࣛࢸࣥࢢᣦᑟ◊✲ࡣ㸪ᖺࡈ◊✲ࣉࣟࢪ࢙ࢡࢺ ࡢᡂᯝࢆ⣖せࡲࡵ࡚ฟ∧ࡋ࡚ࡁࡲࡋࡓࠋࡇࢀࡣࡑࡢ➨ྕ࡞ࡾ㸪➨ḟࣉࣟࢪ࢙ࢡ㸦 ᖺᗘ㹼ᖺᗘ㸧ཧຍࡋࡓဨࡀࡑࢀࡒࢀࡢ◊✲ᡂᯝࢆሗ࿌ࡍࡿࡶࡢ࡛ࡍ
➨ྕࡣ㸪ࡘࡢ◊✲ㄽᩥࡀᥖ㍕ࡉࢀ࡚࠸ࡲࡍࠋࡑࢀࡒࢀ㸪-$&(7㛵すᨭ㒊⣖せࡢㄽᩥᑂᰝࢆ
㋃くࡋࡓ᪉ἲࡼࡾ㸪ᮏ◊✲ဨࡢྡࡢᰝㄞ⪅ࡼࡿࣞࣅ࣮ࣗ㸪ࡑࢀࢆ㋃ࡲ࠼ࡓ᭩ࡁ┤ࡋࢆ
⾜࠸㸪ᥖ㍕ࡀỴᐃࡋࡓㄽᩥ࡞ࡾࡲࡍ
ᙜ◊✲⣖せࡣ⌧ᅾ㸪ᅜ❧ᅜᅗ᭩㤋ಖᏑࡉࢀ࡚࠾ࡾ㸪ᅜࡢ࠸ࡃࡘࡢᏛᅗ᭩㤋ࡶᡤⶶ
ࡉࢀ࡚࠸ࡲࡍࠋࡲࡓ㸪࣮࣒࣮࣍࣌ࢪ KWWSV VLWHVJRRJOHFRPVLWHMDFHWNDQVDLZUJ ࠾࠸࡚3')ࣇ
ࣝࡢᙧ࡛ㄞࢇ࡛࠸ࡓࡔࡅࡿࡼ࠺࡞ࡗ࡚࠸ࡲࡍࠋawxE wxcn)]$V j vࡗ
>-2,B0 e\ wx DK (!( L'_ &"*
7^l
WY~h}YEJACETg
>-2,B0e\wxE z1pwx9A1./3
______________________________________________________________________________________ JACET関西支部ラ テ ング指導研究会紀要 第12号 pp. 1–12. (2017)
A Look at Written Corrective Feedback from a Discourse Perspective:
The Teacher as a Reader Who Infers Learners’ Intentions
Kawanishi, Kei
Mukogawa Women’s University
Abstract
This paper examines teacher corrections that do not accurately take into account learner intentions. Drawing on a dynamic model (Matsuda, 1997), this paper discusses the need for discourse and genre perspectives in error correction. Accordingly, this paper regards the teacher as the reader and the learner as the writer, both of whom are responsible for the communication of ideas through written text. Through error and discourse analysis, this paper determines that corrections by the teacher are merely inferences made of the learner–writer’s intentions. This finding implies that the teacher uses genre knowledge (e.g., evidence comes before generalization in the academic genre) and contextual knowledge to fill in what can only be inferred. In fact, accurate correction can only be determined by the learner, who is the only party with in possession of the intended meaning. The practice of correcting learner errors while the learner is not present may need to be changed entirely. Additionally, the findings indicate that teachers could help learners acquire genre knowledge as well as genre-based grammar (i.e., how some grammatical items are patterned within a certain genre).
Key Words: written corrective feedback, discourse, genre
1. Introduction
L2 writing theory has been modified repeatedly as more is learned about not just learners but how teachers respond to L2 writing. Matsuda (1997) proposed a dynamic model of L2 writing, stating that the text is where the writer and the reader encounter each other in communication. This implies that the learner is not solely responsible for communication, as meaning is co-constructed by the writer and the reader through the medium of the text. This claim, which resonates with the interactionist approach in SLA (e.g., negotiation of meaning), shifts the view created by 1960s contrastive rhetoric, which saw learner text as containing problems that need to be solved.
While this dynamic model has had an impact on L2 writing pedagogy, the area of research known as written corrective feedback (WCF) has continued to use teachers as a source of knowledge who can detect and correct learner errors, giving learners a passive role in correction. The adoption of a static model of writing, where learners are single-handedly responsible for the communication, may need an update.
This paper aims to shed light on teacher corrections that, according to learners, are not accurate representations of what they intended, and discusses the need for a dynamic model in WCF. The two studies reported in this paper employed error and discourse analysis for a qualitative look at classroom data. The results show that, even among native speakers, the teacher is not an omniscient source of knowledge but a reader who is merely making inferences of the learner’s intended meaning.
2. Literature Review
2.1 Written Corrective Feedback and the Classroom
WCF has enjoyed much research attention over the past couple of decades. The procedure of WCF studies has been refined in response to Truscott (1996), who questioned the effectiveness of WCF as a practice. Since then, researchers have worked to ensure that accuracy gains are systematically gauged, and studies examining this topic have become well-controlled experiments rather than classroom observations. Moreover, writing in such research has become mostly input-based; the teacher tells a story to the class and the class writes about it, retelling what they have heard (e.g., Sheen, 2007; Shintani & Ellis, 2013).
2.2 Error Detection, Diagnosis, and Correction
Current WCF research does not often address the process of error correction, but
revision in writing has been being explored since the 1980s, and the common understanding is that revision is a three-fold process: detection, diagnosis, and correction (e.g., Bartlett, 1982; Cho, 2011; Flower, Hayes, Carey, Schriver, & Stratman, 1986). The revision model of Flower et al. (1986) includes evaluation and revising, which can be understood as detection and correction. It appears that learners, in peer review, tend to detect more problems in others’ writing than their own (Bartlett, 1982; Cho, 2011), perhaps because they are less familiar with the content and must carefully evaluate it. After the detection of a problem or an error, the reviewer of the text needs to diagnose its nature. Comparing expert and novice writers, Hayes, Flower, Schriver, Stratman, and Carey (1987) found that expert writers diagnosed 74% of the problems they detected, whereas inexperienced writers diagnosed only 41% of what they detected. It is then no wonder that the reviewer may have an even lower rate of correctly solving detected and sometimes misdiagnosed problems.
It is now clear that today’s WCF studies assume that the teacher can accurately detect, diagnose, and correct all learner errors. This assumption may hold true to a certain extent in research, because these studies deal with focused feedback on specific grammatical items, rather than targeting every possible error. The teacher also knows the content the learner reproduces. However, due to the less-controlled nature of the classroom, the implications of
such research should be taken with a grain of salt. This paper proposes two additional perspectives for adapting error correction for the classroom.
Proposal 1: WCF involves a discourse comprehension
WCF involves reading on the teacher’s part and thus inference making. That is, the teacher– reader’s inference of the learner–writer’s intentions plays an important part in diagnosing and correcting learner writing.
Proposal 2: Genre matters
Certain grammatical patterns are expected in a given genre (i.e., there is a genre-based grammar), and the teacher–reader operates within these, while the learner–writer needs to learn this specific codes of the discourse community.
Proposal 1 implies that the teacher’s inference can be wrong. In discourse
comprehension, it is said that the reader infers the writer’s intended meaning, since he or she does not have direct access to it (Brown & Yule, 1983). For instance, the reader must identify the writer’s intended referent when they hear pronouns or deictic expressions (e.g., the cat, they).
Interestingly for L2 error correction research and practice, only humans can process discourse in a top-down process, and artificial intelligence cannot simulate this (Brown & Yule, 1983). According to Brown and Yule (1983), readers are always predicting from
context and sentences already processed and use existing knowledge to predict “what the next sentence is more likely to mean” (p.234). This means that a human reader can even process and interpret erroneous text quite well because of his or her knowledge and ability to predict
“what is likely to come next in the text” (p.235). The ability to predict what should come next, relating it to the pattern of a text, is known as genre knowledge, and it plays a large role in language processing.
Moreover, the ability to infer the meaning of erroneous text also means that the
accuracy of the reader’s prediction (i.e., error correction) can only be determined by the writer, who produced the text out of his or her knowledge and experience.
The interpretation of writing in context brings further attention to another discourse-related aspect of writing: grammar. Knapp and Watkins (2005) discuss five fundamental genres of school writing—describing, explaining, instructing, arguing, and narrating—and call for the teaching of genre-based grammar. Though he emphasizes patterns more than grammar, their view does not contradict Matsuda (1997), who writes as below.
Teaching ESL students to organize L2 writing, then, does not mean imposing on them the cultural values of native English speakers or prescribing patterns. Rather, it should be considered as a way of raising ESL students’ awareness of various factors that are
involved in structuring the text, including the reader’s expectations of certain organizational patterns. (p.56, emphasis added)
Proposal 2, indicating the importance of genre-based grammar, shows this. This paper examines referential and aspectual patterns, both of which are said to be coded in genre, and sees how the teacher–reader’s inference may not always match the learner–writer’s intentions.
2.3 Genre-based Grammar: Referential and Aspectual Patterns in Academic Genre One characteristic of academic genre is exemplification and generalization. Most often, research papers and academic textbooks first discuss concrete evidence and then abstract away from the findings. This is known as inference, and language also matches this general– specific1 or concrete–abstract pattern in generics (as expressed by reference and aspect). As shown by the examples below, scientific research is conducted not only on particular samples at a particular moment in time—105 female students and 49 cats—but rather to make inferences about the population in general—students and cats. Notice that the reference switches between the generic bare plural and non-generic, accompanied by number, the definite article, or other determiners. Notice furthermore that aspect shifts between simple present and simple past to indicate timeless truths and past events (see Section 2.4 for details). The passages below contain both generalizations and exemplifications with accompanying grammatical patterns.
Researchers at the University of Nebraska-Lincoln and Pennsylvania State University have found that a student's fitness level and iron status could be the difference between making an A or a B. (…) The 105 women analyzed were all enrolled at Penn State and had an average GPA of 3.68. Data showed that women with the highest levels of stored iron had the highest grades. In addition, those who were fittest and had adequate iron stores had higher grades than less-fit women with lower iron stores.
(https://medicalxpress.com/news/2017-01-iron-deficiency-linked-gpa.html)
In a new study, a group of Japanese scientists have found that cats have similar memory abilities to dogs. A study of 49 cats showed they can recall memories of pleasant
experiences, much like dogs.
(http://www.thedebrief.co.uk/news/real-life/science-says-cats-are-as-smart-as-dogs-2017 0166394)
From the above examples, it may be clear that simple grammatical items such as reference and aspect are expected to pattern in certain ways in given genres.
2.4 Generics: Grammar Expressing Habits and Abstract Concepts
As shown above, both reference and aspect represent generalization, and this is through a similar process. Krifka, Pelletier, Carlson, Chierchia, Link and Ter Meulen (1995) emphasize that generic reference allows us to “abstract away from particular object”2 (p.4, emphasis added) and that generic aspect allows us to “abstract away from particular events and facts” (p.5, emphasis added).
Figure 1 shows three sentences only differing in the general–specific dimension of
meaning, seen as the three levels, kind, object, and stage, coined by Carlson (1977). White areas indicate generic, indicating abstract things and actions, while gray areas indicate non-generic, representing concrete things and actions.
Figure 1. Generics
For instance, generic reference, a way to refer to an abstract concept, is seen with words such as dogs and bananas, which, pluralized without an article, do not refer to any specific dogs or bananas. The generic aspect, simple present, indicates that the action is a habit or a timeless truth, rather than a specific episode of that action. Generic reference and aspects can be combined as seen above: a stage-level statement indicating a concrete episode that
occurred to a particular individual(s), an object-level statement representing a habit of an individual(s), and a kind-level, abstract concept.
With learner errors in articles, which are of the plural–singular and present–past (tense/aspect) nature, it can be quite difficult to interpret a learner’s intended meaning. The two studies reported in this paper examine how the teacher–reader examines learners’ erroneous writing to interpret and correct them.
RQ 1. Are teacher–readers’ corrections accurate?
RQ 2. How do teacher–readers detect, diagnose, and correct?
Two studies were conducted: Study 1 focused on the learner’s intentions and the accuracy of teacher corrections (responding to RQ1), and Study 2 focused on the process of teacher correction (responding to RQ2).
3. The Investigation 3.1 Study 1
3.1.1 Methods
In order to determine whether teacher–readers’ corrections are accurate, the following study employed resources from Kawanishi (2015). TOEFL essays written by Japanese university students were examined for general–specific errors. The writing prompts were two expository TOEFL prompts: whether people learn from mistakes, and whether a pet is part of the family (see Kawanishi, 2015)
3.1.2 Participants
To examine a teacher–reader’s correction of a learner–writer’s text, the researcher approached 10 of 34 students in a class surveyed in (Kawanishi, 2015). Of those, five agreed to participate in the study. The errors were corrected by a native speaker with at least two years’ experience of teaching English in Japan.
3.1.3 Procedure
This study employed error-corrected student essays from (Kawanishi, 2015). To check for the match between the teacher–reader’s inference and the learner–writer’s intention, the researcher interviewed the learners. The researcher held the error-corrected version of the essay, while the student was given a copy without WCF. The researcher asked the student to describe their essay in Japanese, and when the student discussed the point where the supposed error was contained, the author asked questions regarding the generalness (or specificness) of the meaning. The topics were learning from mistakes and considering pets as part of one’s family, so questions were asked such as “Is this a mistake you made or mistakes in general?” or “Are you talking about your pet or pets in general?” The conversation was recorded using an IC recorder with the participants’ consent, and the results were analyzed for consistency against meaning the teacher inferred.
3.1.4 Results and Discussion
Of the five learners who volunteered, two learners’ writings were corrected inaccurately. Of the five learners’ essays, four inaccurate instances of teacher–corrections emerged for two writers. Those are presented below.
The two excerpts below are from Learner X. The first excerpt contains three instances of inaccurate corrections and the second excerpt shows this learner’s tendency to misuse pronouns. The italics show the target forms, the square brackets indicate teacher corrections, and asterisks indicate incorrect inferences, unrepresentative of what the learner intended.
First, we can play with pets without thinking anything bad. When I talk with friends, I enjoy talking, but I get tired with the situation. I consider whether they are bored, they are irritated with me, and so on. However, when I play with my pets, I enjoy it without considering such things. (1) Each of members of family [my family*] have a close relationship, so we play with them, considering nothing. (2) That is, pet [my pet*] is a member of family [my family*].
The sentences (1) and (2) contain bare nouns, family, pet, and family. The teacher corrected them to my family, my pet, my family respectively, probably because the sentence previous to (1) contains information about the learner’s family and their pet. However, the learner stated that the sentences (1) and (2) are generalizations he made, not referring to his pet or his family in particular. Thus, the correct forms should have been: a/the family, a pet, the family.
Next, in the excerpt below, the native speakers (NS) teacher caught four instances of aspectual errors, but seems to have inaccurately corrected one referential error.
Third, making mistakes sometimes determine the belief of our life. >> When I was a junior high school student, I had a best friend (…) One day, I quarreled with him over trivial things. (3) Next day, we never talk [talked]. I wanted to apologize him, but I did not have enough aggressiveness. After all I did not apologize him and he did not either. (4) So, we never talk [talked] and [or] play [played] again. I still regret about not apologizing him. (5) Now, when I quarreled [quarrel] with my friend*, I always try to apologize him or her certainly.
The native speaker teacher was accurate in inferring that the quarrel reported in sentences (3) and (4) was a one-time event and not a habit, but he failed to infer that the learner was making a generalization in (5). Apologizing first is something this learner learned to do with any friend, not just with this particular friend. Thus, my friend was an error missed by the NS teacher, which should have been a friend (the expression him or her that followed could have served as a hint).
It is clear that the teacher was correcting errors to make the text either more general or more specific, and his inaccurate corrections were inaccurate because he went wrong in the general–specific dimension: interpreting concrete evidence as a generalization and vise versa. This finding leads to a hypothesis. The teacher may detect incongruence in terms of the general–specific dimension of the meaning first and try to solve the incongruence by
correcting either the subject or predicate for the inferred level of generalness based solely on context. That is, the teacher may be looking for cues in context to see whether the statement with an error is evidence or generalization. For instance, where a student wrote a sentence like
“people made a mistake” (see Figure 2), with a bare plural indicating a kind-level subject and simple past indicating a stage-level predicate, the teacher either changed the subject to make it more specific (e.g., the/some people made a mistake) or the predicate to make it more general (e.g., people make mistakes). On the other hand, when the teacher encountered a sentence like
“my friend made mistakes” (see Figure 3), with a concrete subject and somewhat abstract predicate, the teacher could make the predicate more concrete (e.g., my friend made a mistake) or abstract (e.g., my friend makes mistakes).
Figure 2. Generic subject with non-generic predicate
Figure 3. Non-generic subject and verb with a generic object
3.2 Study 2 3.2.1 Methods
This follow-up study was conducted after the analysis of Study 1. This study was conducted so that the process of teacher error correction could be observed, as the teacher who corrected errors in Study 1 was not available for interview. This study made use of a modified learner text and observed the error correction processes of teachers.
3.2.2 Participants
Four teachers were recruited. Two were NS (NS 1 and NS 2), both of whom have worked with Japanese learners of English. NS1 had 5 years of experience, while NS 2 had over 10 years. The two Japanese teachers of English (JTE 1 and JTE 2) had three and five years of experience, respectively.
The study employed one modified learner text. The text used learner writing from (Kawanishi, 2015) and contained sentences with internal incongruences in terms of the general–specific dimension of the meaning (e.g., Figure 2 & 3). Similar sentences were planted in different parts of a paragraph to appear to be evidence and generalization. The text had 33 instances of these planted errors (see Appendix 1 & 2).
3.2.3 Procedure
Corrections among the four teachers were compared for similarity. Even if the
corrections did not match, if the corrections matched on the general–specific dimension, they were considered a match (e.g., “a dog is an animal” and “dogs are animals” would have both been considered to be correcting the sentence to a general statement, thus denoting similar meanings).
3.2.4 Results and Discussion
Among the four teachers, only 12 out of 33 items were found to have corrections that matched on the general–specific dimension.
Here are five example sentences that were corrected in a very different fashion from teacher to teacher (Table 1–5).
Table 1
We Made Mistakes (No Context)
Correction of reference We made a mistake JTE 2 Correction of aspect We have made mistakes JTE 1
The two NS teachers considered this sentence acceptable, while JTE 2 corrected it to a non-generic reference referring to a specific event, and JTE 1 corrected it to present perfect, indicating events as experience.
Table 2
We Made Mistakes (Same as Table 4, But Now with a Context)
Correction of reference We made a mistake JTE 2
Correction of aspect We make mistakes NS1, NS 2, JTE 1
This sentence, although it was the same one as Table 1, was corrected by all four teachers. JTE 2 thought it was about a specific past event, while the other three teachers thought it was a generalization, correcting the tense/aspect.
Table 3
We Do Not Make a Mistake
Correction of reference We do not make mistakes NS 1, JTE 1 Correction of aspect We did not make a mistake NS 2
JTE 2 thought the sentence was acceptable, while two teachers (NS 1 and JTE 2) considered it to be a general statement. NS 2 thought it was referring to a past event. Table 4
People Learn From a Mistake
Correction of reference People learn from mistakes People learn from their mistakes
NS 1, JTE 1, JTE 2
Correction of aspect People learned from a mistake NS 2
Three teachers (NS 1, JTE 1, JTE 2) corrected the sentence to represent a general statement, while NS 2 corrected it to indicate a specific event.
Table 5
I Learn From a Mistake
Correction of reference I learn from mistakes I learn from my mistakes
NS 2, JTE 2
Correction of aspect I learned from the mistakes I learned from my mistakes
NS 1, JTE 1
NS 2 and JTE 2 corrected the sentence to represent a general statement about an individual, and NS 1 and JTE 1 corrected it to indicate that it was a specific event about this individual.
As can be seen from the above five sentences, the four teachers’ corrections were quite dissimilar, often resulting in meanings different from each other. This seems to be because the teachers inferred the learner–writer’s intended meanings using contextual cues only, without actually communicating with the writer. It also shows how differently contextual cues can be understood by experienced English teachers.
4. Conclusion
The studies reported in this paper investigated whether teacher–readers’ corrections are accurate (RQ1), and if not, how they detect, diagnose, and correct learner errors (RQ2).
Focusing on generic and non-generic referential and aspectual errors in academic genre, Study 1 revealed that NS corrections are not always accurate.
Further investigation was conducted in Study 2, where the focus was on teacher correction. The hypothesis was that the teacher–reader detects incongruence between the subject and the predicate of a sentence on the general–specific dimension; thus some sentences were planted in contexts and teachers’ corrections were observed. The findings showed that the teacher–readers detected erroneous sentences but corrected them differently at times. This implies that contextual cues can be interpreted differently among different individuals and that even among native-speaking teachers, inference of intended meanings in erroneous text can produce disagreements.
The studies reported are not without their limitations. The analyzes involved qualitative investigation of small samples using techniques such as error analysis and discourse analysis. This were not controlled studies, nor were they quantitative, leaving out any calculation of the teacher–readers’ reliability in error correction. However, these studies were meant to show the qualitative nature of discourse comprehension processes that have been overlooked up to this point. Thus, Proposal 1 that WCF involves discourse comprehension, and Proposal 2, genre matters, seem to be generally correct from these two small-scale studies.
The pedagogical implications for writing teachers in the foreign and/or second language teaching context include the following. When correcting learner errors, a teacher should realize that he or she is using genre knowledge (e.g., evidence comes before generalization in the academic genre) and contextual knowledge to fill in what can only be inferred. Accurate correction may only be determined by the learner, who is the only party with the most relevant information. Perhaps the practice of correcting learner errors while the learner is not present should be entirely changed.
Additionally, the findings indicate that teachers could help learners acquire genre
knowledge as well as how some grammatical items are patterned within certain genres. Using the examples of this paper—generic and non-generic reference and aspect—the teacher could present how the two grammatical items may interact and how certain combinations are unlikely (e.g., People make a mistake). Thus, for teachers teaching writing: provide learners
with genre knowledge, including genre-based grammar. In the expository/academic genre, make sure to cover generics. Grammar is in fact part of genre conventions.
Notes
1. This paper only deals with linguistic genericity and not specificity. Thus, the words general and specific are used in non-linguistic senses. For instance, dogs swim is a general statement, while a dog is swimming indicates a specific event and individual.
2. The object within the context of generics is not the same as the grammatical object (cf. grammatical subject) but refers to individuals or things such as a cat and some cats (cf. cats as in species).
Acknowledgement
I would like to thank the reviewers for inferring my intensions and proving valuable
comments and advice on an earlier draft of this paper. I am also grateful to the students and teachers who have kindly participated in this study. Any remaining errors are my own. References
Bartlett, E. J. (1982). Learning to write: Some cognitive and linguistic components. Washington, DC: Center for Applied Linguistics.
Brown, G., & Yule, G. (1983). Discourse analysis. Cambridge University Press.
Carlson, G. N. (1977). Reference to kinds in English. Ph.D. dissertation, University of Massachusetts at Amherst. Published in 1980 by Garland, New York.
Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving comments. Instructional Science, 39(5), 629–643.
Flower, L., Hayes, J. R., Carey, L., Schriver, K., & Stratman, J. (1986). Detection, diagnosis, and the strategies of revision. College Composition and Communication, 37(1), 16–55. doi:10.2307/357381
Hayes, J. R., Flower, L., Schriver, K., Stratman, J., & Carey, L. (1987). Cognitive approaches in revision. In S. Rosenberg (Ed.), Advances in applied psycholinguistics: Vol. 2. Reading, writing, and language processing (pp. 176–240). New York, NY: Cambridge University Press.
Kawanishi, K. (2015). An Error Analysis on English Articles in Learners’ TOEFL-type Essays: A Focus on Generic and Non-generic Errors. The Bulletin of the Writing Research Group, JACET Kansai Chapter, 11, 1–12.
Knapp, P., & Watkins, M. (2005). Genre, text, grammar: Technologies for teaching and assessing writing. Sydney: University of New South Wales Press.
Krifka, M., Pelletier, F. J., Carlson, G. N., Chierchia, G., Link, G., & Ter Meulen, A.
(1995). Genericity: An Introduction in G. Carlson and J. Pelletier (eds.), The Generic Book, The University of Chicago Press, Chicago.
Matsuda, P. K. (1997). Contrastive rhetoric in context: A dynamic model of L2 writing. Journal of Second Language Writing, 6(1), 45–60.
Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning, 46(2), 327–369.
Sheen, Y. (2007). The effect of focused written corrective feedback and language aptitude on ESL learners' acquisition of articles. TESOL Quarterly, 41(2), 255–283.
Shintani, N., & Ellis, R. (2013). The comparative effect of direct written corrective feedback and metalinguistic explanation on learners’ explicit and implicit knowledge of the English indefinite article. Journal of Second Language Writing, 22(3), 286–306.
Appendix 1. Examples of potentially erroneous sentences without context Ø We made mistakes. (see Table 1)
Ø People often make a mistake.
Appendix 2. Examples of potentially erroneous sentences with context
Ø In a body paragraph: One year ago, I made mistakes. I did not study well for the entrance exam for X University, and failed. I was shocked and really sad. It is true that I made mistakes, but I learn from a mistake.
Ø In a conclusion paragraph: In this way, people learn from mistakes. Mistakes are unavoidable, and we made mistakes. But what is important is that we learn from a mistake. (see Table 2) (Underlines were not provided for the participating teachers)
______________________________________________________________________________________ JACET関西支部 ン 指導研究会紀要 第12号 pp. 13–24. (2017)
ーブ ック 2009 の妥当性についての検証―構成を中心に
Analyzing the Validity of Rubric 2009 : Focusing on Organization
OTOSHI, Junko
Okayama UniversityKURU, Yukiko
Aichi Medical UniversityYAMANISHI, Hiroyuki
Kansai University
KINSHI, Kayoko
University of HyogoMASAKI, Michiko
Osaka International UniversityAbstract
This paper is an attempt to analyze the validity of the dimension of “organization” in Rubric 2009, an analytical writing rubric developed for EFL classroom use in higher education in Japan. To this end, four native English speaking teachers holistically evaluated 30 essays of 300 to 350 words written by Japanese college students, referring to no rubrics. At the same time, five Japanese-speaking English teachers analytically evaluated the same essays using Rubric 2009. All the evaluators gave numerical scores as well as reflective comments on evaluation. The study examined the validity in two ways: analyzing the concurrent validity based on the quantitative data, on one hand, and analyzing the qualitative data of reflective comments using the KJ method, on the other. While the concurrent validity between the holistic evaluation and the analytical evaluation of “organization” in Rubric 2009 was not very high, the qualitative analyses revealed no need to revise the sub-dimensions in
“organization”. Through the study, however, various suggestions were obtained on teaching and assessing EFL academic writing in Japan, including the importance of introduction, sentence sequencing, and common problems of Japanese writers. Such suggestions should be made available in the form of interpretations of Rubric 2009.
Key Words: rubric, EFL writing, organization, concurrent validity
1.
評価基準 規定 指標 あ キン や ン
ン 評価 あ い評価 あ 。 ン
ン う 側面 評価 程度 ン 期
待 い 明記 う 評価 方針 評価 側 評価 側 両者
具体的 明示 いう大 役目 果 。実際 評価基準 明記 使用
自 英 作 文 適 評 価 経 験 有 無 関 わ 難 い
(Cumming, 1990)。
著者 日本 大学 短大 以 大学 英語 授業 い 教員 学生 共 使いや い
ン 開発 組 来 。開発 ン
評価 対象 課題 ャン ン ン
教 育 的 行 い や い 析 的 評 価 (analytic scoring) あ 。 目 (dimension domain) 内容 展開 (content & idea development) 構成 (organization)
文法 (grammar) 語彙 (vocabulary) 綴 句 (mechanics) 5 全 日本 語 英語 両言語 書 い 。 当初4段階 評価尺度 (rating scale)
目 い ～ ベ 規定 記述子 (descriptor) あ 実際
運用 検証 後 1 各 目 い 目 特定 2
目 望 い到達基準 明記 3 対象 学生間 相対的 評価 可能
段階 評価尺度 ベ 規定 記述子 削除 3 い 改
訂 行 (Kinshi, Kuru, Masaki, Yamanishi, & Otoshi, 2011)。 結果得 2009 以 斜体 あ 図 。 ーブ ック 2009 一般的
う ベ あ い 評価 記述子 代わ 各 目 目 目標
ン い 特徴 あ 。 ベ あ い 評価 目的
や状況 応 自 設定 い 。
改訂 一番大 変更 行 検証 問題 あ 思わ 構成
目 あ 。改訂前 文 構成 明確 論理的 あ う 比較的単純 記述子
ーブ ック2009 具体的 目 特定 。一般的
ン 構 成 文 章 全 体 構 成 や 単 や 序 共 節
(clause) 以 単 や 序 指 。構成 内容 文章 論理性や流 大い
関わ 部 あ ン 指導 い 評価 い 視 あ 。
構成 何 いう定義 え 評価者 直感的 理解
い (Cumming, 1990) 一方 構成 構成 (rhetorical organization)
(rhetoric) 文章 流 (text flow) 論理的 序 (logical sequence) 結束性 (cohesion) 多
様 用語 表現 関 あ 。 概念 定義 難 い あ
互い 複 。 構成 焦 あ 実践的 研究 著者 知 限 大
変限 い 数少 いう あ Miyazaki (2008) 日本人高校生
ン 構成概念妥当性 検証 Content Organization Cohesion Voice いう 目 析 的 評 価 結 果 回 帰 析 検 証 Organization Cohesion
.827 いう強い相関関 あ 1% 水準 有意差あ 認 。
う 直感的 理解 具体的 明示 難 い ン 構成
い 実践的 研究 通 具体的 知見 得 い 評価 い 指導
い 要 あ う 研究 著者 知 限 い。 時 ーブ ック2009
内容 展開 Content & Idea Development
課題 要求 対応 内容 述 い
The content is relevant to the given topic.
課題 条件 数 時間 満 作文 完成 い
The writing was completed in accordance with the task requirements.
論旨 外 首尾一貫 主張 展開 い
The writing is coherent.
ン ン ン 明確 提示 い
Every topic sentence is clearly shown.
ン ン ン 十 展開 い
Every topic sentence is fully developed.
構成 Organization
導入 展開 結論 構成 い
The writing includes an introduction – body – conclusion structure.
文章 論理的 構成 い
The writing is logically organized.
文章 流 あ
The writing flows smoothly.
言葉 適 使用 い
Connecting words / expressions are used effectively.
文法 Grammar
多様 文 構文や表現 適 使わ い
A variety of sentence structures are appropriately used.
文構造 正 い
Sentence structures are accurate.
主語 動詞 一 時制 数 代 詞 冠詞 前置詞 誤 い
The writing has no errors of subject-verb agreement, tense, number, pronouns, articles, prepositions, and so on.
語彙 Vocabulary
多様 語や表現 使わ い
A variety of words and expressions are used.
語や表現 選択 適 あ
The choices of words and expressions are appropriate.
綴 句 Mechanics
段落 最初 行 適 ～ ½ ンチ程度 い
The first line of each paragraph is appropriately indented (3 to 5 letters or ½ inch).
段落 始 適 置 改行 い
Lines are appropriately changed.
綴 正確 あ
The spelling is accurate.
句 大文 使用 正確 あ
The punctuation and capitalization are accurate.
図1. ーブ ック2009
い 信 性 い 一般化可能性理論 用い 検証 い (Kinshi et al.) 改
訂 行 構成 目 い 追加 検証 特 目 含 妥当性 検証 必
要 あ 考え 。 本研究 日本人大学生 作文 実際 評価 的 析
質的 析 合わ 行い ーブ ック2009 構成 妥当性 検証 。併 析
得 具体的 知見 基 い 構成 関 評価 指導 示唆 得 目的
い。
2. 方 法
2.1 作 文 タ
作文 タ提供 協力者 関東地方 公立大学 い ン
講 生 中 本研究 タ 集 協力 賛 学生 無作 抽出 30
学生 あ 。30 協力者 TOEIC 均 572 あ 。 タ 集
ン Paragraph Writing: From Sentence to Paragraph (Zemach &
Islam, 2005) 共通教科書 ン 習得 ン ン
英語 基本的構造 理解 基 様々 英語 形態 考慮 更
発展的 ン 能力 向 目指 ン あ 。
本研究 使用 作文 タ 中間試験 授業時間内 書 あ 。試験
制限時間 40 え ン 図2 い 300語 350語 作文
タ う指示 。
2.2 作 文 タ 評 価 妥 当 性 析 方 法
構成概念妥当性 (construct validity) 称 妥当性 評価方法 測定 う 概念 程度 映 い 考慮 (Bacha, 2001)。測定 関 評価者 持 概念 総括 映 全体的評価 (holistic scoring) 原理的 構成概念妥当性 高 期待
一方 評価 目 規定 析的評価 原理的 構成概念 生 考
え 。構成概念妥当性 測定 難 い 併 妥当性 (concurrent validity)
検証 方法 考え 。併 妥当性 能力 測定 い 評価方法 相関
確認 妥当性 検証 (Bacha, 2001)。 本研究 原理的 構成概念妥当性 高い全体的評価結果 ーブ ック2009 評価結果 比較 併 妥当性 検討
的 析 。
全体的評価 作文 タ 集 大学 勤務 英語母語話者教員 行 。
独立 参照 断基準 基 い 段階 述 作文 評
価 。一方 ーブ ック2009 評価 著者 あ 日本人英語教員 行 。英語
母語話者教員 評価 作文 独立 ーブ ック2009 目
段階 評価 。 評価結果 基 い ーブ ック2009 目 特性
見 Item-Total相関 析 (Item-Total correlation analysis) 行い ン
α 数 算出 ーブ ック2009 内的整合性 尺度 信 性 確認 。 日本
人英語教員 5 各 目 均値 英語母語話者教員 全体的評価 均値
間 ソン 相関 数 算出 相関 析 行 。 併 妥当性 確認 。
質的 析 作文評価 断根拠 記 回顧的自 記述 い KJ法 川喜 1967
手法 使 行 。KJ法 タ ユ
記述 類 関 図解 新 発想 生 出
目指 研究法 あ 。 類作業 KJ 法 A型図解 川喜 1967 著者
討議 ン 作成 類 化 各 称
Some people believe that language classes should concentrate on studying grammar and vocabulary. Other people believe that language classes should concentrate on practicing conversation. Briefly compare these two types of language classes. Which type do you prefer? And why?
図2. ン
類作業 行 。本研究 回顧的自 記述 新 い発想 求
ーブ ック 2009 目 いう既 当 場合 構成 目内 新
い概念 見 い 確認 目的 。
質的 タ い 英語母語話者教員 日本人英語教員 両方 集 。
評価者 一 一 作文 対 う う え 断根拠 回顧
的 自 記述 記録 い 。 ーブ ック2009 開発 日本 大学 ン 指導 経
験 持 日本人英語教員 行わ 考慮 参照 評価
英語母語話者教員 タ 開発 段階 得難 視 や ン 得
可能性 高い。 英語母語話者教員 タ 新 い タ 得 可能性 高
い いう い わ い 考え 。一方 日本人英語教員 ーブ ック2009 用い
行 評価 自 記述 ーブ ック2009 目 実際 作文 見比
目 示 実際 作文 う 特徴 表 い 確認
いう 期待 。 回顧記録 中 著者 構成 関わ 記述
出 討議 析 作成 。 類 化 各
称 類作業 行 。 精緻 検証 目指 最初 類作業 間後
う一度 必要 応 学生 ン 再度参照 化 称 確認 。
3. 結 果
3.1 併 妥 当 性
日本人英語教員5 30編 作文 タ ーブ ック2009 析的評価 評価
結果 基 ーブ ック 2009 5 目 特性 各 目 合計得 相関 あ 修正済
I-T相関 数 確認 表 段 。 正 相関 高い 目 合計得 寄 率 高い
今回 評価 い 要視 い あ わ い 。 結果 内容 展開 構
成 文法 語彙 目 有意水準5% い 統計的 有意 中程度 相関 r = .44
〜.62 見 , 綴 句 合計得 相関 .09 無相関 あ
統計的 有意 。
ーブ ック2009 5 目 1 尺度 捉え 際 内的整合性 尺度 信 性
析 ン α 数 算出 。 時 特定 目 削除 際 尺度 信
性 増減 表 段 示 。I-T 相関 析 析 各 目 ーブ ック
2009 対 寄 度合い 精緻 検討 。 目 数 α = .69
あ 十 高い 一般的 α = .80以 言え い ン 評価
比較的高い値 あ 。 信 性 数 α = .69 基準 各 目 削除 時 数
増減 見 綴 句 削除 場合 信 性 数 向
α = .69 àα = .78 。 綴 句 削除後 数 十 信 性 あ 考え
α = .80 近い あ 。 結果 綴 句 目 異 振
表1. Item-Total相関
内容 展開 構成 文法 語彙
綴 句
修正済I-T相関 数 r .44* .62* .62* .54* .09
目 削除 場合 α 数 .63 .54 .54 .62 .78
*p<.05
修正 済I-T 相関 係数ｒ
