• 検索結果がありません。

Adaptation, Analysis and Critique of the Survey of Achievement

N/A
N/A
Protected

Academic year: 2021

シェア "Adaptation, Analysis and Critique of the Survey of Achievement"

Copied!
17
0
0

読み込み中.... (全文を見る)

全文

(1)

<&**^tt&&WffftlO (2012) 163

Adaptation, Analysis and Critique of the Survey of Achievement

Responsibility (SOAR) for use in the Japanese SLA Context

Todd Tournat

Abstract

This measurement study reports on the adaptation of the Survey of Achievement Responsibility (SOAR) into the Japanese second language acquisition (SLA) context. The protocol for best practice used in this adaptation process was based on the International Test Commission (ITC) guidelines (Hambleton, Merenda, &Spielberger, 2005). Thefirst step was to translatethe original English version of this instrument into Japanese, and the second step was to back-translate it into English and compare the back translation to the original. This process was conducted to correct for any cultural themes that may not have any relevance in the target population (Japanese university students), and to ensure that the language expressions used were equivalent. Participants in the study came from 654 students at four different universities in western Japan studying in the fields of English, social welfare, science, education, law, engineering, medicine, business, communication, and Japanese. The analytical process included checking the test items for normality and determining the reliability estimates (Cronbach's alphas) of the subscales that comprise the instrument. The fit of the model, hypothesized by the authors of the instrument, was directly tested by conducting a confirmatory factor analysis. Finally, the implications of the results and analysis for the adapted version of the SOAR instrument are discussed.

1.0 Introduction

Attribution theory (Weiner, 1979; 1985, 2010; Weiner, Frieze, Kukla, Reed, Rest, & Rosenbaum, 1971) has partially re-emerged as an important research trajectory owing to its migration into the field of applied linguistics (Gobel & Mori, 2007; Hsieh & Kang, 2010; Hsieh & Schallert, 2008; Peacock, 2010) where it is finding increasing attention. The rationale for this migration was based in part, on a need to understand students' perceptions for success and/or failure in second language acquisition (SLA) so that appropriate intervention could be provided (Banks & Woolfson, 2008; Graham, 1997; Stipek &

Weisz, 1981). The theoretical components of attribution theory used in the area of SLA, and inherited from the general educational research trajectory, include the following four potential causal attributions;

ability, effort, luck, and task difficulty largely articulated by Weiner (1979; 1971). Weiner identified

these causal attributions as being the typical attributions for achievement-related outcomes. However,

the main concern that surrounds this issue is the development and use of appropriate instrumentation that can accurately measure these causal attributions and produce valid scores.

(2)

164 Todd TournaL

This paper contributes to the endeavor of establishing adequate instrumentation for the research area

within SLA, by adapting Ryckman and Ratio's (1983) Survey of Achievement Responsibility (SOAR) from the mainstream educational research context (mainly North American) on attribution theory into

the Japanese SLA context. The adaptation process adopted the guidelines of the International Test

Commission (Hambleton et al., 2005) as a protocol for best practice. An analysis of the psychometric properties of scores generated by the instrument, and a critique of the adapted version of the SOAR

instrument is presented below.

2.0 Literature Review

Attribution theory has a relatively long history within educational research. The early work began in

the 1950s, and there was significant evolution in theory and constructs until the 1970s when it is arguable that the typical constructs employed in theory reached something close to their final state.

The details of this evolution in theory related to causal attribution are beyond the scope of this paper

and the reader is referred to Weiner (1979; 1985) for a more complete review. In brief summary of the

final state of constructs employed in the area, four constructs comprise the foundation of attribution theory and these are locus of causality, stability, personal control and external control. These causal

attribution constructs are represented in the general line of instrumentation as ability (locus of causality), effort (personal control), luck (external control), and task ease/difficulty (stability). These perceived causes, if measurable, are useful in determining future success and/or failure, and in determining the motivation level for a future task (Heider, 1958; Weiner et al., 1971). The following are examples of instruments where authors used these specific constructs: the Causal Dimension Scale II (McAuley, Duncan, & Russell, 1992); the Survey of Achievement Responsibility (Ryckman & Rallo, 1983); and the Critical Incident Attribution Measure (Vispoel & Austin, 1995).

The SOAR instrument, referred to above and which is the focus of this paper, was developed by

Ryckman and Rallo (1983) and was one of the first instruments that incorporated the theoretical work of Weiner. The purpose of the instrument was to assess students' causal attributions for success and failure in school-related situations. The original SOAR instrument comprises 24 subscales covering three different school subject areas (math/science, language arts/social studies, and physical education), under two potential outcomes (success and failure), and indicating four causal dimensions (ability, effort, luck, and task ease/difficulty). The subject areas, outcomes and dimensions, respectively, produce these 24 subscales under the formula 3 x 2 x 4 = 24 (Ryckman, Peckham, Mizokawa, & Sprague, 1990). The authors' rationale for selecting specific subject areas is that they felt the causal attributions the respondent identified may be specific to one subject area (Ryckman et al., 1990). The original instrument comprised 40 brief hypothetical scenarios with four randomly ordered, plausible causes, for a total of 160 items. Subscales were measured by 64 (success - 32, failure - 32) of the 160 items for math/science, 64 (success - 32, failure- 32) of the 160 items for language arts/social studies, and 32 (success-16, failure-16) of the 160 items for physical education (Mizokawa &

Ryckman, 1990). The reason why physical education was measured by a reduced set of the items is that

(3)

Adaptation, Analysis and Critique ofthe Survey ofAchievement Responsibility (SOAR) foruse in the JapaneseSLA Context 165

for this subject area most people attribute their performance outcome to either task difficulty or luck (Mizokawa & Ryckman, 1990) and therefore, only these two scales were used.

The scoring was done by having the students mark only one of the four boxes representing the response for that cause. One point was then given to the selected cause and no points were given to the

causes that were not selected. The median reliability estimates ranged from a low of .24 for the 4-item physical education failure-task subscale to .89 for the total effort subscale. For a complete list of the reliability estimates (Cronbach's alphas) refer to Ryckman, Peckham, Mizokawa, and Sprague's (1990) article entitled, "The Survey of Achievement Responsibility (SOAR): Reliability and ValidityData on an Academic Attribution Scale" (pg. 269). The authors of the SOAR (Ryckman & Rallo, 1983) did not report the normality of the scores for items and they did not perform a confirmatory factory analysis

(CFA) on scores derived from the instrument.

The SOAR instrument has been used in educational research to study gender differences in attributions for success and failure across different subject areas (Ryckman & Peckham, 1987), and to

investigate if there are differences in attributions for academic success and failure across different

Asian-American ethnic groups (Mizokawa& Ryckman, 1990).

The findings in the gender study suggest that patterns of attributions do reflect gender differences.

In general, girls are more likely to choose effort attributions than boys and boys are more likely to choose ability and luck attributions than girls (Ryckman & Peckham, 1987). Moreover, girls tend to display a learned-helplessness pattern of attributions for mathematics and science to a greater extent than boys, but they both tend to be mastery oriented in the language arts (Ryckman & Peckham, 1987).

For clarity, learned-helplessness is defined as attributing success to unstable causes such as effort and/

or luck, and as attributing failure to stable causes such as ability or task difficulty whereas, mastery

oriented is defined as attributing success to stable internal causes and attributing failure to an internal,

unstable cause (Diener & Dweck, 1978, 1980; Wolleat, Pedro, Becker, & Fennema, 1980). However,

even though these patterns do exist, there was virtually no difference in test scores across subject areas

(Ryckman & Peckham, 1987).

The Mizokawa et al. (1990) study compared the causal attributions for academic success and failure

of Chinese, Filipino, Japanese, Korean, Vietnamese, and other Southeast Asian Americans to see if there were any differences between ethnic groups. They found that irrespective of ethnic background most Asians attributed their academic performance to effort rather than ability, and that Asians tend to

believe if they put forth more effort they can avoid future failure (Mizokawa & Ryckman, 1990). These findings provide support for Mordkowitz and Ginsburg's (1987) separate contention that there is a

relatively low level of learned-helplessness among Asian people.

This paper reports on the initial phase in adapting the SOAR into the Japanese SLA context, following

the ITC guidelines (Hambleton et al., 2005).

(4)

166 Todd Tournat

3.0 Methodology 3.1 Instrument

The adaptation of SOAR is an abbreviated version of the original instrument because only one subject area or domain is represented with this being the English oral communication experience of the student. The originalversion (Ryckman &Rallo, 1983),as stated above, included the domains of math/

science, language arts/social studies and physical education. The adaptation reported in this study restricts the domain to English oral communication. This restriction provides an appropriate level of

specificity for interpretive purposes.

Additional modifications included converting the categorical scales into ordinal scales. Under the original conception for the instrument, a scenario was presented and then four response items followed, each representing an attribution dimension, and the respondent had to choose only one of these (i.e.

choose one category). The difficulty with this is that the respondent's real disposition to the scenario may not be substantially categorical. Furthermore, and from a statistical point of view, problems emerged in subsequent analyses using the categorical scoring system because items were not statistically independent (Ryckman et al., 1990). In this study, each previously categorical option was responded to on a Likert scale incorporating 6 points of discrimination with the following semantic anchors; 6=strongly agree; 5=agree, 4=slightly agree, 3=slightly disagree, 2=disagree, and l=strongly disagree, in place of the one point single scoring system. This meant that every option was responded to, and the response on one item was not conditional to the response on another, meaning that items

could be treated as statistically independent.

The adapted version, tested in this study, comprises eight subscales representing one subject area (English oral communication), two outcomes (success or failure), and four causal dimensions (ability, effort, luck, or task ease/difficulty). The formula 1x2x4 = 8 expresses the subscale structure in simple form. The adapted version of the SOAR comprises 16 imaginary scenarios measured with four statements, which are randomly ordered, and which sum to a total of 64 items (see Appendix). This means that, with respect to the subscales, each is measured by 8 of the 64 items. Table 1 shows the

subscales and items for both the success and failure outcomes.

Table 1 Subscales and Items Indicating the Subscales for Success and Failure Outcomes

Subscale Success Items Failure Items

Ability 2b, 3b, 6b, 7d, 9a, 12b, 13c, 15a la, 4d, 5a, 8c, 10a, lib, 14d, 16a Effort 2d, 3c, 6a, 7a, 9d, 12d, 13d, 15b Id, 4c, 5c, 8b, lOd, 1Id, 14c, 16d Luck 2a, 3a, 6c, 7c, 9c, 12a, 13a, 15c lc, 4a, 5d, 8a, 10b, 11a, 14b, 16c

Task 2c, 3d, 6d, 7b, 9b, 12c, 13b, 15d lb,4b, 5b, 8d, 10c, lie, 14a, 16b Ryckman and Rallo's (1983) original instrument was developed in English. The purpose of this study was to adapt the instrument into Japanese and into the SLA context. Thus, methods recommended by the ITC (Hambleton et al., 2005) involving forward and back translation procedures were followed.

These procedures constituted the initial step in adapting the SOAR into the Japanese SLA context. A

(5)

Adaptation, Analysis and Critique ofthe Survey ofAchievement Responsibility (SOAR) forusein theJapanese SLA Context 167

near-native speaker of English did the forward translation into Japanese and a different near-native speaker of English did the back translation into English. Both of these near-native speakers of English were doctoral students who had some training in test construction. The back-translated version and the original English version were compared and a few inconsistencies were identified. For example, the nuance of the idiomatic expressions, "just my time" and "got caught on a bad day" could not be

satisfactorily translated into Japanese. The two near-native translators were consulted and modifications were made to deal with these problems. The author then concluded the modification

process, and proceeded to test the Japanese version of the instrument in the Japanese SLA context

using a large dataset.

3.2 Participants and Procedure

The dataset for this study came from 654 SLA students studying at four different universities in western Japan in the fields of English (n = 23), socialwelfare (n =15)), science (n = 31), education {n

= 112), law (n = 29), engineering (n = 258),medicine (n = 85), business (n = 30), communication (n

= 48), and Japanese (n = 23). Anumberof missing values were identified and 90records were deleted as a result. The statistical analysis was based on the data from the remaining 569 respondents. This did not systematically alter the properties of the sample because the missing values were not systematic.

The lack of systematic missing values was established by inspection. Age ranged from 18 to 24 years.

With respect to gender, there were 311 males and 253 females (5 participants did not indicate their gender) present in the final sample.

The respondents who volunteered for this study were asked to complete the adapted version of the SOAR instrument. Additional background information was also collected and included such things as;

age, major, academic year, and the date of administration. All responses were completely anonymous because no identifying information was collected and informed consent was given by simply filling out

the questionnaire. The time required to complete the responses was about 15 minutes.

The data was placed in a Microsoft Office Access 2010 database. IBM/Statistical Package for the Social Sciences (SPSS) software (Version 19.0) was used to determine descriptive statistics and the reliability estimates (Cronbach's alphas) for the scores. AMOS (Version 5.0.1) was used to conduct a Confirmatory factor analysis (CFA). The reason why a CFAwas conducted in addition to establishing the Cronbach's alpha was because onlya CFA can detennine the unidimensionality of scales (Gerbing &

Anderson, 1988).

4.0 Results

This section presents the study results in three parts according to convention. A table (Table 2) is

provided in the first part to show the descriptive statistics. In the second part, the reliability of scales

was tested using Cronbach's alpha. Finally, in the third part, a CFA was conducted to directly test the

four-factor model hypothesized in the scoring regime of the original instrument (Ryckman & Rallo,

1983).

(6)

OQ OO OO OO OO OO OO OO OO OO OO OO OO OO OO OO

OO *- —. 3> ts tC tO N3 Wt OO OO .W .A 4a i. ^4 l- iU iy Cr Jl 5C CT T> 5< >0 Cr -^ -l J- -- -J CX OO OO OO O^

*-

*-

CD

>c oc a> ao co

>a or a>

Do c3

>a oa 3>

an cD

>c oc o>

Do o3

>

CO CO CO CO

tO tO

CO Ol

«5 tO -J -J tO

<J1 cn CO to CO A. © 01

cn

*».

cn

•^

A CO tc

>^

a in bo b u

^

>- a 1- <o 00 'XI 00 s

A W W W A to CO

0

© C\3

«£> CO CT!

Ol O ^1 O ©

©© H^

ot ot

©y 3*

-*

^o

^-

©t ov oo

yD nC 0C 0C 00 0C 0C 00 >C 0O >C CT 00 SC

7> »— .&. 00 ^- co cn cn 00 i— co oj cn co

*- Ot OO

^-

^- CO

^-

^- H- A.

»t oa

>o iy is oi c5 )>

-^

<o

©

©

© cn <o (O

co cn vo

co

is ji -M cn cn Aw oi Co oo AC iw ui Of ff ic n^

-J

A © as QOVOCXOOCncOH-OOi—>COCnStOH^L)-' •—

h- if cr co oo ^. H- cn to to cn Ci o^ nc oc oo ot -c ©H oo •—

©©

©©

©©

©©

©©

©©

©©

©©

oo oo

CO tO tO CO CO CO CO CO tO tO CO tO tO CO tO CO CO CO CO tO tO

CO

co cn o a

•—

•—> to to

CO CT>

~-l t—•

•—

>C Ot OC OC nt Oa

>H -0 0J i.

©^

|.

£*

.0 0a '>

CO -<

l

>t o- OO tO

©C Ot O*

-C T>

CO CO CO CO OO Ot O>

—'

© i— >. o* 0t n5 cc t> to )o -N ^. <j o- oo Jt .a o^ to

MN NN NM tO NM MN Nt Ot OM Mt ON Mt ON

N ©o ©© ©Q ©p ©o ©p pp ©© ©© ©© ©©

>b a>

to to co co u

>—>

0 cn to co o>

cn

co co

*- cn co cn

©

tO

© tO Oi co o

to <£> 10 o to <5 cn

<y> •—1 cn co •—1 to ©

cn

to co to to to to to

O C>

JSk

•—•

©

1-

^ M Ji

Cn " (O O

CO

tfi

^ O UI

CO a. to cn cn

h- o 00 cn «j

m

to to to to CO

S

>S 3^

S

>&.

H 0>

CT CO ro CO o

» CD at 0) 3 co

ST 3 a.

D

<

5' 3 D co CO CD

$ 0) 3 Q.

c a. o

H to

£3

CT 05

(i) 3 to

« co

<o fD

§

w

w

«-♦■ r*

» 3-

=?. fD

CO 3"

CD P3 D. P CD

5. P r—

5"

3 7?

fD P3 a XT C fD P O 3"

(7)

e-r CD W 2.

CT

<<

en-

3* CD P C

t- r

3* O 00.

P

a a

_ CD

p-3 cr CT 3* fD P 3 a p

2. o

3 3

a. oo -3 p CT CD to

2.

CD rf

CO CD

>0 3.

CD o

o 3 C.

<i

<

30a w

?r CD P 3 a.

>r c

I

CO Q.

O 3 CD CT

<<

Q.

Ou 3*

CD 5«r CD P 3 Q.

?r c

iiL co

V< S*

P 3 CL 3" CD 3

CO CD Q.

7s C 3.

O S2. co'

fe o>

Oi Oi cr ic nc nc nc nj

^.

rf i.

^4 i- co co co co co co to tO H- 'h hh oo oo io

® >D D3 DO 3> OD >a D3 aO O> OC >O CO DO a> OC >D D3 DO O.

O

2:

cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn cn

cn iC CT Q) O) Tl )C C7 cn cn Cn !) iC nO lC CT O) Tl QC T> )C ^O ^C lC CT Tl ^C nC nc nc )< )0 CT 10 C7 0)

n O O^ O^ O^ OC O^ 0? ©< O< OS 0< 0* O* O^ O^ o^ O^ O^ ©s ©« ©^ o* o^ O^ OC 0* O< Ot Ot O< -V «&

AC OC OA CO UA CO CO AC OC O^

CO AC OA CO CO CO

^C OA CO AC OC OA CO CO

co co to co

aj U'

©c

^^

co Ji .t oc

-' bi Mc oi xc

«o

>c nt ob H(

»>

-b

s^ cT h- oo oo oo n^ ic jo ^a ^H to co c^ Oi TO aj to o^ cn j^ co ^^ oj no oc oo Sc

i

O

©

© O

©

©

© tO --J CT> •—' •—' OS

00

<£>

00 00 CO

*•

©

*.

©•

—t oc o>

>—

»—

©c

©

^O OM OO OC Ot SA OC

OA co to cn co co 0o o5 <i -- .H 4i

H-

Oi Ht OO tO OC Ot

OO OO CO ^- ^- A- OO ^J

O^

Ji jJ

Wi 00 i-

<C ff iW Sl tO Ol I .&. cn

H- A.

>—

CO OS

© A.

©

CO

©

© .to-

©

oo co cn cn co

Ot H H CO CO

I I I I I I I I

!-

©c oc oC Ts co H- co to a>

©>

—t oo

>t oS co co

>c nH -

CO CO CO

«D t—>

t£>

© o 1— to t^> O 5 ?

^^

l_>

© 00

© •—•

^)

<o cn to

c 1

*k CO

.to.

CO

© --J CO

*>.

cn as 2

^ 00

«£>

2

co

»

»- a to .f e.

co

*-

A- oo co co cn to oo cn

cn

>o cn

~j to as o

cn oo oi

—' cn as oc oc

n vl i- i— Ti nC OC OO 'C t- tO

-«]

£fc

£h I—' CO CO

>—'

—J

>&•

co

© i—>

A to CO

CO CI A

I I I

SA

*».

cn

crs cn co

oo

co co to co to to co to co tv ot oc oc oc oc oc ot oc ot oc ot oc ot ot oc oc ot ot ot oc o

go

©©

©©

©©

©©

©©

©©

©©

©©

©©

©©

©©

©©

©©

©© >to->to->to-»to-»to.>to->to.>to->to.>to. .to- .to. 4^ £t t> »C o- >t o- >t &. -> »b o. >t o- >t b- fe k. o> >t o. .t &- .. >& >&

£

(8)

170 Todd Tournat

Table 3 Calculated Values for Skew and Kurtosis

Test Item Skewness Kurtosis

Calculated Values Calculated Values

S01A *10.6 *6.4

S01B 1.4 2.6

S01C *4.1 .14

S01D *6.4 *3.7

S02A .10 *4.7

S02B *3.4 1.3

S02C *3.6 .45

S02D *3.3 1.9

S03A 2.9 2.9

S03B 2.5 1.1

S03C *5.9 .45

S03D *4.0 1.2

S04A 2.1 *3.4

S04B *4.0 1.0

S04C *6.9 3.0

S04D *7.5 *4.0

S05A *5.1 2.1

S05B *6.3 *3.6

S05C *4.3 .20

S05D .15 2.1

S06A *5.2 *4.1

S06B *3.6 .80

S06C *6.4 3.0

S06D *4.3 1.1

S07A *8.0 *7.5

S07B *3.8 1.5

S07C *6.0 1.1

S07D *3.3 1.4

S08A *4.4 .70

S08B *5.1 .92

S08C *5.7 2.1

S08D 1.9 .13

S09A 1.8 3.0

S09B 2.1 1.5

S09C 1.3 2.3

S09D .30 2.2

S10A *3.4 .01

S10B 1.5 2.8

(9)

Adaptation, Analysis andCritique oftheSurvey ofAchievement Responsibility (SOAR) foruseintheJapanese SLA Context 171

S10C *3.8 1.6

S10D *4.2 .30

SUA 2.5 *3.2

SUB *6.2 2.5

S11C 1.3 .50

SI ID *8.7 *8.9

S12A *3.4 .07

S12B *4.7 2.9

S12C 2.1 .35

S12D *5.9 *3.1

S13A 2.4 2.9

S13B *4.5 1.4

S13C 1.2 1.8

S13D *4.0 .94

S14A .26 1.6

S14B .62 *3.1

S14C *5.2 1.1

S14D .71 1.4

S15A 2.0 .25

S15B *3.8 .10

S15C .35 .83

S15D 2.4 .16

S16A *6.0 2.2

S16B 2.4 .25

S16C 1.8 .15

S16D ^ 13

"Indicate skewness and kurtosis

In terms of skewness, 39 items fell above the threshold and in terms of kurtosis, 12 items fell above the threshold. The originalauthors (Ryckman &Rallo, 1983) of the SOAR instrument did not report the normality of scores for items making up the original instrument. This presents an analytical difficulty

because it is difficult to determine whether the non-normal items in this adapted version are a result of

a similar tendency to be found in the original version by Ryckman and Rallo (1983), or whether this

non-normality occurred as a result of the adaptation process.

4.2 Reliability Estimates

Reliability estimates (Cronbach's alpha) for scores on the four subscales are available for inspection

in Table 4a and Table 4b.

(10)

172 Todd Tournat

Table 4a Reliability Estimates, Confidence Intervals for Alpha (95%), Scale Means,

and Scale Standard Deviations for Success Outcome

Scale Cronbach's alpha

95% Confidence Intervals for

Cronbach's alpha Scale

Mean SD for Scale Lower Bound Upper Bound

Ability .89 .87 .90 25.18 6.76

Effort .90 .89 .91 33.71 6.67

Luck .75 .72 .78 28.94 5.74

Task .81 .79 .83 32.50 5.40

Table 4b Reliability Estimates, Confidence Intervals for Alpha (95%), Scale Means,

and Scale Standard Deviations for Failure Outcome

Scale Cronbach's alpha

95% Confidence Intervals for

Cronbach's alpha Scale

Mean SD for Scale Lower Bound Upper Bound

Ability .80 .78 .82 34.59 5.43

Effort .83 .80 .85 34.60 5.60

Luck .73 .69 .76 25.73 5.53

Task .81 .79 .83 29.34 5.49

The criterion to determine scale reliability was based on Nunnally and Bernstein's (1994)

recommended value of .70. This is a widely adopted criterion in the literature. All four hypothesized

scales for both the success and failure outcomes produced alphas above this threshold of .70. However, the lower bound of the 95% confidence level for Luck in the failure table fell slightly below the threshold with a value of .69. Overall, the alpha values could be considered satisfactory against the preselected

criterion.

4.3 Confirmatory Factor Analysis

To directly test the four-factor structure hypothesized by Ryckman and Rallo (1983) for the SOAR instrument on both the success and failure outcomes, a CFA was performed. Since there was only one domain to be tested in this study the procedure was only done once. The model had 528 distinct sample moments, 70 parameters, and 458 degrees of freedom. This meant that the model was overidentified and suitable for a confirmatory test. The purpose of the CFA was to test for the unidimensionality of subscales, and thus the unidimensional model was specified meaning that error terms were not permitted to covary. Only the factors (constructs) were allowed to correlate, and this coincides with conventional practice within the social sciences where there is general agreement that psychological constructs are unlikely to be orthogonal (Kline, 1994).

Fit indices and the chi square test are typically used when CFA models are directly tested, and are

presently a form of conventional and best practice in adjudicating the fit of models and tend to appear in

(11)

Adaptation, Analysis and Critique ofthe Survey ofAchievement Responsibility (SOAR) foruse in theJapanese SLA Context 173

most empirical papers involving CFA. The reason for this is that the chi square test is sensitive to

sample size which can lead to Type I errors. However, the fit indices require a priori criteria to

determine whether the value derived for an index is acceptable. In order to do this, cutoffs/thresholds

have been recommended. Hu and Bentler (1999) have used empirical methods to derive thresholds which minimize Type I and Type II error, and these thresholds have been widely adopted in the field.

They are adopted in this study, as with most others, and are consistent with current best practice.

Four indices were adopted in the present study. These were the Tucker-Lewis index (TLI), the comparative fit index (CFI), the root mean square error of approximation (RMSEA), and the standardized root mean squared residual (SRMSR). These four indices represent a typical selection and routinely appear in empirical studies. The RMSEA is useful because it rewards model simplicity, and when the value derived for a model is marginal and close to the threshold, confidence intervals can

also be inspected.

The values obtained in this study for the success outcome were as follows with Hu and Bentler's

cutoffs presented in parentheses: TLI.82 (> .95), CFI.84 (> .95), RMSEA .07 (< .06), and SRMSR .09 (<

.08). The valuesderivedin this study forthe failure outcome wereas follows: TLI .74 (> .95),CFI .76 (>

.95) , RMSEA .07 (< .06), and SRMSR .09 (< .08). All values for both the success and failure outcomes did not meet the recommended thresholds offered by Hu and Bentler (1999). This indicates that the model hypothesized by the originating authors for the instrument is problematic under this adaptation

of the instrument in the present dataset.

5.0 Discussion and Conclusion

An analysis of scores for the adapted version of the SOAR instrument did not indicate that the

adaption process was fully accomplished, and there is empirical evidence that the structure

hypothesized by the original authors does not fit well in this dataset taken from the Japanese population using the adapted version of the instrument. However, the current author also recognizes that the scale structure was significantly altered in the process of conversion from categorical to ordinal scaling in each response scenario. On the other hand, the method of CFA was not employed by the originating

authors, and it is not known whether model fit would have been achieved under the categorical scaling system accompanying the original conception for the instrument. The data from 39 out of the 64 items was skewed, and 12 out of the 64 items indicated kurtosis.

Because Cronbach's alpha is not an index suitablefor determining the unidimensionality of scales (or subscales), a CFA was also conducted (Gerbing &Anderson, 1988),and as noted above this is the first time the method has been employed with respect to this instrument (to the best of the author's knowledge). All the valuesproduced by the CFA were outside Hu and Bentler's (1999) recommended

cutoffs/thresholds for the indices which were selected for this study which means that the data in this

study did not fit the four-factor oblique model. However, CFAs were not that common in the 1980sso it

is difficult to determine if the original SOAR instrument (Ryckman & Rallo, 1983) was flawed, or

whether the unsatisfactory results reported in this study were mostly a product of the adaptation

(12)

174 Todd Tournat

process. Only the reliability estimates (Cronbach's alpha) produced results which could be considered satisfactory as these were all above Nunnally and Bernstein's (1994) criterion of .70 (except for the lower bound of the 95% confidence level for Luck on the failure outcome which was just below the threshold with a value of .69). While these results for alpha are satisfactory, it has to be said (and this has been stated above as well) that alpha does not provide an indication of unidimensionality which is

what is at issue. In this respect, the results for the CFA have to be deferred to.

Even though these results are unsatisfactory overall, the data obtained can be used as an empirical

guide for what needs to be done in the next phase of the adaptation process of the SOAR instrument into the Japanese SLA context. The most important issue for subsequent phases of research is what

needs to be changed in the items making up the instrument. Such determination could lead to further

revision and a version which can again be empirically tested on a new dataset.

One area which presented problems with respect to many of the items was the skewed and kurtotic distribution of the scores on these items. There were 39 items that displayed skewed data and 12 items that displayed kurtotic data. This means that very little information was obtained from these items and that the problem could have been a result of the wording of the scenario and/or the wording of the

responses. One way to deal with this is to elicit the help of other language professionals and carefully look at what in the wording may be causing the skewed and kurtotic distribution. Another, more drastic solution, may involve completely replacing some of the items.

Another solution to the problem of skewness which could be considered would be to adjust the Likert scale from six response-points to ten response-points. The rationale for extending the scale to ten

response-points is to make the subscales more sensitive to the respondent's disposition which would

then make items less likely to generate scores with a skewed distribution. The difficulty which has to be

considered in this approach relates to whether the respondent can actually make such fine discriminations in their own disposition. If they cannot, then forcing them to respond on a more refined

scale can introduce further error into the response scores.

Once any adjustments have been made, they can be empirically tested in a new dataset on a different

sample of Japanese university students to see if the skew and kurtosis problem was sufficiently dealt

with, and if a good overall model fit was achieved in a CFA. This would then constitute the second

phase of an ongoing research trajectory directed at arriving at an instrument which generates valid and

reliable scores in the Japanese population. Furthermore, this trajectory will have been both

theoretically and empirically informed offering confidence to researchers and practitioners. This

process would be consistent with good research practice and consistent with contributing to a sound

psychometric foundation for the research and practitioner interest in causal attributions within the SLA

context in Japan, and how these constructs might impact on SLA outcomes.

(13)

Adaptation, Analysis andCritique ofthe Survey ofAchievement Responsibility (SOAR) forusein theJapanese SLA Context 175

References

Banks, M., & Woolfson, L. (2008). Why do students think they fail? The relationship between attributions and academic self-perceptions.

British Journal of Special Education, 35

(1), 49-56.

Diener, C. I., & Dweck, C. S. (1978). An analysis of learned helplessness: Continuous changes in

performance, strategy, and achievement cognitions following failure. Journal of Personality and Social Psychology, 36

(5), 451-462.

Diener, C. I., & Dweck, C. S. (1980). An analysis of learned helplessness II: The processing of success.

Journal ofPersonality and Social Psychology, 39

(5), 940-952.

Gerbing, D. W., & Anderson, J. C. (1988). An updated paradigm for scale development incorporating

unidimensionality and its assessment. Journal ofMarketing Research, 15,186-192.

Gobel, P., & Mori, S. (2007). Success and failure in the EFL classroom: Exploring students' attributional beliefs in language learning.

Eurosla Yearbook, 7

(21), 149-169.

Graham, S. (1997). Using attribution theory to understand social and academic motivation in african american youth. Educational Psychologist, 32 (1), 21-34.

Hambleton, R. K., Merenda, P. F., & Spielberger, C. D. (2005).

Adapting educational and psychological tests for cross-cultural assessment. Mahwah New Jersey: Lawrence Erlbaum Associates.

Heider,F. (1958). The psychology of interpersonalrelations. NewYork: Wiley.

Hsieh, P. P., & Kang, H. (2010). Attribution and self-efficacy and their interrelationship in the Korean EFL

context. Language Learning,1-22. doi: 10.1111

Hsieh, P. P., & Schallert, D. L. (2008). Implications from self-efficacy and attribution theories for an

understanding of undergraduates' motivation in a foreign language course. Contemporary Educational Psychology, 33,513-532.doi: 10.1016

Hu, L. T, & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:

Conventional criteria versus new alternatives.

Structural Equation Modeling, 6

(1), 1-55.

Kline, P. (1994). An easy guide tofactor analysis. London &New York: Routledge.

McAuley, E., Duncan, T. E., & Russell, D. W. (1992). Measuring causal attributions: The revised causal dimension scale (CDSII).

Personality and Social Psychology Bulletin, 18,566-573.

doi: 10.1177 Mizokawa, D.T, &Ryckman,D. B. (1990). Attributionsof academicsuccess and failure: Acomparison of six

Asian-American ethnic groups.

Journal of Cross-Cultural Psychology, 21

(4), 434-451.

Mordkowitz, E. R., & Ginsburg, H. P. (1987). Early academic socialization of successful Asian-American college students. Quarterly Newsletterof the Laboratory of ComparativeHuman Cognition, 9 (2),

85-91.

Nunnally, J. C, & Bernstein, I. H. (1994).

Psychometric theory

(3rd ed.). New York: McGraw-Hill.

Peacock, M. (2010). Attributionand learningEnglish as a foreign language. ELT Journal, 64 (2), 184-193.

Ryckman, D. B., & Peckham, P. (1987). Gender differences in attributions for success and failure situations across subject areas.

Journal of Educational Research, 81

(2), 120-125.

Ryckman, D. B., Peckham, P. D., Mizokawa, D. T., & Sprague, D. G. (1990). The survey of achievement responsibility (SOAR): Reliability and validity data on an academic attribution scale. Journal of

Personality Assessment, 54

(1,2), 265-275.

Ryckman,D. B.,& Rallo, P.H. (1983). Surveyofachievement responsibility. Seattle: Authors.

Stipek, D. J., & Weisz, J. R. (1981). Perceived personal control and academic achievement. Review of

(14)

176 Todd Tournat

Educational Research, 51 (1), 101-137.

Vispoel, W. P.,& Austin, J. R. (1995). Success and failure in junior high school: A critical incident approach to understanding students' attributional beliefs. American Educational Research Journal, 32 (2),

377-412.

Weiner, B. (1979). A theory of motivation for some classroom experiences. Journal of Educational

Psychology, 71,3-25.

Weiner, B. (1985). Anattributional theory of achievement motivation and emotion. PsychologicalReview,

92

(4), 548-573.

Weiner, B. (2010). The development of an attribution-based theory of motivation: A history of ideas.

Educational Psychologist, 45,28-36.

Weiner, B., Frieze, I. IL, Kukla, A., Reed, L, Rest, S., &Rosenbaum, R. M. (1971). Perceiving the causes of

success and failure.

Wolleat, P. L., Pedro,J. D., Becker, A. D., & Fennema, E. (1980). Sex differences in high school students' causalattributionsof performance in mathematics. J. Res. Math. Educ, 11 (5), 356-366.

Appendix

Surveyof Achievement Responsibility (SOAR)

Directions: In the following questions, imagine that each of the things happened to you in your English oral communication class. Circle one number for each of the statements that best fits why you think that might have happened.

1. You failed to answer many questions during your English oral communication task. This might, happen because:

Strongly

Agree Agree Slightly Agree

Slightly

Disagree Disagree Strongly Disagree I just can't, seem to do well at English oral

communication 1 2 3 4 5 6

The questions were too hard for most people 1 2 3 4 5 6

There were too many tlungs happening that day 1 2 3 4 5 6

I probably didn't work hard enough 1 2 3 4 5 6

2. On your English oral communication task, you see that the teacher gave you a good mark. The likely reason this happened is that:

Strongly

Agree Agree Slightly Agree

Slightly

Disagree Disagree Strongly Disagree

It must have been my day 1 2 3 4 5 6

English oral communication tasks are easy for me 1 2 3 4 5 6

It. was so easy, nobody had trouble 1 2 3 4 5 6

I worked a long time on the task 1 2 3 4 5 6

3. On a weekly English oral communication test., you got a very high score. That, might be because:

Strongly

Agree Agree Slightly Agree

Slightly

Disagree Disagree Strongly Disagree

I got lucky on that test 1 2 3 4 5 6

I have always been good at English oral

communication 1 2 3 4 5 6

I worked hard on the English oral communication

tasks in preparation for the test 1 2 3 4 5 6

Table 1 Subscales and Items Indicating the Subscales for Success and Failure Outcomes
Table 3 Calculated Values for Skew and Kurtosis
Table 4a Reliability Estimates, Confidence Intervals for Alpha (95%), Scale Means,

参照

関連したドキュメント

This paper develops a recursion formula for the conditional moments of the area under the absolute value of Brownian bridge given the local time at 0.. The method of power series

Amount of Remuneration, etc. The Company does not pay to Directors who concurrently serve as Executive Officer the remuneration paid to Directors. Therefore, “Number of Persons”

In case of any differences between the English and Japanese version, the English version shall

In case of any differences between the English and Japanese version, the English version shall

In case of any differences between the English and Japanese version, the English version shall

In case of any differences between the English and Japanese version, the English version shall

In case of any differences between the English and Japanese version, the English version shall

In case of any differences between the English and Japanese version, the English version shall