The present study investigated the effects of computer delivery mode by comparing monologic tasks delivered in the computer and face−to−face modes from

CHAPTER FIVE

DISCUSSION ANI）CONCLUSIONS

     The present study investigated the effects of computer delivery mode by comparing monologic tasks delivered in the computer and face−to−face modes from

three perspectives， namely， test score， speech sample， and examinee attitudes toward the

test−taking conditio ns． This chapter summarize s the major fmdings o f the study and

discusses the relevant issues in respect to each of the research questions， and this is

fbllowed by a general discussion of the issues related to the findings． Implications fbr

language assessment， test development， and SLA research are discussed based on the

fmdings ofthe study． Finally， the chapter concludes with a discussion ofthe limitations

ofthe study and suggestions for fUrther research．

5．1 Summary of results and discussion

5．1．1 Score comparability

Researchρuestionゐ・4re there differences in the magnitude（Z∫舵ぷcorεM∬igned to

／Oρanese、EF乙learners 1戊eηfomzance 1戊roduced in〃ionologic tasksαcroぷぷdeliveりノ

modeぷ（ computer VS．プbce−toプCaceジ2

     This question was answered by comparing test scores derived from ratings on the

analytic scale． Given that a practice effect was detected fbr the group that took the

computer mode first， between−subject t−tests were conducted on scores obtained from

the frrst test and the second test administered separately． Only the results on the first test

were adopted， which showed that no significant difference across delivery mode was

fbund either in mean scores fbr the rating elements on each task or in total scores． This

suggests that there was no delivery mode effect on the magnitude o f test scores awarded

at either the rating element level， i． e．， grammar， vo cabulary， fluency， and pronunc iation，

or test level indicated by total scores．

      The findings concurred with the expectation that differences across modes

would not be manifest at the level of raw scores． One interpretation is that the

examinees did perfbrm similarly across modes， which may have led raters to assign

similar ratings to examinees perfbrmance in both modes． In that case， the findings of

this study that were based on an analytic scale were consistent with those of studies on

tape−based speaking tests， in which significant differences were not found in ho listic

scores either（Shohamy，1994；Kenyon＆Tschirner，2000）． Taking these findings

together， it seems that delivery mode is not a factor affecting the magnitude of scores，

regardless ofthe type ofrating scale used． However， bef（）re jumping to this conclusion，

more empirical evidence is needed to confirm the results ofthis study．

     Another possible explanation of the lack of significant differences in the

magnitude of raw scores across modes is that the examinees may have perfbrmed

differently across modes， but not to an extent that raters were able to discern to assign

different ratings． In fact， the lack of score sensitivity to variation in task has been well

documented in language testing research． Fulcher（2003）， after reviewing several studies，

concluded that studies（Chalhoub−Deville，1995a；Upshur＆Tumer，1999）that have

found large significant perfbrmance differences across tasks are only those where the

tasks were maximally different and multiple rating scales f（）r tasks were employed．

Chalhoub−Deville， fbr instance， compared a modified OPI， a picture narration task， and

areading aloud task． Upshur and Turner used a story retell㎞g task and an audio−parラ

task in which examinees sent a recorded message to an exchange student．

     In the present study， tasks compared differed only in terms of delivery mode． In

addition， the same rating scale was used fbr tasks delivered in the computer mode and in

the face−to−face mode． Taking these factors into acco皿t， the unobserved differences in

raw scores would not seem surprising． However， to support this interpretation， a closer

examination of how the actual speech samples differ across modes is essential． This

issue will be addressed in the discussion in relation to Research Question 3． Any

disparity that may emerge from the analyses oftest scores and speech samples will be

discussed in the general discussion section．

Research 2uestion 2． For lapanese EF．乙 1eamers， to what extent are the constructs

Z）eing  〃1〈ヲasured  Zリノ monologic tasks dOゲferent or 5∫〃z ilar acroぷ5 deliveりノ 〃z odeぷ

（co〃zputer vs．プbce−to−faceジ2

     Exploratory factor analyses were perfo rmed to explore the psychometric constmcts measured by monologic tasks delivered in the computer mode and the

飴ce−to−face mode． Results revealed that there was a single factor for each mode， as well

as fbr the combined data fヒom both mode． Further， the observed variables representing

the element scores on each task all loaded highly on the single factor， and their factor

loadings generally showed a similar pattem across modes． These findings imply that

mono logic tasks delivered in the computer mode and the face−to−face mode measure the

same underlying constnlct．

     Then， what kind of speaking ability might this single factor represent？In other

words， what do the monologic tasks delivered in the two modes measure in the present

data？The results ofthe analysis of speech samples on lexical density in Section 4．3．3．3

shed some light on this question． The lexical density figures showed a very small and

consequently insignificant difference between the cornputer mode（62％）and the

飴ce−to−face mode（63％）． This lack of significant di脆rence in lexical density figures

suppo丘s the血ding from exp loratory factor analysis that there is a common factor

shared by the two modes．

     Also， compared with the literature， the magnitude of the lexical density f（）r both

modes was quite high． Ure（1971）fbund that the spoken texts had a lexical density of

less than 40％and the written texts had a lexical density greater than 40％． Thus， the

speech samples elicited by monologic tasks in both modes in the current study could be

considered as being similar to written language or monologic discourse， which

represents a mono logic speaking ability． Further， the high figures on lexical density also

suggest that a similarly low degree of interaction was involved in monologic tasks

delivered in the two modes．

     The findings regarding the underlying constructs are， however， not congnlent

with the prediction that monologic tasks delivered with computers measure monologic

ability， while those in the face−to−face mode tap interactive ability． This prediction has

been based on the fhldings of O Loughljn （2001）that lexical density figures on

monologic tasks between the tape−based and the face−to−face modes were significantly

different． Contrary to the suggestion made by O Loughlin， the simple feedback

provided by the interviewer does not seem to result in a certain degree ofinteraction．

     The disagreement with the results of O Loughlin（2001）may be attributable to

the methodology of the present study． It is important to note that the lexical density

figures in the study ofO Loughlin were compared across modes based on the rank order

of the median scores due to the small sample size（η＝20）． Thus， the results of the

present study， which were obtained from a comparison o f mean scores of lexical density

figures with a larger sample size（n＝79）， may be coml）aratively more reliable．

5．1．2 Speech sample comparability

Researchρuestion 3：1b｝吻t extent do lapanese EE乙learners vary in their sρeech

ぷamples」りroduced加monologic tasks acro∬delivery modes・（Kcomθuter vs．ノbce−toプcac（…ジ

oη〃leasureぷ（〜力侮θη（ッ，αcc醐（ッ， and complexity2

     This issue was investigated mainly through discourse analysis using qualitative

measures． Results were then submitted to statistical analyses．

     As fbr fluency， ANOVA results showed a main effect of delivery mode on the

measures of dysfluent words and filled pauses． A closer look at dysnuent words

revealed that the differences emerged only in repetition． Specifically， examinees made

more repetitions with the face−to−face mode but used more filled pauses with the

computer mode．

     The results concerning repetition were not consistent with the prediction that

examinees would prioritize meaning over accuracy and complexity in the presence ofa

listener， resulting in higher fluency in the face−to−face mode． This finding may have

been due to examinees higher level ofnervousness when facing the interviewer， which

made them hesitant and consequently caused them to use more repetition words．

     This may be explained with reference to Exarnple l fbr the opinion task． The

examinee in this example is a female student who left a very deep impression on the

author． She seemed very nervous in the face−to−face test with her vo ice trembling． Her

responses to the questionnaires go someway to confrrming this． In answering the

question of which test did you feel more nervous taking ， she chose the face−to−face

test． In responding to the statement of I fe lt nervous when I was tak廿1g the test ， she

cho se   StronglJ／ agree   f〜）r  the  face−to−face  test  but   1＞診utrar  fb r  the

computer−delivered speaking test． Her speech sounded so dysfluent that it was quite

difficult to understand． An examination of the transcripts shows that she did use more

repetition words in the face−to−face mode， whereas her speech in the conlputer mode

was more f［uent without so many repetition words． The repetition words are underlined．

Example l

Face−to−face test

Iagree the opinion． Um血the students every day must read newspaper um um

because because many student get the material on intemet． But telebi or intemet is

not true sometimes． Um um㊤一the newspaper true sometimes not

not true but um but mo re mo re true mo re true more true um mo re true inf（）rmat io n

can get I thhik． Um um um more more over um can um can more over more small more more smaller uh read the letter read the letter． In in um in recent in recent days並逗it s more small more srnaller uh read letter． Um so it is it is important to read the newspaper um to read better 1 think．

Computer−delivered test

Iagree in this opinion because TV or Internet 1mnany−s1！uggg！an st dent many student get

information TV or Intemet． But TV or Internet一㎜ny

infbrmation is not true I think． Um the newspaper is less is less not tnue information． And the student can get ability to read to read letter． Many student recently less read read books． And and I thml（it is I think less ability to read letter．

So I thhik student student reading newspaper is good．

     Another illustration of this possible explanation is provided in Example 2 fbr the

opinion task． This is an example from a high school male student who also chose the

face−to−face mode as the one more anxiety−provoking． A close inspection of the

transcripts revealed that he also used more repetition words in the face−to−face mode．

Although this may be not very obvious through eyeballing the transcripts given the

different number ofwords in each test， he actually used over two times more repetition

words per second in the face−to−face mode（ll．04）than in the cornputer mode（4．05）．

Examp le 2 Face−to−face test

Ithmk student don t have to read the newspaper every day because we can we can get inf（）rmation from another way for example TV lntemet and conversation with with someone and so on（2）． I think（6）we should um we should we should get infbrmation（10）． Someone someone is not interested in social infbrmation（4）．

but if（7）but without infbrmation uh I』the people the person the people may

（21）旦旦y㎜yanyway I thmk些皿we had better uh we I think we should we

should get infbrmation． But we don t have to we don t have to read newspaper．

Comlワu旋ヲr−delivered tEヲst

       ロ

Ithmk塑we don t have to read the newspaper every day because we can get

info rmat ion liilIt；1mm ffo m ano ther way fo r examp le TV Internet conversat ion and so

on． but 1 thnk it s not good． That s there are some people who are not interested in infbrmation social infbrmation（6）． I think it is it is important（2）to get inf（）rmation．

Examples 3 and 4 present the transcripts of the narrative task from another two

examinees that indicated feeling more nervous in the face−to−face test． As shown in the

fb llowing transcripts， while the examinees used repetition words hl several places in the

face−to−face test， they had no repetition（Example 3）or only one（Example 4）in the

computer−delivered speaking test． This difference in their speech across modes was

quite impre ssive， given the relatively fewer words they spoke in the face−to−face test．

Example 3

Face−toヲ〔bce・teぷt

Iparticipated in a speech contest． Um LI gΩΩd good I did good魎g speaking．

But uh I lose the speech contest． Um but旦』um my teacher happy to my

speaking job．⊥um I will um try to next speech contest．

Co〃zputer−delivered・test

Iparticipated in a speech contest． I did a good speaking in English． But I lose a speech contest． I next to a man is a won． Uh let me see． My teacher is said uh my teacher said you your speakmg is good． I hapPy to this advice． I try it to next speech contest．

Examp le 4 Face−to−face test

LI going to speaking contest． I speaking y亘y y壁y y里y very story slowly． But I will I will I will（3）1（4）1m！y my best speak． But（2）1（2）Iwas I was（3）um I was sad（3）Iwas sad．

Co〃4puter−delivered tes・t

Today is speech contest． I speaking English very well． But I don t won glannpuri．

But（2）George was coming after a contest speech contest． You speaking English very nice he said（6）． L』口y（2）IhapPy now．

     Two characteristics ofthe participants in this study may be responsible f（）r their

higher level ofnervousness in the face−to−face test． First， Japanese learners， compared

with those of other nationalities， seem to pay more attention to others opinions and

have a higher level of communicative anxiety（Klopf， 1984；McCro skey， Fayer，＆

Richmond，1985）． Hill（1998）also fbund that examinees with Asian language

backgrounds reported feeling more nervous in interview tests more丘equently than expected． Second， being high school and university students−mostly freshmen and

sophomores−they were likely to care even more about what others thought about them

and thus fe lt more nervous when talking in front of a stranger than would the adult

learners in most previous studies． Another viable explanation is that， as noted by

O Loughlin（1995）， examinees may have assumed the assessment was being carried out

at the time of the test by the interviewer in the face−to−face mode． Had the examinees

been infbrmed in advance that the interviewer was not acting as the rater， the results

might have been different．

     With respect to filled pauses， the results corroborate the expectation based on the

learner orientation of listener sensitivity（Orgeta，2005）as well as the research on

tape−based tests（Luoma，1997；Koike，1998）． That is， examinees used filled pauses

more fヒequently in the co叫）uter mode． Two potential explanations could be offered for

this finding． As suggested by Ortega（2005）， when an authentic listener is present，

examinees may attempt to avo id stopping during the speech to avo id creating confUsion

f（）rthe listener．

     An altemative explanation may lie in the supportive fUnction ofback−channeling

behavior on the part of the interviewer．0 Loughlhl（2002）， fbr instance， fbund that

backchamels廿om interviewers encouraged examinees to continue speaking by

providing a signal to show active listening． More recently， Wolf（2008）showed that

Japanese learners were more fluent with verbal English backchannels廿om listeners

than the condition without． Wolf fUrther argues that the丘nding may only apply to the

Japanese who use more backchannels in their native language and may thus attach a

high value to such listener behavior． On the basis of these findings， it seems that

backcha皿els丘om the interviewer in the present study may have helped Japanese

participants maintain the fbw of speech． On the other hand， examinees may have felt

pressured when there was no reaction fro m the computer and may thus have used filled

pauses to fill any blanks during the test．

     Here， two examples from different examinees are provided：Example 5 fbr the

narrative task and Example 6 fbr the opinion task． As shown in the following transcripts，

in both cases， the examinees used more filled pauses （underlined） in the

computer−delivered speaking test than in the face−to−face test． The examples show how

the interviewer used backchamels（in bo ld face）to encourage the exa血nees to continue

the idea they are trying to express．

Example 5

Face−to：face test

Today is speech contest．1（2）1 speech English is U1mL famous everyone mm hm

（5）．But（5）not but no collaboration mm hm mm hm（8）． But not I I haven t（3）

haven t trophy mm hm（4）but．

Co〃zρuter−c〕elivered tes t

Today is speech contest． I speaking English（8）．1．ILm 1 speak is U1mL very good（3）．

Winner is next旦h wi皿er is（2）golden hair man（3）． I don t have trophy． But I

（2）mvited hl teacher（5）旦1n next speech contest．

Example 6

Face−to−face test

Idisagree the opinion that students should read mm hm newspaper because almost seventy five percent people mm hm get social infbrmation from television or intemet． mm hm旦坦only twelve percent are getting丘om newspaper． mm hm And（2）television and intemet can旦坦can mm hm watch

or use any places． But newspapers should buy fヒom like a convenient store． mm hm mm hm so it is not so usefU1 for students mm hm．

Co〃rputer−delivered test

First I I read this opinion I ve got one question why the students have to read

newspapers because television and intemet had enough旦h infbrmation堕1 as newspaper So they don t need to read newspaper Because because newspapers

are旦＿m twelve percent． And television are sixty six percent． So（5）so（3）so my opinion is that television and intemet is（2）旦坦is．

As discussed above， we seem to be able to provide plausible explanations fbr the

findings on repetitions and filled pauses separately． However， it is rather difficult to

interpret the results when taking them together， as they showed opposite tendencies

from the perspective of fluency． That is， examinees were more fluent in terms of

repetitions in the coMl）uter mode， but spoke more且uently in terms of the number of

filled pauses in the face−to−face mode． These mixed results may be due to different

dimensions of fluency that these two measures represent， i．e．， repetition f（）r hesitation

pheno mena and filled pauses for the temporal dimension． Certainly， empirical evidence

through factor analysis is needed to corroborate this interPretation．

     In contrast with fluency， measures of accuracy do not show any significant

difference across modes． These results丘om analyses using general measures of

accuracy are congment with those of previous studies using specific measures， which

found no difference in accuracy between the tape−based and face−to−face tests either

（Shohamy，1994；Ko ike，1998）． These findings seem to suggest that examinees perfbrm

similarly in terms o f accuracy， irrespective o f the condition ofbeing with or without an

㎞terlocutor．

     Similar to accuracy， none of the cornplexity measures reveals any significant

difference across modes， indicating that the two modes produced similarly complex

speech samples． However， the results regarding lexical density are contradictory to

O Loughlin（2001）， in which a significant difference in lexical density figures was

fbund between the tape−based and the face−to−face test． One explanation fbr the

disparity in these findings may be that the present study uses tasks that are more parallel

in various aspects across modes．

     Taken broadly， the findings ofcornparing speech sarnples only partially supported

the prediction that the absence of an interlocutor may make examinees prioritize

accuracy and complexity over fluency． As such， it did not support the trade−off effects

of the three aspects of perfbrmance proposed by Skehan（1998）． However， it is

important to note that the prediction was based on evidence from task−based research

conducted in instnlctional settings． As argued convincingly by Elder and Iwashita

（2005），examinees in a test situation may fbcus more on display to have their ability

assessed， rather than on getting the message across． Iwashita et al．（2001）also made the

suggestion that it is possible that a fbcus on accuracy may be paramount in the testing

situation regardless ofthe condition under which the task is perfbrmed （p．431）． By the

same token， the lack of difference in complexity may also have resulted丘om

examinees awareness that the廿speech was being evaluated， making them reluctant to

take risks beyond what they could say properly． Overall， the fmdings l of the present

study highlight the need to be cautious in apPlying task−based research results to test

Sltuatlons．

Research （〜uestion 4： 1b what extent does the ｛7ffe（三t（〜∫delivecソ〃20de （computer vぷ．

face−to−face？on measures（〜ffluency，αcc醐（ッ， and comjワlexity（7f speech sa〃zpleぷ仇

〃zonologic tasks vaりノaccording to the 1フrOL17cien（ツlevels oflal）αηθ5θEF，乙1eamers 2

     ANOVA results showed that it was not possible to establish any interaction effect

of proficiency level with delivery mode on any measure of f［uency， accuracy， or

complexity． This suggests that no examinees in any ofthe different proficiency groups

perf（）rmed differently in terms of linguistic features between、the computer and

飴ce−to−face modes． This finding does not agree with the hypothesis that speech samples

of examinees of lower proficiency may be more affected by the change of delivery

mode． This may be due to the limited range ofproficiency levels ofparticipants in this

study． In other words， it is possible that the oral proficiency of examinees assigned to

the lower level was not low enough． Recruiting participants from among high school

students in lower grades or junior high school students may lead to the detection of an

interaction between delivery mode and examinees proficiency． It should also be noted

that the criterion for grouping the participants was the scores obtained from the ratings

on the two computer−delivered monologic tasks used in this study． Using a speaking

proficiency criterion from another established speaking test may produce a different

outcome． Nevertheless， the unobserved interaction between delivery mode and

examinees proficiency is good news fbr the test validation， suggesting that the test is

not biased toward examinees ofdifferent proficiency levels．

5．1．3 Examinee attitudes

Res〈〜arch 2uestion 5：1b what extent do、ノOPanese EF，乙learners va（ソin their attitudes

toward〃ionologic tasks acro∬4θ1ive7夕〃iodes （co〃rputer vs．プbce−to−face？ 2

     This question was explored by asking examinees to respond to Likert−type

statements on the two modes， as well as to compare the two modes d廿ectly． Comments

on the open−ended questions were also collected and analyzed to shed light on

examinees Likert−type responses． In general， it was fbund that examinees rated both

modes quite positively on various aspects． The fbllowing summarizes and discusses the

findings ofdirect comparisons ofthe two modes in each aspect．

     Examinees reported the computer mode to be less anxiety−provoking than the

飴ce−to−face mode． These results are consistent with those in Sapsirin（2007）but at odds

with those in Kenyon and Malabonga（2001）， in which the COPI and the OPI were

fbund to be similarly anxiety−provokmg． Note that participants in Sapsirin（2007）were

Thai university students who shared an Asia background with those in this study，

whereas participants in the study by Kenyon and Malabonga were American leamers．

As discussed earlier in relation to Research Question 3， examinees with Asian language

backgrounds seem to feel more nervous in face−to−face tests（Hill，1998）． The different

profiles of the participants may help to explain the discrepancy of the fndings with

Sapsirin（2007）． Another explanation is that the adaptive nature of the COPI may have

made examinees feel more relaxed， compared with the less flexible computer−based

type of speaking tests used in this study as well as in Sapsirin（2007）．

     Comments fro m examinees show that the presence o f the interviewer is the main

reason for their higher level of anxiety in the face−to−face mode． Interestingly， those

who fe lt more nervous in the computer mode also mentioned the presence of the

interviewer， but in this case， they fbcused on her helpfUl role in making them feel

relaxed． Therefore， while a tendency f（）r examinees to feel less nervous with the

computer mode was fbund， it is worth fUrther investigating the issue on the individual

level by exploring which characteristics o f the examinees determined the different level

ofanxiety aroused by the two delivery modes．

     In addition to their nervousness， participants were asked to evaluate and compare

the two modes from three aspects， namely， test difficulty， test fairness， and test validity．

First， participants perceive the computer mode and face−to−face mode to be equally

difficult． These results are consistent with those in Kenyon and Malabonga（2001），

which found no difference between the difficulty of the COPI and the OPI． As many

examinees pointed out， this may be due to the design ofthe present study， in which the

same task contents and testing procedures were used across modes． The examinees

perception of test difficulty also corresponds to the results of raw score comparison，

which revealed no statistical differences in the magnitude oftest scores across modes．

     Second， it was found that the fairness of the speaking tests delivered in the two

modes was also evaluated similarly． These results are not congruent with those in

Kenyon and Malabonga（2001）， in which the OPI was considered to be a飽irer test than

the COPI． As some examinees commented， agam， e卿loymg the same task contents

and testing procedures fbr the two modes in the present study may be responsible fbr the

disparity of the findings． In this regard， fUture studies may use different task topics

across modes after validating their equivalence through expert judgment or ideally

through a pilot study． In this way， when responding to the questioma仕e， examinees

would focus their comments more on the delivery mode than on other aspects of task．

Nevertheless， some participants who chose the computer mode as the fairer test did

express their concems over the influence of the interviewer in the face−to−face test．

Others worried about the inflexible nature ofthe computer−delivered test， in which they

could not ask questions concernhlg the testing Procedures．

     Third， as fbr the validity in testing speaking， the face・・to−face test was considered

as a better measure ofreal−lifb speaking skills． The primary reason given by examinees

was that the飽ce−to−face test was similar to a real communication situation because of

the presence of the interviewer． It seems that the fact that oral communication is a

human phenomenon was paramount in examinees minds， which has led them to give a

higher evaluation to the validity of the test delivered in the face−to−face mode． In

addition， some examinees were concerned that their non−verbal communication， such as

gestures and facial expressions， could not be captured by the computer． These findings

are consistent with those in Kenyon and Malabonga（2001）， in which the OPI was also

perceived to be a more valid test than the COPI as a measure oftesting speaking ability．

It then seems that regardless of the type of test（i．e．， computer−based or

conlputer−adaptive）， fbr examinees， the computer−delivered speaking test could not

replace the face−to−face test that involves real human interaction．

     Finally， when asked which type o f speaking test they liked better， most examinees

favored the face−to−face mode over the computer mode． Also， the majority of the

examinees cho se the face−−to−face mode as the test they would prefer to take in the fUture．

Examinees gave similar replies to these two questions， which were related to the

presence of the interviewer． That is， they felt less nervous and more natural taIking in

the face−to−face mode． The similarity o f the face−to−face test with a real communication

situation was another main reason for their preference o f the face−to−face test．

     From the discussions so far， it is interesting to note that although a tendency

toward a higher level o f anxiety in the face−to−face mode was found， the examinees who

preferred the face−to−face mode appeared to feel a lower level of anxiety with the

presence of the interviewer． These results suggest again that when interpreting the

attitudes of examinees toward testing speaking， examinees characteristics， such as

degree ofanxiety， need to be taken into account．

     Regarding the computer mode， it is worth noting that several examinees

mentioned being able to control preparation and response time as a factor fbr the

affective appeal of the computer mode． This is consistent with a proposition made in

Schumann s theory（1997）， that is， providing examinees with more control over an

assessment has a positive influence on their affect．

5．1．4 GeneraR discussion of results

     Taking the findings of the five research questions together， two issues need

fUrther discussion． They are concerned with the disparity in fmdings concerning the test

scores and speech samples， as well as the different fUnctions that the computer mode

and the face−to−face mode may have regarding practice effect．

     In this study， partly in order to complement each other in analysis， examinees

performances血terms ofboth test score and speech sample were compared between the

computer mode and the face−to−face mode． However， the findings ofdiscourse analysis

of speech samples were only partially matched by the judge−mediated data． That is，

while there were significant differences in qualitative measures o f fluency acro ss modes，

they did not result in different test scores obtained from ratings awarded by raters． The

discrepancy of the results may be due to the different nature of the two methods of

assessing perf（）rmance． Generally， test scores are considered to be crude indications of

proficiency， since they are based on impression alone， aided by a rating scale with rather

sketchy characterizations of perfbrmance（Elder＆Iwashita，2005）． In contrast，

discourse analytic measures have been considered to be relatively more o句ective hldicators ofactual perfbrmance（Ellis＆Barkhuizen，2005）．

     Another viable explanation fbr the disparity ofthe fmdings may lie m the scormg

rubric on which raters were asked to evaluate performance o f the examinees． This study

used the original scoring rubric fbr the speaking part of the GTEC fbr STUDENTS．

However， a closer look at l the scoring rubric reveals that it fbcuses on fbatures of

linguistic performance different from the measures used for analysis of speech samples．

That is， the scoring rubric inc luded the rating elements o f fluency， grammar， vocabulary，

and pronunciation， while speech samples were examined in temls of fluency， accuracy，

and complexity． Thus， there is a need to discuss the disparity of the findings on these

elements separately．

     With respect to nuency， an examination ofthe content description in the scoring

rubric shows that it is only concemed with repetition． However， although difference in

fluency in terms of repetition was fbund in examinees speech samples across modes，

the raters seem to have failed to detect this． As mentioned earlier， it is possible that in

order to obtain a better score on fluency from raters， a larger decrease in the frequency

of repetition use may be essential． As fbr the use of filled pauses， which was the other

difference fbund in speech samples， it was not included in the content description of

fluency．

     Further， an examination o f the content description ofother rating elements reveals

that tho se regarding grammar and vo cabulary appear to reflect the aspect o f accuracy

fbr discourse measures． Thus， from the perspective of accuracy， the results of

comparing test scores converge with those of speech samples， as neither shows any

difference across modes． With regard to complexity， since it is not included in the

scoring rubric， there is no way to compare the results of speech sample analysis with

those of ratings． An implication of this discussion is that， in order to make an effective

comparison， fUture research needs to employ a scoring rubric with descriptions that

more clo sely reflect measures used f（）r discourse analysis．

     Another interesting fmding o f this study relates to the practice effect o f the

computer mode on test performance produced with the face−to−face mode． In order to

minimize a possible test order effect， participants in this study were assigned to two

groups， each group taking the two tests in a different order． However， an interaction

effect was fbund between test mode and test order． That is， the group who took the

computer−delivered test first perf（）rmed better on the later face−to−face test， whereas the

group who took the face−to−face test first did not achieve a better performance on the

computer−delivered test．

     The question then arises as to why the computer−delivered test seems to have a

practice effect on the face−to−face test but not vice−versa． One possible interPretation

lies in the different roles the two test modes may play regarding practice effect． It is

possible that the examinees who took the computer mode first may not have achieved

the best performance due to the加nfamiliarity with the test format． When given the

opportunity to take a second test， they may have attempted to give a better performance．

During the face−to−face test， besides having become familiar with the test content， the

reactions丘om the interviewer may have motivated them to give better verbal responses

to test tasks， accompanied with such non−verbal responses as facial expressions， eye

contact， and gestures． Tentative support f〜）r this speculation is provided by questionnaire

feedback from examinees． Some participants indicated that they did a better job in the

face−to−face test， not only because they were familiar with the task content， but also

because they felt encouraged to do their best by the presence ofthe interviewer．

     On the other hand， the face−to−face test may not fUnction as practice when

administered prior to the computer test． Similar to tho se who took the computer test first，

the participants who took the face−to−face test frrst may also have hoped to achieve a

higher score on the second test． However， during the computer−delivered test， although

they did become familiar with the test content， they may not have fe lt motivated to

strive for better performance， as there was no reaction丘om the computer． Accordingly，

the first administration of the face−to−face test tumed out not to be bene丘cial fbr

perf（）rmance in the co碑）uter test．

     The above discussion of the practice effect of the different modes in delivering

speaking tests has two implications． One is that the practice fUnction of

computer−delivered speaking tests may be used as a test preparation strategy fbr

face−to−face tests． In other words， practicing with computer−delivered speaking tests

may be effective in achieving better performance in face−to−face tests． Methodologically，

rather than adopting a counterbalanced design， fUture research may administer the

face−to−face mode first， fb llowed by the computer mode． As such， the practice effect

that has led to the decrease of data used in the statistical analyses in this study may be

avo ided．

5．2 1mplications

     Based on the fmdings ofthe present study， several implications can be offered for

language assessment， test development， and the SLA field．

     This study provides positive evidence fbr the validity of the computer−delivered

speaking test in terms o f test scores and fairne ss o f the test． The fact that there were no

observed differences in test scores implies that test users could use the test result丘om

the computer−delivered test to infer their scores in a face−to−−face test． Also， the finding

that proficiency level was not a factor affecting what examinees produced across test

modes suggests that the test is not biased toward examinees of any particular

proficiency level．

     However， examinees speech samples were found to differ in terms of fluency．

This suggests that delivery mode is a confbunding factor， which should be taken into

account in modeling perfbrmance in computer−delivered speaking assessment． It also

presents a threat to the validity o f the cornputer−delivered speaking test and to the

interpretation of test scores． This research thus calls fbr particular attention from

language testers when converting a face−to−face test to a computer mode． Although

technology enables things to be done more ef6ciently， and practicality is a crucial

aspect of test design， other factors， particularly validity， should also be considered．

When implementing a computer−delivered speaking test， ideally the option of a

face−to−face test should also be o ffered， and any known effects of computer delivery

mode should be reported in order to help examinees choose between test modes and

interpret test results． When there are practical constraints fbr administration of a

face−to−face test， an issue that comes into play is the degree of importance we attach to

the differences． In other words， whether or not it matters to have a choice may depend

on whether differences， fbr instance regarding fluency as found in this study， are

considered important for the purpo ses ofthe test and test users．

     The fndings on the comparability of speech samples are relevant to the SLA field

as well． Instruments used fbr eliciting speech samples from language learners play a

critical role in the type of data used to make inferences about the L21earning process．

Given the differences in linguistic performance across delivery modes found in this

study， SLA researchers need to reconsider the notion of elicitation instnlments as a

monolith and should exercise caution in generalizing their findings when using

computer−delivered monologic tasks fbr data collection．

     This research also provides usefUl infbrmation on the development of a scoring

rubric． In this study， differences in speech samples across delivery modes were found

f（）r fluency measures related to repetition and filled pauses． This implies that these two

measures are important indexes that should be taken into consideration when evaluating

examinees perf（）rmance． However， in the scoring rubric fbr the speaking part of the

GTEC for STUDENTS， only repetition is included in the content description o f fluency．

Accordingly， adding a description about the evaluation of filled pauses would lead to

the impro vement o f the scoring rubric．

     Furthermore， test development could also benefit from examinees feedback to

the questionnaires in this study． lt was found that some examinees appear to favor the

fUnctions ofcontrol over testing procedures． Thus， test developers may endeavor to add

more contro l fUnctions to add merit to the computer−delivered speaking test． In addition，

many examinees mentioned lack of response as the major drawback of the computer

mode． It would be worthwhile to explore the potential of computer tec㎞010gy fbr

providing various f（）rms o f feedback that would be effective in making the test situation

mo re authent ic．

5．3 Limitations of the study

     The findings of the present study must be interpreted in the context of several

potential limitations．

     At least three factors that might have affected the generalization o f the results are

worth discussion here． First， the participants in this study were Japanese EFL learners

from two high schools and three universities． The generalization of the results to a

population ofother nationalities must be made with caution． Moreover， being volunteers，

the participants−the high school students in particular−were noted by their teachers as

having higher oral proficiency and higher motivation f（）r English study than their

classmates． Thus， they may not be very representative of other students in their own

category．

     The second factor is related to tasks compared in this study． Due to the limitation

of techno lo gy， interactive tasks that require examinees to interact with another

interlocutor have not been realized on the computer． This leads to a methodological

concem regarding confbunding variables of tasks when comparing the

computer−delivered speaking test and face−to−face test directly． Thus， this study fbcused

on mono logic tasks， which only elicit performance representing a narrow interpretation

of speaking proficiency． Given that the relationship of monologic ability to interactive

ability is still unclear（lwashita et al．，2008）， caution is advised in generalizing the

results to interactive tasks．

     Also， two types of monologic task， namely， a narrative task and an opinion task，

were investigated in this study． It would be ideal to collect speech samples fro m a wide

variety of speaking tasks to better understand the effects of computer delivery mode．

However， it was not feasible to ask examinees to respond to all types of monologic

tasks， particularly in a test setting． The interpretation of the results is thus limited to

these two particular types ofmono logic task．

     The third factor concerns the gender and the nationality o f the interviewer in the

face−to−face test． A female served as the interviewer in all face−to−face tests． However， it

should be noted that some studies have suggested that the gender ofthe interviewer may

be a factor affecting examinees performance．0 Sullivan and Porter（1996）， fbr

instance， found that Japanese examinees achieved significantly higher scores when the

interviewer was a woman． Buckingham（1997）revealed that women seemed to feel

more comfortable talking to a female interviewer than to a male． Given that two thirds

of the participants in the present study were Japanese females， employing a male

interviewer may have led to a different outcome．

     It is also irnportant to recogniZe that the interviewer is a non−native English

speaker， whose native language is Chinese． In this study， the examinees were not

informed that the interviewer was not a Japanese native speaker． However， it is still

likely that they were aware of that simply through greeting the interviewer in Japanese

before the test． Although some examinees commented that the interviewer was quite

easy to talk to， others may have felt more nervous than if they had been talking to a

native Japanese． Thus， the nationality o f the interviewer is another factor that needs to

be taken into acco皿t when generalizing the findings ofthe present stud》た

     Methodologically， there is a concem over the equivalence of the proficiency of

participants assigned to different testing orders． This study was orighlally intended to

use a within−subject design to compare test scores， but in fact between−su句ect analyses

were conducted where practice effects were observed． In such cases， the equivalence of

the two groups compared is critical to support the intemal validity of the丘ndings．

Although this study randomized the order of test taking， there is no objective index fbr

the equivalence of the proficiency of the participants who took the tests in different

orders． Consequently， there may still have been differences between these participants

in terms ofproficiency．

5．4 Further research

     The tentative nature of the fmdings of the present study highlights the need f〜）r

fUrther research in several areas．

     Clearly， fUrther work is needed to extend this study to a broader range of

examinee groups engaging in a wider range of speaking tasks． In particular， there is a

need to investigate interactive tasks that can elicit interactive ability， which is an

important part of speaking proficiency． In this regard， the challenge f（）r the fUture is to

develop cornputer−delivered speaking tasks that can be used to test interactive ability．

Given the great potential of computer technology， the delivery of interactive tasks by

computer may be realized in the near fUture． This may mean that as the Intemet

continues to spread， interaction between the examinee and an interviewer could take

place synchronously over the lnternet through teleconferencing， as happens with Skype

nowadays． Also， recently some researchers have been attempting to apply speech

recognition techno logy to testing speaking（e．g．， Zec㎞er， Bejar，＆Hemat，2007）． With

advancement in this area， it may even be possible to replicate interactive speaking in a

realistic situation on the computer， with the computer offering proper responses to

examinees speech， just as an interviewer would．

     Furthermore， fUture studies may approach the investigation of the effect of

coMI）uter delivery mode by fbcusing on task proMI）ts． Nowadays， the multi−media

capability of the computer has made various types of feedback possible， fbr example，

audio or video fbrms． It would be interesting to血ld out how examinees perfbrmance

may differ according to these types of feedback． Furthermore， some examinees have

commented on the facilitative role of the fUnction o f controlling pla皿ing and response

time． Research into the effect of the presence vs． absence of these fUnctions on

examinees performance will also be o f interest．

     Investigations into two issues regarding Planning time are essential to advance

our understanding regarding the effect of computer delivery mode， i． e．， the length and

the actual use ofpla皿ing time． In this study， examinees were given the same amount of

plan血ig time for the tasks acro ss modes． They were also instmcted to respond to tasks

whenever they were prepared within the p lanning time． Desp ite this， in the face−to−face

test， the author noticed that most examinees did not start talking until the author

indicated that the planning time had ended． On the other hand， in the co叫）uter test，

since examinees could respond to tasks by clicking the Record button on the

computer screen， they may have used a shorter period ofplaming time than they did in

the face−to−face test． Literature has provided evidence about the effect ofthe length of

pla皿ing time on speech production in second language acquisition and testing（e．9．，

Wigglesworth，1995；Foster＆Skehan，1996；Mehnert，1998）． Thus， fUture research

may need to address its potential impact on examinees perfbrmance in the two types of

speaking tests．

     Apart from the length of planning time， how examinees actually use the planning

time across modes is another issue to be fUrther investigated． The author observed that

most of the examinees did not rehearse their speech aloud in the face−to−face test．

However， in the computer test where the interviewer was absent， they may have

combined preparing the tasks with rehearsing fbr the tasks． Thus， fUture research may

address the different appro aches for planning and their potential effects on performance

across modes by recording examinees behavior with a video camera and conducting a

retrospective interview or verbal protocol analysis．

     Moreover， fUture research on examinees characteristics could fUrther our

understanding o f the constructs that the computer−delivered speaking test assesses．

Although this study did not find examinees proficiency to be a factor affecting speech

samples across modes， its potential effects on test scores need fUrther exploration by

using a more general pro fic iency index for grouping examinees．

     In addition， two other types of characteristics of examinees may be ofparticular

relevance to the effects of computer delivery mode on testing speaking． First， it was

found that feeling nervous has been one of the primary reasons given by both the

examinee group favoring the face−to−face mode and that preferring the computer mode

for their attitudes toward the two test modes． Thus， fUture studies need to explore the

possible interaction of such affective factors as communication anxiety and computer

anxiety with test perfbmance across the two test modes． Second， the use of cornputer

f（）rassessment may cause construct−irrelevant variance due to computer familiarity on

the part ofexaminees． Some examinees indicated that they were not used to taking tests

on the cornputer， smce they did not use computers 廿equently in the丘 daily lives，

especially outside schoo1． It may be argued that the computer−delivered speaking test

involves only rudimentary computer skills， such as clickmg and dragging with a mouse．

Nevertheless， considering the low expo sure to computers fbr high school and university

students， fヒeshmen and sophomores in particular， computer familiarity is a factor worth

fUrther hlvestigation．

     From a methodological perspective， the analyses of underlying constructs

measured by the two modes in this study were limited to an exploratory factor analysis

approach． By collecting data丘om larger samples， a fUrther study may use more

sophisticated statistical procedures， such as multi−group confirmatory factor analysis

（MGCFA）． With a larger sample， this approach would allow a more rigorous

co叫）arison of psychometric properties of the tests across modes and provide more

mfbrm［ation on the potential differences of factor loadings and underlying structure of

the constnlcts assessed by the two modes．

5．5 Concluding remarks

     In a speaking test delivered by computer， instead of interacting with another

interlocutor， examinees are required to talk to the conlputer． There have been concerns

about the iMI）act of this feature on examinees perf（）rmances and their attitudes toward

the test． The present study addressed these concerns with empirical evidence that

examinees performance was not affected in terms oftest scores but did differ on some

aspects o f their linguistic performance when compared with the face−to−face test． Also，

examinees appear to prefer talking to a person rather than to a computer．

     Despite the fact that the delivery o f speaking tests with computers is increasingly

popular， particularly in large−scale tests， the potential effects ofcomputer delivery mode

have seldom been investigated． It is hoped that this study will provide helpfUl

information to develop and validate a cornputer−delivered speaking test． It is also hoped

that other studies will fbllow to extend the findings of this study and shed additional

light on the complex nature ofspeaking perfbrmance in computer−delivered assessment．