Results and Discussion

A s F i g u re 1 . a n d Ta b l e s 1 a n d 2 re v e a l t h a t t h e r e we r e m o re p o s i ti v e re s po n s e s a t t h e A 1 a n d A2 b a n ds . F e we r po s i t i v e re s po n s e s we r e f o u n d a t t h e B1 a n d B 2 b a n d s . On e n o t a b l e a n d s e e m i n gl y co n t r a d i c to r y e x c e p ti o n i s t h e s po k e n p r o du c t i o n s ki l l c a te g o r y i n t h e A1 . 2 a n d A 2 . 2 l e v e l s . H e re , m o re p o s i ti v e r e s po n s e s f o r t h e m o r e a d v a n ce d l e ve l ( A2 . 2 ) w e re g i ve n t h a n f o r t h e m o r e b a s i c l e ve l ( A 1 . 2 ) . T h e r e m a y b e t w o re a s o n s f o r t h i s . O n e , e a c h q u e s t i o n o n t h e q u e s t i o n n a i re a s k s p a r ti c i pa n t s to re s po n d i n a b i n a r y f a s h i o n ( e i t h e r “ ye s ” o r “ n o ” ) . S t u de n t s m a y h a v e m o r e di f f u s e o r s u b t l e r e s p o n s e s t o t h e q u e s t i o n s a s ke d a n d t h u s a L i k e r t s c a l e t y p e f o r m a t m a y h a ve b e e n m o r e a p p r o p r i a t e . I n a dd i t i o n , t h e t wo q u e s t i o n s a t t h e A 1 . 2 l e v e l ( Q u e s ti o n 25. I can express simple opinions related to limited, familiar topics, using simple words and

basic phrases in a restricted range of sentence structures, provided I can prepare my speech in advance. Question 29. I can give simple descriptions e.g. of everyday object, using simple words and basic phrases in a restricted range of sentence structures, provided I can prepare my speech in advance.) could be perceived as more challenging as the A2.1 questions (Question 26. I can introduce myself including my hobbies and abilities, using a series of simple phrases and sentences. Question 30. I can give a brief talk about familiar topics (e.g. my school and my neighborhood) supported by visual aids such as photos, pictures, and maps, using a series of simple phrases and sentences.). A possible rationale for the perceived level of difficulty could be that participants concentrated on the phrasing

“restricted range of sentence structures” in the A1.2 questions. What is meant by this expression is open to interpretation and each participant may be unclear as to how to gauge the meaning of

“restricted range”. Thus, these modifying statements in the questions at the A1.2 level may be perceived by participants as being more difficult.

Another apparent anomaly can be seen in the second spoken production question (questions 29 through 32. See Appendix 1 and 2). Here, the largest number of students (161) responded positively to the B1.1 level statement (154 participants stated “yes” for the A1 level statement and 149 stated “yes” to the A2 level statement).

Here, two possible explanations can be provided for this seeming contradiction. Firstly, in the A1 and A2 category questions, modifying phrases related to grammatical structure (“restricted range of sentence structures” in question 29 [A1] and “series of simple phrases and sentences” in question 30 [A2]) which do not exist in the B1 question may cause Japanese students who are particularly sensitive to grammatical points when speaking to not respond in the positive. A second reason for the apparent contradiction is the A1 and A2 level questions could be perceived as connoting that the respondent will give a public speech; the two questions explicitly state: “…provided I can prepare my speech in advance” (question 29) and “I can give a brief talk…”

(question 30). The B1 category question does not explicitly or implicitly connote a public speech The

B1 category question does not explicitly or implicitly connote a public speech as the language is vague (“I can talk about….”).

Ta bl e 1 : R a w N u m b e r o f P a r t i c i pa n t s ’ P o s i t i v e R e s p o n s e s to C E F R - J Q u e s t i o n s A c c o r di n g to S k i l l G r o u p

* N o te : T h e n u m b e r o f s t u d e n t s a t a g i ve n C E F R - J l e v e l o r a b o v e a n d t h o s e w h o r e s po n d e d “ y e s ” to C E F R- J c a n - d o q u e s t i o n s e x ce e d s t h e t o ta l n u m b e r o f p a r t i c i p a n ts . T h e n u m b e r a t s t u de n t s a t gi v e n C E F R - J l e ve l o r a b o ve i s m o r e t h a n th e n u m b e r o f pa r t i c i pa n t s b e c a u s e s t u de n t s a t t h e A 2 o r h i g h e r l e v e l s h o u l d , i n p r i n ci p l e , re s po n d “ y e s ” t o q u e s ti o n s

C E F R - J L e ve l L i s t e n i n g

A 1 . 2 4 7 0

A 2 . 1 4 0 5

B 1 . 1 2 3 8

B 2 . 1 1 3 0

C E F R - J L e ve l R e a d i n g

A 1 . 2 4 4 5

A 2 . 1 4 1 9

B 1 . 1 2 3 0

B 2 . 1 8 3

C E F R - J L e ve l

S p o k e n I n t e r a c ti o n

A 1 . 2 4 3 2

A 2 . 1 3 7 9

B 1 . 1 2 3 9

B 2 . 1 1 2 5

C E F R - J L e ve l

S p o k e n P r o d u c t i o n

A 1 . 2 3 2 3

A 2 . 1 3 7 0

B 1 . 1 2 5 6

B 2 . 1 6 2

C E F R - J L e ve l Wr i t i n g

A 1 . 2 4 4 9

A 2 . 1 3 5 8

B 1 . 1 1 8 1

B 2 . 1 8 0

90 a t t h e A 1 . 2 l e ve l ; t h i s w o u l d a l s o b e t r u e f o r t h e A 2 a n d B 1 l e v e l s f o r B1 a n d B2 s tu d e n ts r e s pe c t i v e l y.

F i g u r e 1 : R e l a ti o n s h i p a m o n g ra w n u m b e r o f s e l f - r a t i n g s i n f i v e s k i l l ca t e go r i e s

Williams and Andrade, in a study of 243 Japanese university students studying English in Japan, found that “ a n x i e t y w a s o f te n a s s o c i a t e d w i th t a s k s i n v o l v i n g s pe a k i n g i n f ro n t o f o t h e rs ” [ 2 5 ]. T h e re f o r e , a n xi e t y r e l a te d t o g i vi n g a m o re f o r m a l p u b l i c s pe e c h r a t h e r t h a n s i m pl y ta l k i n g a b o u t a to p i c c o u l d h a ve s k e we d r e s po n s e s .

The measure of internal consistency of participants’ self-ratings among the five skill groups in this study was statistically reliable when the raw responses of individual students was compared with each skill category as the Cronbach alpha was 0.9062. This scale is from 0 to 1.0 and the higher the number, the more statistical reliability. A reliability coefficient of 0.7000 or higher is deemed “acceptable” in the majority of social science studies. The coefficient in this present study mirrors the finding of internal consistency that Tokeshi and Gao [26] had. In their study that the C r o n b a c h ’s a l p h a v a l u e a m o n g t h e f i v e s k i l l c a t e go r i e s w a s 0 . 8 7 2 w h e n t h e y c a l c u l a te d “ t h e a ve r a ge s e l f- ra t i n g s o f

i n d i vi d u a l re s po n de n t s f o r e a c h s ki l l c a te g o r y ” [ 2 7 ] . A n o th e r s t u d y w i th s i m i l a r re l i a b i l i t y d a ta i s Runnels’ [28]. When calculating data on the reliability of the entire CEFR-J A level can-do statements with 590 first- and second-year Japanese university students in Runnels’ study, across all statements a strong reliability as measured by Cronbach’s alpha 0.944 scale was found.

Ta b l e 2 : Ra w N um b e r o f P a r ti c i p a n t s ’ P o s i t i v e Re s po n s e s t o Ea c h Q u e s ti o n A l o n g C E F RJ S k i l l

G r o u p s ( q u e s ti o n n u m b e r n o t e d i n p a re n t h e s e s )

Ta b l e 3 . C ro n b a c h Al p h a f o r C E F R - J L e ve l s A 1 . 2 A 2 . 1 B 1 . 1 B 2 . 1

L i s t e n i n g

2 5 0 ( 1 )

2 2 7 ( 2 )

1 7 3

( 3 ) 7 4 ( 4 ) 2 2 0

( 5 )

1 7 8

( 6 ) 6 5 ( 7 ) 5 6 ( 8 )

R e a d i n g

2 2 7 ( 9 )

2 2 2 ( 1 0 )

7 7 ( 11 )

3 0 ( 1 2 ) 2 1 8

( 1 3 )

1 9 7 ( 1 4 )

1 5 3 ( 1 5 )

5 3 ( 1 6 )

S p o k e n I n t e r a c ti o n

2 1 6 ( 1 7 )

1 8 8 ( 1 8 )

1 4 2 ( 1 9 )

6 3 ( 2 0 ) 2 1 6

( 2 1 )

1 9 1 ( 2 2 )

9 7 ( 2 3 )

6 2 ( 2 4 )

S p o k e n P r o d u c t i o n

1 6 9 ( 2 5 )

2 2 1 ( 2 6 )

9 5 ( 2 7 )

2 4 ( 2 8 ) 1 5 4

( 2 9 )

1 4 9 ( 3 0 )

1 6 1 ( 3 1 )

3 8 ( 3 2 )

Wr i t i n g

2 2 8 ( 3 3 )

1 7 0 ( 3 4 )

9 8 ( 3 5 )

4 6 ( 3 6 ) 2 2 1

( 3 7 )

1 8 8 ( 3 8 )

8 3 ( 3 9 )

3 4 ( 4 0 )

C E F R - J

L e v e l C r o n b a c h a l ph a

A 1 . 2 0 . 7 4 4

A 2 . 1 0 . 6 8 4

B 1 . 1 0 . 7 9 9

B 2 . 1 0 . 8 5 3

A s s h o w n i n Ta b l e 3 . , t h e A 2 . 1 l e ve l q u e s t i o n s h a d a f e w a n o m a l i e s a n d t h e y m a y e xp l a i n t h e r e l a t i ve l y l o we r C r o n b a ch a l p h a c o e f f i c i e n t i n co m pa r i s o n w i t h t h e o t h e r l e v e l s . A M o k k e n S ca l e a n a l y s i s f o r t h e A 1 . 2 , A2 . 1 , B 1 . 1 , a n d B2 . 1 s t a t e m e n t s a s g ro u p w a s c o n d u c t e d ( r e s u l ts d i s p l a y e d i n Ta b l e 4 . b e l o w ) . R u n n e l s ’ [ 2 9 ] r e s e a r c h u n de r t o o k s i m i l a r a n a l y s i s a n d s h e p o i n te d o u t t h a t “ M o k k e n s ca l i n g i s a s ta t i s ti c a l te c h n i q u e t h a t a s s u m e s t h e o r de r o f di f f i c u l t y o f i te m s i s n o t t h e s a m e a c ro s s a p o pu l a ti o n ( v a n S c h u u r, 2 0 0 3 ) a n d i t p ro v i de s a m e a s u re o f r e l i a b i l i t y b y i de n ti f y i n g i te m s f o r w h i c h G u t tm a n p a t te r n i n g i s o c cu r r i n g a t h i g h e r r a te s ( M o l e n a a r, 1 9 9 7 ; S i j ts m a & M o l e n a a r, 2 0 0 2 ) ” [ 3 0 ] . G u t t m a n s ca l i n g i s u t i l i ze d i n s u r v e y s , t e s t s o r q u e s t i o n n a i re s h a vi n g b i n a r y i te m s ( “ y e s ” o r

“ n o ” a s i n th e o n e e m pl o y e d i n t h e p r e s e n t s tu d y ) . I n a G u t t m a n s ca l e , i te m s a r e a r r a n g e d i n o r de r s o th a t i f a p a r ti c i p a n t a n s we r s “ y e s ” to a n B 2 . 1 l e ve l q u e s t i o n , t h e y s h o u l d a l s o re s po n d “ y e s ” to l o w e r p ro f i c i e n c y i te m s ( e . g . a t th e A 1 . 2 , A2 . 1 , a n d B 1 . 1 l e v e l s ) . T h u s , s t a t i n g “ y e s ” t o a n i t e m a t a h i g h e r l e ve l i m p l i e s th a t “ ye s ” s h o u l d b e n o te d a t a l l l o w e r l e ve l q u e s t i o n s . R u n n e l s [ 3 1 ] w r i te s t h a t M o k k e n s c a l i n g ge n e r a te s a r a t i o t h a t d i s p l a y s t h e pe r c e i ve d d i f f i c u l t y o f e a c h ca n - d o s ta t e m e n t a c c o r di n g to t h e s e l f - r a te d a b i l i t y o f e a c h p a r ti c i pa n t a n d t h e d e g re e to w h i c h a l a r g e r n u m b e r o f h i gh e r p r o f i c i e n c y p a r ti c i pa n t s p e r ce i ve d th e c a n - d o s t a te m e n t m o r e c h a l l e n g i n g . T h e s t a ti s t i c t h a t i s p ro d u c e d i s ca l l e d t h e c o e f f i c i e n t o f h o m o g e n e i t y ( H o r H - v a l u e ) , a n d i t re v e a l s t h e s t r u c tu r e o f a n s w e rs f o r e a c h s ta t e m e n t b y m e a n s o f i t e m l i m i t a ti o n s . T h u s , t h i s c o e f f i c i e n t n o te s t h e r e l i a b i l i t y s c a l e f o r e a c h c a n - d o s ta t e m e n t a n d e x po s e s t h e de g r e e t h a t a G u t t m a n m o d e l ca n b e s e e n f o r e a c h a n s w e r.

Co e f f i c i e n ts o f h o m o ge n e i t y r a n ge f ro m 0 t o 1 . 0 . A h i g h e r H - s co r e c o r r e l a t e s w i t h a n e l e m e n t t h a t m e a s u r e s m o r e i n s y n c w i t h G u t tm a n ’s p ro p o s i ti o n ( i . e . i t e m s a re a r r a n g e d i n o r de r s o t h a t i f a re s po n de n t a n s w e r s “ y e s ” to a n B 2 . 1 l e ve l q u e sti o n , t h e y s h o u l d a l s o r e s po n d “ ye s ” to l o we r p ro f i ci e n c y i t e m s ; s t a ti n g “ ye s ” t o a n

i te m a t a h i g h e r l e v e l i m pl i e s t h a t “ ye s ” s h o u l d b e n o te d a t a l l l o w e r l e v e l q u e s t i o n s . ) . H - v a l u e s b e n e a t h t h e 0 . 3 t h r e s h o l d a r e u n a c c e p ta b l e , a n d s c o re s o v e r 0 . 6 de n o te s s t ro n g re l i a b i l i t y [ 3 2 ] .

Table 4. Mokken Scales Coefficients for Each CEFR-J level Examined

C E F R - J L e ve l H - c o e f f i ci e n t

A 1 . 2 0 . 3 9 4

A 2 . 1 0 . 3 8 7

B 1 . 1 0 . 3 7 6

B 2 . 1 0 . 4 8 5

Two hypotheses for relatively lower h -coefficient results (all are on the lower cusp of acceptable results of 0.3 as aforementioned). Firstly, in contrast to Runnels’

study [33] where participants responded to the five A levels in the CEFR-J (A1.1, A1.2, A1.3, A2.1, A2.2), the present study took a larger view and examined a range from of levels from A1.2 to B2.1. As previously noted, conspicuous incongruities in participants’ answers to two A2.1 can-do statements relative to A1.2 and B1.2 queries may have skewed reliability. Secondly, the present study only had respondents select from binary options (“yes” or

“no”). Having a more restricted field of options, while producing more discrete data, constrains and compels participants in an either-or situation which may not truly reflect participants’ ability. In a similar study, Runnels’

[34] utilized a five category Likert-scale response form (strongly disagree to strong agree). Providing more breadth of responses possibly reduces the error co-efficient and diffuses the data.

A number of weaknesses with this study must be detailed. Firstly, the entire 110 CEFR-J can-do list was not employed and thus while an overview of the participants’ self-ratings can be recognized, a complete picture of the group’s CEFR-J self-assessment cannot be seen. This would be a step for a further study. The rationale for this limited survey was twofold; first, at the start of an academic year appropriating time for freshmen students to participate in a voluntary-based questionnaire is limited and therefore the researcher decided to use an abbreviated version, and second, a t t h e o u ts e t o f th e a ca d e m i c y e a r, t h e s t u de n ts i n q u e s t i o n s a t f o r a n i n - h o u s e

92 pl a c e m e n t te s t th a t wa s d e ve l o pe d a l o n g C E F R l i n e s . T h e p u r po s e o f t h i s t e s t w a s to s t r e a m s tu d e n t s i n to s e c ti o n s s o t h a t s t u de n ts w o u l d b e wi t h s t u de n t s a t s i m i l a r p ro f i ci e n c y l e v e l s . T h i s i n - h o u s e p l a c e m e n t te s t w a s de s i g n e d b a s e d o n th e Ca m b r i d ge K E T e x a m i n a ti o n ( ge a r e d f o r A1 a n d A 2 l e a r n e rs ) . F o l l o w i n g n u m e r o u s p i l o t t ri a l s a n d f o l l o w - u p a n a l y s i s a n d m o di f i c a t i o n s , t h e i n - h o u s e te s t r e ve a l e d m e a s u re a b l e r e l i a b i l i t y t o t h e K E T te s t ( 0 . 7 5 3 C r o n b a c h a l p h a ) . T h u s , a dm i n i s te r i n g a q u e s t i o n n a i re w i th q u e r i e s f r o m t h e A1 , A2 , B 1 , a n d B 2 l e ve l s w a s d e e m e d s u i ta b l e . A s e co n d d r a w b a c k o f t h i s s tu d y w a s t h e b i n a r y r e s po n s e o p t i o n s ( y e s / n o ) . Wi t h m o r e n u a n c e d o p ti o n s i n a L ik e r t s ca l e ( e . g . “ s t ro n g l y a g r e e , / s o m e w h a t a g re e / s o m e w h a t di s a g re e / s t r o n g l y di s ag re e ” ) m a y h a v e p r o vi de d m o r e te l l i n g a n d i n f o r m a ti v e da t a . T h i r dl y, ce r t a i n t e rm s i n t h e C E F R - J ca n - do l i s t m a y b e u n f a m i l i a r t o s tu d e n t s a n d t h u s ca u s e c o n f u s i o n a n d t h e r e f o re r e s u l t i n m i s l e a d i n g d a t a . A s t h e m a j o ri t y o f pa r t i c i pa n t s i n t h i s s t u d y w e r e f i r s t - y e a r J a pa n e s e u n i v e r s i t y s t u de n ts w h o e n t e re d th e u n i v e rs i t y i n A p r i l , q u e s t i o n s pe r t a i n i n g to a wo r k e n v i ro n m e n t w o u l d n o t n e c e s s a r i l y pe r t a i n to r e s po n de n ts . F o r e xa m pl e , q u e s t i o n 16 states “I can understand in detail specifications, instruction manuals, or reports written for my own field of work, provided I can reread difficult sections.” Students in their first year of studies at university in Japan most likely would not be exposed to interacting with English in such a fashion and therefore may be at a loss as to how to respond. A fourth weakness of the study revolves around the term “native speakers” which arises a number of times throughout the questionnaire. The entire notion of

“native speaker” and its relevance to English proficiency is a different topic (e.g. English as lingua franca); however, germane to this study is that numerous participants may not have had opportunities to interact with “native speakers”.

Thus, a question such as number 24, “I can discuss abstract topics, provided they are within my terms of knowledge, my interests, and my experience, although I sometimes cannot contribute to discussions between native speakers” may have

confounded certain respondents.