51
第1回「言語教育評価フォーラム」 講演1
The TCF
(
Test de Connaissance du Français
)
Making, Analyzing, and Implementing
Bruno MÈGRE
Program Manager, CIEP (Centre International d’Études Pédagogiques)
1. Presentation of the CIEP (Centre International d’Études Pédagogiques)
The Centre International d’Études Pédagogiques is an official institution sponsored by the French Ministry of Education and the Ministry of Higher Education. It is located in Sèvres, in the Paris suburb. It was establish in 1945 and became a govern-ment institution in 1987. It has the responsibility of promoting the French language, training teachers of French as a foreign language and the training making. It also acts as coordinator in education expertise for international organisations as well as for French and foreign ministries. Regarding assessment, it is responsible for the development and organisation of examinations established by the French Ministry of Education to assess the teaching of French as a foreign language throughout the world.
2. Overview of the Department of Assessment and Certification
In order to meet its specific mission, there is a Department of Assessment and Certification. This department is divided into four sub-departments. Two depart-ments are in charge of French language diplomas. One supervises the DILF,Diplôme Initial de Langue Française. The DILF used to evaluate the initial achievements in French. This diploma is below the level A1. Another is responsible for the DELF, Diplôme d’Études de Langue Française and the DALF, Diplôme Approfondi de Langue Française, which constitute 96% of certifications taken in French around the world. A third department is responsible for the TCF, Test de Connaissance du Français, and the fourth, the expertise department, has as mission to cooperate on the development of certifications for French and other languages at the request of foreign ministries or institutions.
With the exception of the DILF, which is applicable only inside France, it is pos-sible for candidates wishing to be assessed reliably and accurately to take the certifi-cations throughout the world. The number of candidates forecast for 2008 shows that these certifications are an undeniable motivation for a large number of learners, both French and foreigns. Because of this, the latter constitute a veritable market. By the end of 2008, a total of 331,000 persons should be assessed by one of the official certi-fications developed by the CIEP.
The CIEP is a member of ALTE (Association of Language Testers in Europe). This association founded by Cambridge University brings together European insti-tutions developing and establishing certifications at international level. The CIEP is a member of ALTE for French certifications along other institutions such as the Goethe Institute for German, the Cervantes Institute for Spanish, the University of Lisbon for Portuguese, the University of Perouse for Italian, and many others. Additionally, the CIEP participates in the work of the Council of Europe by assessing appraisals and the testing assessment tools.
All certifications developed by the CIEP are all along the line with the six levels of language skills defined by the Council of Europe in the CEFR (Common European Framework of Reference). The CEFR was published in English in 2000 and trans-lated into French in 2001. Besides its humanist role in contesting discrimination, this tool allows the learner’s linguistic capabilities to be defined throughout his learning period, regardless of the foreign language concerned.
The same six-level grid has been developed for global skills, for all languages. Other six-level ability grids are available for each of the following skills : spoken and written understanding, spoken and written production, as well as spoken and written interaction. There are also grids for each linguistic function: writing a message, tak-ing part in a conversation, understandtak-ing a spoken message, etc. These grids are used for teaching as well as for learning or assessment.
The levels are divided into three categories: • A for elementary level.
• B for intermediate level. • C for advanced level.
Each of the categories is sub-divided into • A1: discovery level;
• A2: survival level;
• B1: threshold level (adequate for travelling and getting by in the country where the target language is spoken);
• B2: independent level (minimum required to follow university studies in a country where the target language is spoken);
• C1: independent level; and
• C2: proficient level (translation level or level that can be reached by any for-eign language student).
• Level A1: capable of simple interaction when the subject is familiar. • Level A2: capable of exchanging information meeting fundamental needs. • Level B1: capable of maintaining discussions, expressing opinions, managing
one’s affairs alone, and establishing relationships with others.
• Level B2: capable of arguing and conversing without effort and is aware of his/her errors.
• Level C1: capable of interacting with subtlety and accuracy in specialist fields, both orally and in writing.
• Level C2: capable of interacting in a structured way in complex fields and dif-ferent socio-cultural environments.
• A1 - Beginners level: Has a basic knowledge of the language, familiar everyday expressions and simple phrases.
• A2 - Pre-Intermediate level: Is familiar with frequently used expressions and conversations on routine matters.
• B1 - Intermediate level: Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes and ambitions, and briefly give reasons and ex-planations for opinions and plans.
• B2 - Independent User (upper intermediate): Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that makes regular interaction with native speak-ers quite possible without strain for either party. Can produce clear, de-tailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and disadvantages of various options.
• C1 - Advanced level: Can understand a wide range of demanding, longer texts, and recognise implicit meanings. Can express himself/herself fluently and spontaneously without much obvious searching for expressions. Can use language flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive devices.
• C2 - Proficient User: Can understand with ease virtually everything heard or read. Can summarise information from different spoken and writ-ten sources, reconstructing arguments and accounts in a coherent pre-sentation. Can express himself/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations.
3. Overview of Certifications
Fashioned in 1985, the DELF and DALF were the first assessment tools created by CIEP. They are managed by the French embassies in each country. In 2005, the two diplomas were reformulated in order to be calibrated to the CEFR levels. Today, they are six mutually independent diplomas, each of which corresponds to the CEFR com-petence levels. The diplomas were designed to be incorporated into a French learn-ing curriculum. The candidate sits for one of the six diplomas, accordlearn-ing to his ability when taking the tests. The tests are presented to candidates in the form of tasks to be completed. Candidates are marked in the examination centre by a trained examiner, who is approved by the CIEP. Approval of scorers and examiners takes three years. Marking is supervised by a jury of several experts, chaired by the Cultural adviser.
The DELF is in two formats, one for adults and the other specifically designed for adolescents, accordingly to their needs and interests. The DALF tests are intended for over 16 years old young people and adults in general. Both exams entail a concept of pass or fail. If the candidate obtains a sufficient number of points, he is awarded the diploma. Otherwise, as being below the standard, he fails.
In addition to these diplomas, the CIEP has devised another assessment tool: the TCF, Test de Connaissance du Français (a French competency test). The TCF was created in response to a French ministry of education directive. After official consul-tations, the ministry concluded the increasing demand for a “light” assessment tool, i.e., a test assessing French skills of non French-speaking students wishing to follow undergraduate studies in French universities.
The French foreign ministry has carried out a survey in 1996-1997 among French overseas diplomatic missions. The goal was to collect data on potential candidates (in the planning phase). The analysis survey revealed than a wider population than the initial one (i.e., students) was potentially interested in the TCF. There was a com-pletely different group in needed, for professional or personal reasons, of a reliable and rapid means of validating their French skills.
Contrary to the DELF and the DALF, the making philosophy of this tool does not entail the idea of pass or fail. A TCF session can be applied to candidates of different levels. It is a linear test which is intended for all groups, regardless of the session or test version. During a TCF session, the level of difficulty increases as the candidate progress. After the session, candidates receive a certificate attesting for their skill level. The TCF allows the candidate to obtain his/her linguistic profile. It is a photograph of the competence level in French of a person at a given time.
The TCF is used by numerous institutions for different requirements. In particu-lar, the TCF is required by
• French universities and their parent ministries (Ministries of Education, Higher Education, Culture and Communication, and the Foreign Ministry), for foreign students wishing to follow studies in France;
• the European Commission from European civil servants and new staff to evalu-ate their French level knowledge; and
• the Quebec Ministry of Immigration and Cultural Communities, which requires that candidates for permanent immigration to this Canadian province possess an adequate knowledge of French.
The linguistic skill model chosen is backed by a deep research (specification of threshold level, followed by lower and higher levels). Its influence has been world wide and has been adopted by the professional language-teaching community. The model provides the TCF with a sound basis for defining the field of assessed compe-tence. This field encompasses the four linguistic skills, spoken and written expression and understanding, and includes spoken interaction. The support documents used for spoken and written texts are all representative of a French and French-speaking socio-cultural reality.
The specifications produced by the TCF are based on the linguistic skills model quoted before and on the Framework descriptors of calibrated competences. They have been produced for four linguistic capacities: written and spoken understanding as well as written and spoken expression.
Consequently, the TCF comprises two sets of tests. The first set is compulsory. It lasts for 1 hour 30 minutes. It is made up of 80 items covering the six CEFR levels, from A1 to C2:
• 30 spoken understanding items. • 20 language structure items. • 30 written understanding items.
The second set is constituted by additional non-compulsory tests that candidates may take if they wish to. Often, an institution would require the TCF for different reasons. The non-compulsory tests are composed as follows:
• written expression (six subjects from levels A1 to C2 to be answered in 1 hour 45 minutes).
• spoken expression (six subjects from levels A1 to C2 to be covered face to face with an examiner in 15 minutes).
4. Making of the TCF®
Language evaluation is no longer dominated solely by the grasp of linguistic skills assessed in isolation, excluding any context. What is tested today reflects a view of the way the language can be used in a daily context. Since the publication of the Threshold Level1, assessment addresses the ability of subjects to be independent in
1 For English: van Eck, J. A. (1975). The threshold level in a European unit/credit system for modern
language learning by adults. Strasbourg: Conseil de l’Europe.
their practice of the language. This autonomy must enable them to form relationships with other people for personal, professional or academic reasons. The linguistic skills to be acquired must include the written and spoken production and understanding (describing, informing, telling, arguing, convincing, qualifying etc). Subjects must be assessed for what they are capable of doing in non-test situations, and not only to establish whether they are proficient in the French linguistic code. Linguistics, syntax, wealth and mastery of the vocabulary, phonology, prosody and fluency are assessed in context. The assessment cannot be limited to the grasp of these elements, essential though they are. It is essential to assess the way in which these various elements are grasped in a communicative situations, particularly in the production of linguistic tasks. The TCF verifies that the subject is not only capable of doing his shopping, re-serving a hotel room, or asking for information in French, but also that he is also able to exchange experiences and opinions so as to have an efficient social relationship with people he is in contact with.
The making of TCF tasks or types of items follows the recommendations of the Council of Europe in the Common European Framework of Reference2. It is based
on a model of communicative language competence which recommends an actional approach placing emphasis on what a subject should be capable of mastering in a foreign language in order to meet his daily needs and form relationships with other people.
The TCF has favoured the Canale and Swain3 model of communicative
compe-tence, which divides communicative competence into four components: grammatical, socio-linguistic (socio-cultural and discursive) and strategic. The Chomskian model, in which “linguistic competence” is a question of knowledge and concerned essential-ly with procedures for treatment of syntax not of know-how, was firmessential-ly rejected. That of Hymes, although more open, was not adopted either because it was considered to be too restrictive. Bachman has retained the approach of Canale and Swain and pos-tulates that competence in a language covers linguistic knowledge (or competence) combined with the ability to apply that knowledge in an appropriate manner.
The Common European Framework of Reference also recommends a language competence model inspired by Canale, Swain and Bachman and defines commu-nicative competence (socio-linguistic, linguistic, pragmatic) as a form of general competence that leads to language activities (in interaction, production, reception and mediation) which use tasks, texts (or speeches) and strategies in four main areas (public, professional, educational, and personal) where situations arise which are characterised by places, organisations, actors, objects, events and operations4.
2 Council of Europe. (2001). Common European framework of reference for languages: Learning,
teaching, assessing. Strasbourg, Paris: Editions Didier.
3 Canale, M., & Swain, M. (1980). Theoretical bases for communicative approaches to second
lan-guage teaching and testing. Applied Linguistics. 1(1), 1-47.
It was this latter approach that was adopted. The CEFR, whose data were used as the point of departure, supplied the theoretical approach necessary for the develop-ment of the TCF.
In order to give the TCF a quality approach for its making, management and marketing procedures, it was decided to have recourse to international standards en-abling them to be measured and certified. Accordingly, the operational management process was created so that the standardized TCF procedures would be submitted to internal and external audits several times per year. In 2003 this process obtained ISO 9001: 2000 certification, which was renewed in 2006 for three years.
This operational process can be divided into two major sets: the upper part cor-responding to the making phase and the lower part to the phases of operation and installation in the centres.
The making phase begins with an order for items which is made to the item writ-ers by one of the managwrit-ers six times per year. The item writwrit-ers are specialised for each of the types of competences and form three different teams: a team of item writers for spoken understanding, a team for language structure items and a third for written understanding items. Each team is placed under the responsibility of team leaders. Item writers and team leaders are trained twice a year at the CIEP by the TCF service program managers.
Drafting of items and their validation by the three team leaders is done by the Internet using a technical tool that allows remote communication. Throughout the item drafting phase the team leaders
• guide the item writers on drafting methods; • give advice to the item writers;
• request modifications; • validate item content; and
• submit the validated items to the members of an external commission.
When the items have been validated by the team leaders, and internal item validation commission is convoked at the CIEP in order to validate their teaching content a second time. The internal commission is comprised of four specialists in methods of teaching French as a foreign language and in assessment, and the TCF manager. During the drafting phase, it
• advises the team leaders.
After the drafting phase, this internal assessment commission
• validates the content of items submitted by the three team leaders; • incorporates the items into the first item bank; and
• informs the psychometricians of the final closure of the educational validation phase of the items ordered.
5. The TCF (Test de Connaissance du Français): analyzing
The second stage of the making phase involves the creation of a pre-test version that is identical in every aspect to a final version of the TCF. The organisation and testing methods are also strictly identical. The pre-test comprises 30 spoken under-standing items, 20 language structure items and 30 written underunder-standing items. A sample group of a minimum of 500 candidates sits for the test. These candidates all have the social, linguistic and cultural profiles of TCF candidates. There are as many men as women, and all ages are represented. To make the sample as representative as possible, the pre-test centres are spread over different parts of the world. These centres are situated in France, Mexico, Canada, Japan, Armenia, Russia, Belgium and Switzerland.
Once the pre-test has been sat by the 500 potential candidates, the answer sheets are sent to the CIEP for marking by scanner and psychometric analysis. The psychometric analysis is an integral part of the TCF making quality approach. The psychometricians, who are part of the TCF team, will analyse the results obtained by the pre-test candidates by applying two statistical approaches: the classical test theory and the item response theory (Rasch model).
The psychometric analysis is performed to guarantee test accuracy so that the results are consistent, i.e., reliable and comparable, regardless of which session a can-didate is registered for.
With the classical measurement model, it is possible to establish • that an item is neither too easy nor too difficult
(What is the proportion of correct answers?); • whether or not the item key functions correctly (Was it identified by whom it was supposed to?); • whether or not distractors play their role correctly (Are they chosen by a minimum of candidates?); and • whether or not the item is sufficiently discriminating (Is there a biserial point correlation?).
With the Rasch model, it is possible to establish whether or not • an item behaves as expected
(graph comparison);
• an item provokes incoherent responses
(e.g., an easy item failed by a strong candidate); • an item provokes correct choices by ignorance (discriminatory power of the item); and • an item displays differential functioning
(e.g., the item is easier for women than for men). Items not meeting the psychometric criteria are rejected.
6. The TCF (Test de Connaissance du Français): implementing
After this last making stage the operational process enters its functioning phase. The internal validation commission meets for the last time to
• proceed with a final validation of the items selected by the psychometricians; and
• incorporate the items into the item bank that will be used to design future ver-sions of the TCF.
When the items have been incorporated into the item bank, they will be used to draft versions of the TCF according to the requests from the centres. For each order for TCF session from a centre around the world, the CIEP reproduces the test book-lets and testing material according to the number of candidates. The centre chooses the date to organise its TCF. The files of registered candidates are sent by e-mail to the CIEP two weeks before the date of the session. The material is nominative and cannot be used by a candidate who is not registered. The material is then sent to the centre by private secured courier (Fedex, DHL or diplomatic bag). The centres are required by covenant to respect the testing methods scrupulously, as they are set out in the different manuals provided to the centre’s session manager: centre manager’s manual, candidate’s manual, invigilator’s manual, and spoken expression examiner’s manual. Inspections take place regularly in the centres to verify that the organisation guidelines are properly respected. When the session is over, the centre sends all the materials to the CIEP for marking: answer sheets, written expression papers, and records of spoken expression test interviews. The test booklets are also sent to the CIEP to be destroyed. When the material has been received by the CIEP, the answer sheets, written expression papers and records of spoken expression test interviews are marked. The CIEP has 18 days to send the official attestations to the centres, with effect from reception of the material.
The conditions of administration and marking are standardized by • same testing methods;
• same answer sheets; • same instructions; and • same marking teams (CIEP).
Markers for supplementary examinations (written expression and spoken expression) are recruited according to their professional and academic profiles. They are trained three times per year and are subjected to various assessments. The examiners are under the responsibility of marking supervisors who
• verify that marking criteria are properly applied; • advise markers when in doubt;
• manage double marking;
• modify, if necessary, the scores given by the markers; and • ensure that marking sessions at the CIEP proceed correctly.
Attestations given to candidates are nominative, and have an attestation code and a candidate code, with a validity of two years. They are printed on security paper. Institutions, which require the TCF to select students for a university curriculum in France, to obtain a visa, or for some other purpose, are able to check the validity and authenticity of an attestation using an on-line system available from the CIEP. The results shown on the attestations are
• a global level (CEFR levels) and the scores in spoken understanding, language structure and written understanding examinations;
• a level and score by competence (spoken understanding, language structures); and
• a level and a score out of 20 for spoken and written expression examinations The CEFR descriptors corresponding to the global and detailed levels are shown on the rear of attestations.
Institutions wishing to become a test centre must complete a request for accredi-tation, available on the CIEP Internet site. This accreditation request is then sent to the CIEP for analysis by an accreditation commission. Various criteria are taken into account: the type of institution, number of teachers, number of invigilators, size, ven-tilation and acoustics of the examination rooms, technical equipment, geographical region, experience in organising tests examinations, access facilities for persons with disabilities, etc. Generally speaking, the TCF centres are selected by priority from the following institutions: French overseas institutions, French and foreign universities (Foreign language or French language departments), private schools, and profes-sional training centres.
The number of foreign universities which have adopted the TCF to offer it to their students or incorporate it into their programmes is increasing all the time. The most represented countries are: Canada, USA, Mexico, Japan, Hong Kong, Poland, Taiwan, Germany and Peru.
In Japan, in addition to French institutions established in your country, the TCF has obtained a real success in the following universities: Nanzan University, Kansai University, Dokkyo University, Gakushuin University and, soon, Keio University.
Since its creation in January 2002, the TCF has seen a constant growth in the number of candidates. For 2008, this number is expected to top 46,000.
This strong increase in the number of candidates is due to the TCF being adopted by foreign universities and the possibility for centres to offer the TCF on computers. This is a computerised version, not an on-line test. The TCF sessions are made avail-able to the centre four days before the date of the session by e-mail; the sessions are then installed in the computers dedicated to TCF testing. This approach was adopted to avoid the constraints related to Internet bandwidth during the test, and also to improve session security. The advantages of the computerised TCF are as follows:
• no costs of dispatch of the session material by courier.
• no problems connected with delays in delivery of the material. • provisional results at the end of the session.
Finally, the TCF includes material for assistance with test preparation, both for candidates and anyone involved with organisining a session. Books and CD-ROMs have been issued by various French publishers specialised in the teaching of foreign languages. Free Internet sites also exist, making it easy for candidates to become familiar with item typology. These interactive training pages are available on the Internet sites of Editions Didier, Radio France Internationale and (from September 2008) TV5 Monde (the international French language television station).
