[an error occurred while processing this directive]

ESTABLISHING PROFICIENCY-BASED LANGUAGE REQUIREMENTS


Barbara F. Freed


IT HAS been said that only two things interest all students—sports and sex—and only two things interest all faculty—salaries and parking. At times, however, there appears to be a third thing, one that interests both groups: the language requirement. 1

It is probably safe to assume that most language educators are familiar with the recommendations of the MLA-ACLS Language Task Forces and with those of the President's Commission on Foreign Language and International Studies. Among the most frequently cited of these recommendations are those that encourage language study in general and those that emphasize the need to set clearly defined, realistic proficiency goals for each level of study and to develop tests to measure such proficiency.

The response to these recommendations has been characteristically slow. While none of us can point to a ground swell of new foreign language requirements, however, it appears that some institutions are considering reinstating such requirements 2 and that the decline in language enrollments is leveling off.

It is imperative for those who are considering the reinstatement of language requirements and for those who withstood the pressure to abandon them in the 1960s, to avoid perpetuating ineffective traditional evaluation standards. The problem with past requirements, if I may quote Richard Brod's persuasive letter of 16 May 1980 to Representative Paul Simon, is that

they are usually stated in units of time: years, semester hours, quarters. Neat, definable, computable—but empty. Time requirements say nothing about learning, only about sitting. It is an appropriate way to measure and define sentences in criminal justice [many students perceive these requirements as penal sentences] but not, we think, to measure learning.

What complicates this situation is that really “time” requirements are veiled by other terms. While most colleges and universities that require language study evaluate it in time units, catalog descriptions frequently begin by invoking such admirable goals as “competence,” “performance,” or even “proficiency.” The problem is that competence or proficiency tends to be measured by semester hours at worst and by scores on standardized multiple-choice discrete-point tests at best. We have all come to recognize the utter meaninglessness of course grades and units of study, at least for basic language learning. One school's A is another's C; four semesters at one institution may be equal to two at another; an MLA Cooperative or a CEEB test score tells us nothing about what a student is able to do with a language.

Lest you think that I am about to spend the next twenty minutes adding to the rhetoric on “America's scandalous incompetence in foreign languages,” 3 let me assure you that the remainder of this presentation will focus on the specific steps we have taken at the University of Pennsylvania to change our foreign language requirement. I hope that my description of the making of these changes—the why, the how, and the what—will serve as an incentive to other institutions faced with similar problems.

The Faculty of Arts and Sciences at the University of Pennsylvania has maintained a foreign language requirement in one form or another since the mid-1800s. Although the number of hours and even the languages required have changed over the years, the basic catalog description has not changed at all in the last twelve years. The 1980–81 undergraduate catalog describes the foreign language requirement as follows:

Every student is required to attain a certain competence in a foreign language. Such competence may be demonstrated either by passing a foreign language course numbered 4 (or equivalents)…or by the student's score in the Advanced Placement Test or the College Entrance Examination Board (CEEB) Achievement Test or on a departmental placement examination.
Students who are placed in an intermediate or advanced level language course upon entrance to lite University may not receive credit for a lower level course in the same language. Students are admitted to first, second or third year courses in language according to the amount of work they have had in high school and their score in the CEEB test.
Foreign language courses taken to fulfill the foreign language requirement as well as foreign language electives not being used in a major may be taken on a pass/fail basis. ( University fo Pennsylvania Course of Study, 1980–81 , p. 14)

At Penn, as elsewhere, the language requirement has not been, to say the least, one of the most popular aspects of the undergraduate curriculum. Moreover, it has been popularly believed, though not established as a fact, that basic language instruction was not what it could or should be. Before 1979, there had been no intensive study of student achievement at the 1–4 level, (those courses that constitute the current language requirement). In September 1979, the foreign language departments began exploring the feasibility of redefining our longstanding “time” requirement as a proficiency requirement. We, like others, had become increasingly aware of the futility of an evaluation based almost exclusively on time units. It had become clear that many students who fulfilled such requirements were unable to use the languages they had studied. While they had satisfied the demands of the university, they had experienced no personal satisfaction, nor had they gained a skill. This perception notwithstanding, we did not make the transition to a proficiency requirement easily or quickly. Where student opinion manifestly opposes any foreign language requirement, a change that apparently makes the requirement yet more stringent would provoke intense student reaction. Objection to a proficiency requirement was not limited to students. The notion aroused skepticism on both practical and philosophical grounds among certain faculty members. We therefore moved cautiously, in several distinct stages.

During the first stage we conducted two studies of student achievement, one on the effect of “pass/fail” grading of language achievement, the other on student achievement as measured by the College Entrance Examination Board Language Achievement Test.

The findings of the “pass/fail” study are relatively unimportant here, since “pass/fail” grading for language requirements is not a common option. In summary, our findings confirmed what we had suspected all along. Roughly a third of all students enrolled in courses numbered 1 to 4 take courses on a pass/fail basis. The actual grades for students enrolled on that basis are considerably lower than for students who elect to take these courses for a grade. The real grade value of 65% of the P grades would have been C or below. The grade value of P grades diminishes with each succeeding semester and the number of students registered on a pass/fail basis increases throughout tile 1–4 sequence. In other words, the pass/fail option does not help our students acquire second language skills.

Of greater relevance are the results of our CEEB study. This study was undertaken for several reasons: to measure student achievement on a nationally standardized test; to determine correlations between CEEB scores and grades and between the CEEB scores of currently registered students and the cutoff scores used to place incoming students in language classes. Another purpose of the study was to provide a basis for changes in interpreting and enforcing our current language requirement.

We administered the CEEB Language Achievement Tests to all students studying French, Spanish, Italian, German, Russian, and Latin at Levels 1–4 at the end of the 1980 spring and fall semesters. At the end of four semesters of study the mean scores of students in French were 558; in Spanish, 533; in Italian, 561; in German, 575; in Russian, 593; and in latin, 580. It should be understood that not all the students tested necessarily studied language at Penn for four consecutive semesters. Many students had been placed (by CEEB scores) in language level 2,3, or 4 on arrival at Penn. The results for students completing one, two, and three semesters of study have also been computed and are reported elsewhere. 4

Now, given these data, what did we learn? For one thing, we learned that there was a near-perfect correlation between CEEB scores and grades. In almost all instances, the CEEB scores for A students were higher than those for B students, those for B students higher than those for C students, and so forth. It is equally true that the CEEB scores increased with the level of study. It also became clear that our own students were not achieving scores consistent with the scores we use to place or exempt incoming students. Despite this somewhat disappointing finding, we further learned that our students' scores were roughly consistent with the national norms of the Educational Testing Service (ETS) for students who have completed four years of high school study, 5 said to be the equivalent of two years of college study.

Beyond this, we had learned little. We knew before beginning our study that students were dissatisfied with what they were learning. They have continued to tell us that they want to learn to speak and to understand the language, that relevance dictates teaching the spoken language, and that any functional utility for them depends on oral proficiency. Our two studies provided no information on overall language proficiency. This deficiency led to the second stage of our investigation.

To evaluate the oral skills of our students, we administered the Foreign Service Institute (FSI) Oral Proficiency Interview to a subset of the students enrolled in French 1–4 who took the CEEB Test. We gave this oral interview to a sample of students who were completing French 3 during the 1980 spring semester. Roughly 10% of the 226 students in French 3 were tested. These students were selected at random from those who volunteered to be interviewed. 6 The FSI tests were administered by a member of the Foreign Service Institute staff or by two of us who had been trained by FSI during the spring 1980 Testing Kit Workshop. We were assisted by French faculty who were native speakers and by teaching assistants trained in FSI procedures.

The results of this study were again disappointing but not surprising. The mean FSI rating for this group of 24 students was slightly more than level 1+. The Department of State publication, Testing Kit: French and Spanish , gives a functional description of the skills required for an S-1 rating and a comment on the S-1+ rating.

An S-1 is able to satisfy routine travel needs and minimum courtesy requirements. Can ask and answer questions on very familiar topics; within the scope of very limited language experience can understand simple questions and statements, allowing for slowed speech, repetition or paraphrase: speaking vocabulary inadequate to express anything but the most elementary needs: errors in pronunciation and grammar are frequent, but can be understood by a native speaker used to dealing with foreigners attempting to speak the language: while topics which are “very familiar” and elementary needs vary considerably from individual to individual, any person at the S-1 level should be able to order a simple meal ask for shelter or lodging, ask and give simple directions, make purchases, and tell time. (Marianne L. Adams and James R. Frith, eds. [Washington, D.C.]. pp. 13–14)

While no specific description is given of the abilities of a candidate rated S-1+, it is understood that the “S-1+ exceeds S-1 primarily in vocabulary and is thus able to meet more complex travel and courtesy requirements … Fluency may vary, but even quite voluble speech cannot compensate for all the other serious weaknesses” (p. 25).

Beyond the FSI published descriptions, we have also learned that the S-1+ will characteristically use linguistic forms not within the repertory of an S-1. These are most specifically the futur proche and, with limitations, the passé composé .

From this description it is rather obvious that the command of the spoken language demonstrated by an S-1+ is limited. Moreover, this 1+ is a mean score that represents a range from 0+ to 2+. These findings are supported by those of two major studies of oral language proficiency. The first is John Carroll's 1967 study of foreign language majors near graduation from college. Carroll reported a median FSI score of 2+. The second study, ETS's recently completed “Global Understanding” study, reported a mean score of 2.09, or something over an S-2, for students at beginning, intermediate, and advanced levels. 7 In both studies, the group tested was larger than the Penn group.

In addition to establishing the mean oral proficiency scores for students at the end of three semesters of college study, we were able to establish a statistically significant relation between the CEEB scores and the FSI scores. The 24 students who were given the FSI Oral Proficiency Interview had a mean CEEB score of 545. This score was somewhat higher but roughly equivalent to the mean score for all students completing French 3 during the academic year 1980. The mean FSI score of S-1+ correlated at the .48 level {significant at the .01 level) with the CEEB scores. 8 This correlation is important for those who are accustomed to evaluating student achievement by standardized test scores. For many language educators, the meaningfulness of proficiency test scores has yet to be established. A certain CEEB or MLA Cooperative test score has a well-recognized, if somewhat dubious, value to educators. By contrast, “S-2+” is still a relatively vague term. To persuade the skeptics who question the validity of oral language instruction, therefore, it was important to demonstrate that students' scores on the standardized achievement test correlated with scores on a global test of oral proficiency. Since we found a correlation, we can slate that, for this population of students, the student who speaks well (according to his or her FSI score) will, on the average, also read well. 9

With this information, we could now move to the third stage of our project. We had established the level of student achievement as measured by CEEB scores and of oral proficiency as measured by FSI scores. We already knew that students were eager to develop functional skills but that many faculty members were hesitant to support a language requirement that seemed to be so skill-oriented that it excluded broader humanistic goals. We thus asked each individual department to consider which standards of performance they deemed acceptable and realistic for students completing a language requirement. All departments emphasized standards and not tests. We have steadfastly maintained that learning must be directed toward goals and not toward tests. While evaluation will ultimately be by tests designed to measure proficiency according to the goals we establish, the performance goals themselves will determine the method and content of instruction.

As a result of the steps we have taken, several language departments (or sections of departments) are prepared to proceed with a new version of our language requirement (these include the German Department, Arabic in Oriental Studies, Russian in Slavic Studies, and Italian and French in Romance Languages.) I here present the standards developed by one of these groups, the French section of the Romance languages department, and describe our procedures in dealing with grades, time, and credit.

The chart below outlines five areas of proficiency agreed on by the French faculty. We believe that these standards are comprehensive and reasonable and that they will prepare students for use of the language and for literary or cultural study.

The standards, as listed in the first column of the chart, are oral interaction, listening comprehension, reading comprehension, writing, and culture. Temporarily, the CEEB Language Achievement Tests will be added to this set of five proficiency standards. The numbers (5, 10, 15, 20) across the top of the chart represent the numerical scores for performance at each of the described levels.

Column three of the chart represents the level needed to satisfy the proficiency requirement. For example, students will have to demonstrate oral language skills as described by an FSI 1+ and listening comprehension skills that permit partial comprehension of factual news broadcast and/or partial comprehension of native speaker conversation. To demonstrate reading comprehension, they will have to answer questions that require factual and inferential responses to literary texts and to provide a summary with the most salient features of a nonliterary text correctly represented. To test writing proficiency, students will be given guided compositions in which they will need adequate vocabulary and some use of complex sentences beyond subject-verb-object organization. The standards to measure competence in the culture are still to be determined. In addition, students will have to achieve a score of 500 on the CEEB Language Achievement Test in French.

It should be emphasized that proficiency will be a composite score on all six portions of the test. Therefore, a student will be able to compensate for weakness in one area by excellence in another. To pass the proficiency tests, students will have to achieve a total score of 10. Each portion of the total test is assigned a score, as the preceding chart shows. (The appendix describes in detail how scores will be calculated and how cumulative performance will be determined.)

Policies for enforcing the proficiency requirement are:

  1. Students must merely pass the test to be eligible for a grade in French 4. There will be no letter grade on the test, only P (proficiency/pass) or F (fail).
  2. Students who have passed the test will be given a course grade based on their cumulative performance throughout the semester. This grade will be based on quizzes, homework, hourlies, etc. The content of many of these will be similar to areas tested in the proficiency test.
  3. We have eliminated any relation between the scoring system and A, B, C, D grades by using the categories 5, 10, 15, 20. We have established 10 as the minimal passing level.

For the academic year 1981–82, students who fail the proficiency test will be dealt with individually and will be required to improve their performances on the portion(s) of the test they have not passed or raise their scores on passed portions to equal a total of 10 on all sections. Students who receive A's in French 3 will continue to take the CEEB and be required to achieve a score of 650 on that test if they would like to be considered exempt from French 4. A small number of specially selected A students in French 3 will be able to take the new proficiency test so that we can compare their results to those of students in French 4.

You will recall that according to Penn's present language requirement, students must “attain a certain competence in a foreign language…demonstrated by passing a foreign language course numbered 4.” The requirement we will begin to enforce in September 1981 will be consistent with the earlier catalog description. The difference is that in the past, to successfully complete a course numbered 4, a student had merely to achieve a passing grade. In the future, students will have to demonstrate competence. The new procedure requires students to pass the proficiency test Io be eligible for a grade in French 4. There will be no letter grade for the test, other than P (proficiency/pass) or F (fail). Students who have passed the test will receive a course grade based on quizzes, homework, hourlies, and so forth, many of which will relate in content to the established standards and will cover the areas included in the proficiency test.

Changes in policy will be made at the end of the spring 1982 semester based on our evaluation of the entire procedure. Our ultimate intent is to use our own proficiency test instead of CEEB scores to determine placement in language courses. This test will relate to our minimum standards for language competency.

We expect problems in implementing these policies during the first year, and we anticipate modifications and refinements in our testing procedures. Nevertheless we are confident that these changes will significantly raise the level of instruction, achievement, and motivation. We believe further that establishing minimum competency levels at appropriate levels of difficulty will prevent the lowering of standards. By maintaining a system that encourages the highly motivated student and raises the acceptable competency level for the weaker or less motivated student, we are convinced, we will be providing all students with opportunities for meaningful accomplishment during their years of language study. We are equally confident that the institution of well-defined and realistic proficiency goals will do much to meet the repeated demands of students without sacrificing any of rite skills or values we believe in. I might stress here that functional proficiency standards, defined as “real-life tasks,” are not limited to ordering a meal or giving directions. Discussing a novel or a work of art is a “real-life task.”

Now that our students are pressing for greater emphasis on oral skills, we can hope and indeed should expect that our colleagues in departments of language and literature will continue to emphasize the humanistic values of literary appreciation and cultural study. If we can reliably show that instruction based on functionally defined proficiency standards increases student interest and motivation, we should earn our colleagues' needed support in promoting such standards. if we are to increase our credibility and respond to today's demands fur accountability, we must, to quote Paul Gaeng's eloquent address to the ADFL seminar held at William Paterson College, “establish a bridge between the life of the mind and the world of work and show the contributions that young people trained in humanistic disciplines can make to society in both the private and public sectors (“Educational Accountability and the Humanities,” ADFL Bulletin , 12, No. 1 [Sept. 1980], 25).

Many of us at the University of Pennsylvania are enthusiastic about the potential of our redefined language requirement. We hope and anticipate that the direction we are taking will result in increased student interest and achievement in all areas of language learning. If future research proves this to be so, we will have come a long way toward promoting foreign language learning in this country.


This article is based on a paper presented at the ADFL Tenth Anniversary Seminar, held at Middlebury College, 8–11 June 1981. The author Assistant Dean for Language Instruction at the University of Pennsylvania.


NOTES

1 I wish to express my gratitude to many colleagues who contributed in various ways to our proficiency project at the University of Pennsylvania. These include members of the Advisory Committee on Language Instruction and colleagues in French who participated in the FSI Testing Project. Appreciation is also extended to the teaching assistants in all the departments whose burden it was to hand-score the CEEB Language Achievement Tests. Special thanks go to Clifton Cherpack, chairman of the Department of Romance Languages, for his constructive comments on an earlier draft of this paper. Finally, I am grateful to Robert Dyson, dean of the faculty of arts and sciences, for his commitment to funding and supporting all efforts to improve basic language instruction.

2 Richard I. Brod. ed., Language Study for the 1980s: Reports of the MLA-ACLS Language Task Forces (New York: MLA, 1981), p. 19.

3 President's Commission on Foreign Language and International Studies, Strength through Wisdom: A Critique of U.S. Capability (Washington, D.C.: GPO. 1979), p. 6.

4 Barbara Freed, “Achievement in Elementary and Intermediate Language Classes as Measured by the College Entrance Examination Board Language Achievement Tests,” MS (1981), Univ. of Pennsylvania. Philadelphia.

5 Chart, “Means and Standard Deviations for 1978–79, Achievement Test Scores in Foreign Languages for Two, Three, and Four Years of Study,” Admissions Testing Program, Program Direction, Educational Testing Service.

6 There are acknowledged risks in using a “volunteer” population. However, the range of scores awarded to the volunteers (0+-2+ as described on p. 8) clearly shows that not only the better students volunteered for the oral interview.

7 John B. Carroll, “Foreign language Proficiency Levels Attained by language Majors near Graduation from College,” Foreign Language Annals , 1 (1967), 131–51; John L. D. Clark, “Survey Measures: Languages,” in College Students' Knowledge and Beliefs: A Survey of Global Understanding , ed. Thomas S. Barrows (New Rochelle, N.Y.: Change Magazine Press, 1981), p. 32.

8 Barbara Freed, “Applications of FSI Oral Proficiency Testing at the University Level,” paper presented at the preconference workshop on oral proficiency testing, Georgetown University Roundtable on Language and Linguistics (Washington, D.C.: Georgetown Univ. Press, forthcoming).

9 A note of caution is called for in using this correlation for predictive purposes. While a significant correlation between the two tests was obtained, it is important that future research seek better ways to measure the relationship between language skills.


APPENDIX

Proposed Scale for French 4 Proficiency
Examination: Points: 5.00 10.00 15.00 20.00
LEVEL
CEEB Language Achievement Test ^
|
v
450
^
|
v
500
^
|
v
550
^
|
v
600
FSI Oral Proficiency Interview S-1 S-1+ S-2 S-2+
Listening Comprehension 5 10 15 20
Reading Comprehension 5 10 15 20
Writing 5 10 15 20
Culture 5 10 15 20

Explanation


  1. All students will be required to take all six sections of the test as listed above. (See chart, pp. 9–10, for a description of the proficiency standard at each level lot each skill).
  2. Performance below 10 on any portion of the exam constitutes failure for that section.
  3. Cumulative performance , however, is the ultimate criterion for passing. One may accrue from 0 to 20 points on each section, but the cumulative average must be 10 or higher (i.e., a student may fail sections of the exam and still pass the proficiency test; see examples a and b, below). Those who fall below this average must retake the section that brought the average down, or they may retake the entire exam.
  4. 4. Sample passing scores:
    a) CEEB 500 (=10.00)
         FSI 1 (= 5.00)
         Listening Comprehension 70% (=10.00)
         Reading Comprehension 85% (=15.00)
         Writing 55% (= 5.00)
         Culture 85% (=15.00)
         Total 60.00 ÷ 6 = 10.00
    b) CEEB 475 (= 7.50)
         FSI 1+ (=10.00)
         Listening Comprehension 75% (=11.70)
         Reading Comprehension 85% (=15.00)
         Writing 80% (=13.30)
         Culture 85% (=15.00)
         Total 72.50 ÷ 6=12.10


Functional Proficiency Standards for French
Score 5 10 15 20
Skill and Method of Testing
Oral Interaction S-1 S-1+ S-2 S-2+
[FSI interview] (Adams and Frith, 1979, pp. 13–15) - able to satisfy routine travel needs and minimum courtesy requirements. Can ask and answer questions on very familiar topics; can understand very simple questions and statements, allowing for slowed speech, repetition, or paraphrase, vocabulary inadequate to express anything but the most elementary needs, errors in pronunciation and grammar are frequent, but can be understood; should be able to order a simple meal, ask for shelter, ask and give simple directions, make purchases and tell time
- linguistic elements usually limited to the present tense
- avoir, aller, être; might have pidginized forms, and/or phrases without verbs
- exceeds S-1 primarily in vocabulary, can meet more complex travel and courtesy requirements; grammar weak and usually can't cope with social conversation; fluency may vary
- linguistically has a notion of the past tense but confuses uses and may occasionally fail to use; can use futur proche
- able to satisfy most routine social demands and limited work requirements. Can handle with confidence most social situations including introductions and casual conversations about current events, work, family, autobiographical information, can get the gist of most conversations on non-technical subjects, speaking vocabulary sufficient to respond simply with circumlocutions; does not have thorough or confident control of grammar but distinguishes the future tense even if he doesn't fully control it, uses the passé composé and l'imparfait , although with error, may use negative, “depuis” with present, present with infinitive - exceeds 2 in grammar or vocabulary; better comprehension
- has control of the future; still makes mistakes with the passé composé and l'imparfait
- usually no command of the conditional or subjunctive; must use passé composé correctly
Listening Comprehension
[review in writing radio broadcasts just heard - topic identification and salient features] a few key words of factual news broadcasts; tour guide;
- partial comprehension of slow, careful, simplified telephone speech
partial comprehension of factual news broadcast; partial comprehension of native speaker conversation; partial comprehension of normal phone conversations; partial comprehension of lectures or formal presentations on a subject with which he/she is familiar - reasonably complete comprehension of phone speech, can detect affective components of speech
- very good comprehension of factual news broadcasts, partial comprehension of news commentary and analysis; partial comprehension of movie sound tracks; can catch some words of popular songs, can get the gist of native speaker conversation
- complete comprehension of phone speech, can detect affective components of speech
- very good comprehension of factual news broadcasts, partial comprehension of news commentary and analysis; partial comprehension of movie sound tracks; can catch some words of popular songs; can get the gist of conversation
Reading Comprehension
(Questions and summary of literary and nonliterary texts) - ability to interpret:
a) literary and b) nonliterary texts:
a) questions requiring factual responses
b) summary of nonliterary text in which the topic is correctly identified but which contains gross misstatement of information presented
- ability to interpret:
a) literary and b) nonliterary texts:
a) questions requiring factual and inferential responses
b) summary of nonliterary text in native language
- ability to interpret:
a) literary and b) nonliterary texts:
a) questions requiring factual, interpretive and inferential responses
b) summary of nonnative text in native language
- ability to interpret:
a) literary and b) nonliterary texts:
a) questions requiring factual, interpretive, inferential and stylistic responses
b) summary of nonliterary text in native language
Writing
(Guided compositions) deficient vocabulary
- can write only the most simple sentences; frequent spelling errors, many grammatical errors, lack of idiomatic usage
- adequate vocabulary for writing about familiar topics, some use of complex sentences beyond SVO organization, morphological problems remain but basic control of past, present, and future evident, some idiomatic usage evident - varied basic vocabulary, varied but somewhat limited syntactic patterns, fewer morphological problems, good use of idioms - extensive vocabulary, well developed use of various syntactic patterns, natural and well formed transitions, frequent use of idioms
Culture
(To be determined)
CEEB Language Achievement Test 450 500 550 600


© 1981 by the Association of Departments of Foreign Languages. All Rights Reserved.

[an error occurred while processing this directive]