Re: [dogme] testing
- Also for a more general discussion about assessment this paper published
in Educational Researcher Online is very interesting.:
"The Role of Assessment
in a Learning Culture by
This is also quite long but informative and thought provoking. It
discusses the history of standardised testing, its links to
social/economic efficiency curriculum models and behaviourism and the
rationale for alternative assessment methods which are more in line with
sociocultural/social constructivist curriculum and current
pshycholinguistic ideas about learning and SLA.
Haven't read it yet, but perhaps it's something you can use. I got it from
the Japanese ETJ list courtesy of one John Stark. Happy New Year, hugs,
- --- In firstname.lastname@example.org, "zosia grudzinska" <zosia_g@w...>
> I am writingwould like to
> an article to one of Polish educational periodicals in which I
> lay bare the false assumption that testing gives an accuratepicture of any
> student's level of language competence. I know my feelings andconclusions
> based on observations - but I also know that any such article willbenefit
> if the writer can roll of a list of names, references, sourcesetc.
You might like to look over writings by Lyle Bachman, now at UCLA
(try a google search for links). He's got a pretty even keel and has
devoted much of his career to the complexities of language testing,
and has been "in on it" since the whole communicative revolution
began back in the 70's.
As to the accuracy of any test as a measure of language competence,
while it's true that nothing definitive has been developed, that's
not to say that considerable progress isn't being made regarding
what certain tests reveal and what they don't, which I think is more
the point. Tests can now be made in ways that increase their
validity (the degree to which a test measures the mental/linguistic
constructs it is meant to measure, and no others) and their
reliability (the "imperviousness" of that measure to outside
factors) in ways that are mathematically demonstrable.
Alas, however, what no one can do is to give any guarantee that the
people who have to make decisions based on a person's test scores
(teachers, admissions boards, human resources departments,
scholarship committees, etc) will actually have any real
understanding how to interpret those scores judiciously.
On a lighter level, you know, I find it extremely curious and
frustrating to realize that, within about 15 minutes of talking to
any non-native speaker, I have a darn good idea of his/her
communicative level (especially compared with the thousands of
others I have talked to and taught in my life). What surprises me
and frightens me the most is that I am rarely very far off the mark--
I hardly ever discover that my initial assessment was very wrong.
And I can do that with no real formal test, almost with my eyes
I can only conclude that H. Widdowson, as Scott cited earlier, was
just being a bit grouchy that day because he's never been very good
at language testing and thinks nobody else can be, either.
John in Madrid
(Nice to be back on line again and reading everyone's posts--sorry
for the long absence!)
How good are you at assessing a learner's written English? I always thought I was pretty
good and that two pages of A4 was more of a sample than I needed to roughly assess a
writer's standard on (the German method) a scale of 1 (excellent) to 6 (miserable).
And then I had a go at qualifying as an IELTS examiner and missed (their) mark nearly
all the time!
(I had no difficulty assessing spoken English).
- John wrote:
> On a lighter level, you know, I find it extremely curious and frustratingto realize that, within about 15 minutes of talking to
> any non-native speaker, I have a darn good idea of his/her communicativelevel (especially compared with the thousands of
> others I have talked to and taught in my life). What surprises me andfrightens me the most is that I am rarely very far off the > mark-- I
hardly ever discover that my initial assessment was very wrong. And I can do
that with no real formal test, almost > with my eyes closed.
Two things caught my attention here. Firstly, 'any non-native speaker', I
wonder how you know they are 'non-native'? I have met hundreds of
non-natives (sic) whose language ability is better than most native
speakers. Secondly, what 'markers' are you using for this assessment? I
wonder if these 'markers' are grammatical?
I know what you mean. The linguistic competencies at play in
producing written English really do seem to require using a
different way to measure them. Just try breaking down that "1" in
the "1 (excellent) to 6 (miserable) scale" you mention into
descriptors. Then try doing the same for a 1-to-6 scale in spoken
English, and you'll see we're dealing with a different kettle of
fish. As for the IELTS, and of course for any other test, you may
find that their descriptors are at odds with your own set of
criteria, and that, even when you both agree on some specific
construct (say, for example, "smooth and logical transitions"),
your "tolerance" level and criteria for what constitutes acceptable
and what does not--especially when the test subject's competence in
that construct is only partial--varies considerably.
I did some work with the IELTS when I was in Bangkok two years ago
and was a bit disappointed in its sampling and scoring method. They
do try to raise inter-rater reliability by insisting that all IELTS
scorers assess in the same way, which is good. But to take it to an
extreme, I too could attain high inter-rater reliability if I were
to form a group of examiners by telling them to give high scores to
papers using a wide margin, and low scores to people who leave no
margin at all. The test would then be extremely reliable, but of
course not at all valid, at least not if anyone were silly enough to
try using those scores as a judge of language competence. My own
conclusion from my IELTS experience there was that the IELTS is in
fact a business first and foremost, and that those students who are
lucky enough to study an IELTS prep course with a teacher who
actually does IELTS scoring will get the best scores. An outstanding
business plan, no doubt, but any real advances in language testing
in such a context can only be haphazard at best.
All the best,
John in Madrid
PS: Yes, Scott and all, I'm still skirting the issue regarding
appropriate testing consonant with DOGME. It'll come, it'll come!!
--- In email@example.com, djn@d... wrote:
> How good are you at assessing a learner's written English? I
always thought I was pretty
> good and that two pages of A4 was more of a sample than I needed
to roughly assess a
> writer's standard on (the German method) a scale of 1 (excellent)
to 6 (miserable).
> And then I had a go at qualifying as an IELTS examiner and missed
(their) mark nearly
> all the time!
> (I had no difficulty assessing spoken English).
- --- In firstname.lastname@example.org, "Adrian Tennant" <adrian.tennant@n...>
> John wrote:frustrating
> > On a lighter level, you know, I find it extremely curious and
> to realize that, within about 15 minutes of talking tocommunicative
> > any non-native speaker, I have a darn good idea of his/her
> level (especially compared with the thousands ofme and
> > others I have talked to and taught in my life). What surprises
> frightens me the most is that I am rarely very far off the > mark--I
> hardly ever discover that my initial assessment was very wrong.And I can do
> that with no real formal test, almost > with my eyes closed.speaker', I
> Two things caught my attention here. Firstly, 'any non-native
> wonder how you know they are 'non-native'? I have met hundreds ofassessment? I
> non-natives (sic) whose language ability is better than most native
> speakers. Secondly, what 'markers' are you using for this
> wonder if these 'markers' are grammatical?Dr Evil,
> Dr Evil
> Two things caught my attention here. Firstly, 'any non-nativespeaker', I wonder how you know they are 'non-native'? <
First point: as an (albeit slovenly) rule of thumb based only on
neurolinguistic and psycholinguistic research findings, I tend to
classify anyone (myself included) whose acquisition of Language X
has principally taken place after puberty as a "non-native" speaker
of that language. It has nothing to do with how well or how poorly
that speaker's use of the language approximates a norm or standard,
nor the particular variety or varieties of Language X that person
happens to speak.
You also wrote:
>I have met hundreds of non-natives (sic) whose language ability isbetter than most native
I enjoy thinking about what you mean by someone's language ability
being better than someone else's. That's the crux of it, and any and
all decriptions you and everyone else in the group can provide will
be of enormous use to us as we untangle the DOGME testing question.
It's a scary one to answer, since I think we're all aware that it
reveals a lot about our perceptions, opinions, and misconceptions.
For my part, I'll say in answer to your second point (You wrote:
>Secondly, what 'markers' are you using for this assessment? Iwonder if these 'markers' are grammatical? <) that grammatical
competency indeed is one of the many pieces of the puzzle. It's also
the piece that has been put most under the microscope, discussed the
most in all corners of the Earth, and most successfully been
encapsulated in test design and procedure. Too bad it's just one
piece. What the other pieces are is what we need to speak about here.
So, my markers (in a nutshell) include sociolinguistic appropriacy
to a great extent, listening ability, and, to a lesser extent,
phonological competence (pronunciation). My feeling is that what we
like so much about DOGME are all the things that fall under the
realm of sociolinguistic competence. It's what has too long and too
often been shirked off in more "traditional" language teaching
approaches. It's the most elusive and least understood of the
competencies, and yet, many sincere testing experts are concurring
that it may be the biggest single contributor to general language
proficiency. So what is it, folks? What is DOGME enabling our
students to do well at? Go ahead and dive in with your views,
whatever they may be, as long as they are sincere, so we can start
to see what pieces of the puzzle we're looking at.
John in Madrid