Loading ...
Sorry, an error occurred while loading the content.

Re: [dogme] testing

Expand Messages
  • Renata Suzuki
    Also for a more general discussion about assessment this paper published in Educational Researcher Online is very interesting.: The Role of Assessment in a
    Message 1 of 10 , Jan 1, 2004
    View Source
    • 0 Attachment
      Also for a more general discussion about assessment this paper published
      in Educational Researcher Online is very interesting.:

      "The Role of Assessment
      in a Learning Culture by
      Lorrie Shepard
      Online at
      http://www.aera.net/pubs/er/arts/29-07/shep01.htm

      This is also quite long but informative and thought provoking. It
      discusses the history of standardised testing, its links to
      social/economic efficiency curriculum models and behaviourism and the
      rationale for alternative assessment methods which are more in line with
      sociocultural/social constructivist curriculum and current
      pshycholinguistic ideas about learning and SLA.

      Haven't read it yet, but perhaps it's something you can use. I got it from
      the Japanese ETJ list courtesy of one John Stark. Happy New Year, hugs,
      Renata
    • John Franklin Nelson
      ... would like to ... picture of any ... conclusions ... benefit ... etc. Zosia, You might like to look over writings by Lyle Bachman, now at UCLA (try a
      Message 2 of 10 , Jan 1, 2004
      View Source
      • 0 Attachment
        --- In dogme@yahoogroups.com, "zosia grudzinska" <zosia_g@w...>
        wrote:
        > I am writing
        > an article to one of Polish educational periodicals in which I
        would like to
        > lay bare the false assumption that testing gives an accurate
        picture of any
        > student's level of language competence. I know my feelings and
        conclusions
        > based on observations - but I also know that any such article will
        benefit
        > if the writer can roll of a list of names, references, sources
        etc.

        Zosia,
        You might like to look over writings by Lyle Bachman, now at UCLA
        (try a google search for links). He's got a pretty even keel and has
        devoted much of his career to the complexities of language testing,
        and has been "in on it" since the whole communicative revolution
        began back in the 70's.

        As to the accuracy of any test as a measure of language competence,
        while it's true that nothing definitive has been developed, that's
        not to say that considerable progress isn't being made regarding
        what certain tests reveal and what they don't, which I think is more
        to
        the point. Tests can now be made in ways that increase their
        validity (the degree to which a test measures the mental/linguistic
        constructs it is meant to measure, and no others) and their
        reliability (the "imperviousness" of that measure to outside
        factors) in ways that are mathematically demonstrable.
        Alas, however, what no one can do is to give any guarantee that the
        people who have to make decisions based on a person's test scores
        (teachers, admissions boards, human resources departments,
        scholarship committees, etc) will actually have any real
        understanding how to interpret those scores judiciously.

        On a lighter level, you know, I find it extremely curious and
        frustrating to realize that, within about 15 minutes of talking to
        any non-native speaker, I have a darn good idea of his/her
        communicative level (especially compared with the thousands of
        others I have talked to and taught in my life). What surprises me
        and frightens me the most is that I am rarely very far off the mark--
        I hardly ever discover that my initial assessment was very wrong.
        And I can do that with no real formal test, almost with my eyes
        closed.

        I can only conclude that H. Widdowson, as Scott cited earlier, was
        just being a bit grouchy that day because he's never been very good
        at language testing and thinks nobody else can be, either.


        John in Madrid
        (Nice to be back on line again and reading everyone's posts--sorry
        for the long absence!)
      • djn@dennisnewson.de
        John, How good are you at assessing a learner s written English? I always thought I was pretty good and that two pages of A4 was more of a sample than I needed
        Message 3 of 10 , Jan 1, 2004
        View Source
        • 0 Attachment
          John,

          How good are you at assessing a learner's written English? I always thought I was pretty
          good and that two pages of A4 was more of a sample than I needed to roughly assess a
          writer's standard on (the German method) a scale of 1 (excellent) to 6 (miserable).

          And then I had a go at qualifying as an IELTS examiner and missed (their) mark nearly
          all the time!

          (I had no difficulty assessing spoken English).


          Dennis
        • Adrian Tennant
          ... to realize that, within about 15 minutes of talking to ... level (especially compared with the thousands of ... frightens me the most is that I am rarely
          Message 4 of 10 , Jan 2, 2004
          View Source
          • 0 Attachment
            John wrote:

            > On a lighter level, you know, I find it extremely curious and frustrating
            to realize that, within about 15 minutes of talking to
            > any non-native speaker, I have a darn good idea of his/her communicative
            level (especially compared with the thousands of
            > others I have talked to and taught in my life). What surprises me and
            frightens me the most is that I am rarely very far off the > mark-- I
            hardly ever discover that my initial assessment was very wrong. And I can do
            that with no real formal test, almost > with my eyes closed.

            Two things caught my attention here. Firstly, 'any non-native speaker', I
            wonder how you know they are 'non-native'? I have met hundreds of
            non-natives (sic) whose language ability is better than most native
            speakers. Secondly, what 'markers' are you using for this assessment? I
            wonder if these 'markers' are grammatical?

            Dr Evil
          • John Franklin Nelson
            Dennis, I know what you mean. The linguistic competencies at play in producing written English really do seem to require using a different way to measure them.
            Message 5 of 10 , Jan 2, 2004
            View Source
            • 0 Attachment
              Dennis,

              I know what you mean. The linguistic competencies at play in
              producing written English really do seem to require using a
              different way to measure them. Just try breaking down that "1" in
              the "1 (excellent) to 6 (miserable) scale" you mention into
              descriptors. Then try doing the same for a 1-to-6 scale in spoken
              English, and you'll see we're dealing with a different kettle of
              fish. As for the IELTS, and of course for any other test, you may
              find that their descriptors are at odds with your own set of
              criteria, and that, even when you both agree on some specific
              construct (say, for example, "smooth and logical transitions"),
              your "tolerance" level and criteria for what constitutes acceptable
              and what does not--especially when the test subject's competence in
              that construct is only partial--varies considerably.

              I did some work with the IELTS when I was in Bangkok two years ago
              and was a bit disappointed in its sampling and scoring method. They
              do try to raise inter-rater reliability by insisting that all IELTS
              scorers assess in the same way, which is good. But to take it to an
              extreme, I too could attain high inter-rater reliability if I were
              to form a group of examiners by telling them to give high scores to
              papers using a wide margin, and low scores to people who leave no
              margin at all. The test would then be extremely reliable, but of
              course not at all valid, at least not if anyone were silly enough to
              try using those scores as a judge of language competence. My own
              conclusion from my IELTS experience there was that the IELTS is in
              fact a business first and foremost, and that those students who are
              lucky enough to study an IELTS prep course with a teacher who
              actually does IELTS scoring will get the best scores. An outstanding
              business plan, no doubt, but any real advances in language testing
              in such a context can only be haphazard at best.

              All the best,
              John in Madrid

              PS: Yes, Scott and all, I'm still skirting the issue regarding
              appropriate testing consonant with DOGME. It'll come, it'll come!!


              --- In dogme@yahoogroups.com, djn@d... wrote:
              > John,
              >
              > How good are you at assessing a learner's written English? I
              always thought I was pretty
              > good and that two pages of A4 was more of a sample than I needed
              to roughly assess a
              > writer's standard on (the German method) a scale of 1 (excellent)
              to 6 (miserable).
              >
              > And then I had a go at qualifying as an IELTS examiner and missed
              (their) mark nearly
              > all the time!
              >
              > (I had no difficulty assessing spoken English).
              >
              >
              > Dennis
            • John Franklin Nelson
              ... frustrating ... communicative ... me and ... I ... And I can do ... speaker , I ... assessment? I ... Dr Evil, ... speaker , I wonder how you know they are
              Message 6 of 10 , Jan 2, 2004
              View Source
              • 0 Attachment
                --- In dogme@yahoogroups.com, "Adrian Tennant" <adrian.tennant@n...>
                wrote:
                > John wrote:
                >
                > > On a lighter level, you know, I find it extremely curious and
                frustrating
                > to realize that, within about 15 minutes of talking to
                > > any non-native speaker, I have a darn good idea of his/her
                communicative
                > level (especially compared with the thousands of
                > > others I have talked to and taught in my life). What surprises
                me and
                > frightens me the most is that I am rarely very far off the > mark--
                I
                > hardly ever discover that my initial assessment was very wrong.
                And I can do
                > that with no real formal test, almost > with my eyes closed.
                >
                > Two things caught my attention here. Firstly, 'any non-native
                speaker', I
                > wonder how you know they are 'non-native'? I have met hundreds of
                > non-natives (sic) whose language ability is better than most native
                > speakers. Secondly, what 'markers' are you using for this
                assessment? I
                > wonder if these 'markers' are grammatical?
                >
                > Dr Evil


                Dr Evil,
                You wrote:
                > Two things caught my attention here. Firstly, 'any non-native
                speaker', I wonder how you know they are 'non-native'? <

                First point: as an (albeit slovenly) rule of thumb based only on
                neurolinguistic and psycholinguistic research findings, I tend to
                classify anyone (myself included) whose acquisition of Language X
                has principally taken place after puberty as a "non-native" speaker
                of that language. It has nothing to do with how well or how poorly
                that speaker's use of the language approximates a norm or standard,
                nor the particular variety or varieties of Language X that person
                happens to speak.

                You also wrote:
                >I have met hundreds of non-natives (sic) whose language ability is
                better than most native
                speakers.<

                I enjoy thinking about what you mean by someone's language ability
                being better than someone else's. That's the crux of it, and any and
                all decriptions you and everyone else in the group can provide will
                be of enormous use to us as we untangle the DOGME testing question.
                It's a scary one to answer, since I think we're all aware that it
                reveals a lot about our perceptions, opinions, and misconceptions.
                For my part, I'll say in answer to your second point (You wrote:
                >Secondly, what 'markers' are you using for this assessment? I
                wonder if these 'markers' are grammatical? <) that grammatical
                competency indeed is one of the many pieces of the puzzle. It's also
                the piece that has been put most under the microscope, discussed the
                most in all corners of the Earth, and most successfully been
                encapsulated in test design and procedure. Too bad it's just one
                piece. What the other pieces are is what we need to speak about here.

                So, my markers (in a nutshell) include sociolinguistic appropriacy
                to a great extent, listening ability, and, to a lesser extent,
                phonological competence (pronunciation). My feeling is that what we
                like so much about DOGME are all the things that fall under the
                realm of sociolinguistic competence. It's what has too long and too
                often been shirked off in more "traditional" language teaching
                approaches. It's the most elusive and least understood of the
                competencies, and yet, many sincere testing experts are concurring
                that it may be the biggest single contributor to general language
                proficiency. So what is it, folks? What is DOGME enabling our
                students to do well at? Go ahead and dive in with your views,
                whatever they may be, as long as they are sincere, so we can start
                to see what pieces of the puzzle we're looking at.

                John in Madrid
              Your message has been successfully submitted and would be delivered to recipients shortly.