Loading ...
Sorry, an error occurred while loading the content.

Case insensitivity and the tolower function

Expand Messages
  • John Lambert
    I have been having more problems with the tolower function and it s handling of EOF_CHAR. It seems that STL port will truncate the -1 down to 0xff :-( So I
    Message 1 of 3 , Nov 29, 2000
    • 0 Attachment
      I have been having more problems with the tolower function and it's handling
      of EOF_CHAR.

      It seems that STL port will truncate the -1 down to 0xff :-(

      So I propose that EOF is not passed through the tolower function by changing
      the function:


      int CharScanner::toLower(int c) const
      {
      return (c == EOF_CHAR ? EOF_CHAR : tolower(c));
      }


      Ric, is this a reasonable change to include in the next version?

      Thanks,

      John
    • Ric Klaren
      Hi, ... Depends if this code is still in the next version for wide character support all this stuff needs to be changed anyway. EOF_CHAR should really be
      Message 2 of 3 , Nov 30, 2000
      • 0 Attachment
        Hi,

        On Wed, Nov 29, 2000 at 11:20:06AM -0800, John Lambert wrote:
        > I have been having more problems with the tolower function and it's handling
        > of EOF_CHAR.
        >
        > It seems that STL port will truncate the -1 down to 0xff :-(
        >
        > So I propose that EOF is not passed through the tolower function by changing
        > the function:
        >
        > int CharScanner::toLower(int c) const
        > {
        > return (c == EOF_CHAR ? EOF_CHAR : tolower(c));
        > }
        >
        > Ric, is this a reasonable change to include in the next version?

        Depends if this code is still in the next version for wide character
        support all this stuff needs to be changed anyway. EOF_CHAR should really
        be something like char_traits<char>::eof() (or char_traits<wchar>::eof()
        for wide characters). But this probably breaks a lot of compilers/support
        libraries. Btw could you check if changing EOF_CHAR to
        char_traits<char>::eof() fixes this? Or as a longshot shouldn't that be
        std::tolower(c)....

        Anyway for now I'll check this workaround into my devel tree.

        The new version will be up in a few minutes at:

        http://wwwhome.cs.utwente.nl/~klaren/antlr/antlr-devel.tar.gz

        Cheers,

        Ric


        >
        > Thanks,
        >
        > John
        >
        >
        >
        >
        >

        --
        -----+++++*****************************************************+++++++++-------
        ---- Ric Klaren ----- klaren@... ----- +31 4893722 ----
        -----+++++*****************************************************+++++++++-------
        Why don't we just invite them to dinner and massacre them all when they're
        drunk? You heard the man. There's seven hundred thousand of them.
        Ah? ... So it'd have to be something simple with pasta, then.
        --- From: Interesting Times by Terry Pratchet
        -----+++++*****************************************************+++++++++-------
      • Alexander Lenski
        Hi, I am a little bit confused with the CharScaner::consume() implementation (the code from the CharScanner is shown below). Anyway (on true == caseSensitive
        Message 3 of 3 , Dec 3, 2000
        • 0 Attachment
          Hi,
          I am a little bit confused with the CharScaner::consume() implementation (the code from the CharScanner is
          shown below). Anyway (on true == caseSensitive or on false == caseSensitive) the inputState->getInput().LA(1)
          is called to get a char (see the CharScaner's  consume() and LA() implementation below).
          Why is part under the question not the (to keep the caseSensitive in use):
          // This is a start of the ??? 
                    int c = LA(1);
                    append(c);
          // This is the end
          Or why is it not the (to do the same just with better performance):
          // This is a start of the ??? 
                    int c = inputState->getInput().LA(1);
                    append(c);
          // This is the end
          Original code:
           
          void CharScanner::consume()
          {    if (inputState->guessing == 0)
               {
          // This is a start of the ??? 
                    int c = LA(1);
                    if (caseSensitive)
                    {   append(c);
                    }
                    else
                    {   // use input.LA(), not LA(), to get original case
                         // CharScanner.LA() would toLower it.
                         append(inputState->getInput().LA(1));
                    }
          // This is the end
                    if (c == '\t')
                    {    tab();
                    }
                    else
                    {    inputState->column++;
                    }
               }
               inputState->getInput().consume();
          }
          inline int CharScanner::LA(int i)
          {    if ( caseSensitive )
               {    return inputState->getInput().LA(i);
               }
               else
               {    return toLower(inputState->getInput().LA(i));
               }
          }
          Thanks,
          Alexander Lenski
           
          ----- Original Message -----
          From: "Ric Klaren" <klaren@...>
          Sent: Thursday, November 30, 2000 4:17 AM
          Subject: [antlr-interest] Re: Case insensitivity and the tolower function

          > Hi,
          >
          > On Wed, Nov 29,
          2000 at 11:20:06AM -0800, John Lambert wrote:
          > > I have been having
          more problems with the tolower function and it's handling
          > > of
          EOF_CHAR.
          > >
          > > It seems that STL port will truncate the -1
          down to 0xff :-(
          > >
          > > So I propose that EOF is not passed
          through the tolower function by changing
          > > the function:
          > >
          > > int CharScanner::toLower(int c) const
          > > {
          > >
          return (c == EOF_CHAR ? EOF_CHAR : tolower(c));
          > > }
          > >
          > > Ric, is this a reasonable change to include in the next
          version?
          >
          > Depends if this code is still in the next version for
          wide character
          > support all this stuff needs to be changed anyway.
          EOF_CHAR should really
          > be something like char_traits<char>::eof()
          (or char_traits<wchar>::eof()
          > for wide characters). But this
          probably breaks a lot of compilers/support
          > libraries. Btw could you
          check if changing EOF_CHAR to
          > char_traits<char>::eof() fixes this?
          Or as a longshot shouldn't that be
          > std::tolower(c)....
          >
          >
          Anyway for now I'll check this workaround into my devel tree.
          >
          >
          The new version will be up in a few minutes at:
          >
          >
          href="http://wwwhome.cs.utwente.nl/~klaren/antlr/antlr-devel.tar.gz">http://wwwhome.cs.utwente.nl/~klaren/antlr/antlr-devel.tar.gz
          >
          > Cheers,
          >
          > Ric
          >
          >
          > >
          > > Thanks,
          > >
          > > John
          > >
          > >
          > >
          > >
          > >
          >
          > --
          >
          -----+++++*****************************************************+++++++++-------
          >
                ---- Ric Klaren -----
          klaren@... ----- +31 4893722  ----
          >
          -----+++++*****************************************************+++++++++-------
          >
            Why don't we just invite them to dinner and massacre them all when they're
          >   drunk? You heard the man. There's seven hundred thousand
          of them.
          >   Ah? ... So it'd have to be something simple with pasta,
          then.
          >
                                        --- From: Interesting Times by Terry Pratchet
          >
          -----+++++*****************************************************+++++++++-------
          >
          > -------------------------- eGroups Sponsor
          -------------------------~-~>
          > eLerts
          > It's Easy. It's Fun.
          Best of All, it's Free!
          >
          href="http://click.egroups.com/1/9699/1/_/492272/_/975575859/">http://click.egroups.com/1/9699/1/_/492272/_/975575859/
          > ---------------------------------------------------------------------_->
          >
          >
          >
          >
        Your message has been successfully submitted and would be delivered to recipients shortly.