Loading ...
Sorry, an error occurred while loading the content.

Re: [Cheetahtemplate-discuss] unicode troubles 2

Expand Messages
  • ml
    I looked into the compiler s source code and I found no trace of any switch. So I m suggesting two solutions: 1) automatic ... +++ /tmp/CompilerNew.py
    Message 1 of 9 , Jan 27, 2007
    View Source
    • 0 Attachment
      I looked into the compiler's source code and I found no trace of any
      switch. So I'm suggesting two solutions:

      1) automatic
      --- /tmp/Compiler.py 2007-01-27 09:53:00.000000000 +0100
      +++ /tmp/CompilerNew.py 2007-01-27 09:54:24.000000000 +0100
      @@ -437,7 +437,10 @@
      self._methodBodyChunks[-1] = self._methodBodyChunks[-1] + appendage

      def addWriteChunk(self, chunk):
      - self.addChunk('write(' + chunk + ')')
      + if self._moduleCompiler._moduleEncoding == 'ascii':
      + self.addChunk('write(' + chunk + ')')
      + else:
      + self.addChunk('write(u' + chunk + ')')

      def addFilteredChunk(self, chunk, filterArgs=None, rawExpr=None,
      lineCol=None):
      if filterArgs is None:

      2) switch driven
      --- /tmp/Compiler.py 2007-01-27 09:53:00.000000000 +0100
      +++ /tmp/CompilerNew.py 2007-01-27 10:00:00.000000000 +0100
      @@ -53,6 +53,7 @@
      # builtin, or local var that doesn't need
      # looking up in the searchlist BUT use
      # namemapper on the rest of the lookup
      + 'useUnicode': False, # unicode output is preferred
      'allowSearchListAsMethArg': True,
      'useAutocalling': True, # detect and call callable()'s, requires
      NameMapper
      'useStackFrames': True, # use NameMapper.valueFromFrameOrSearchList
      @@ -437,7 +438,10 @@
      self._methodBodyChunks[-1] = self._methodBodyChunks[-1] + appendage

      def addWriteChunk(self, chunk):
      - self.addChunk('write(' + chunk + ')')
      + if self._moduleCompiler.setting('useUnicode'):
      + self.addChunk('write(u' + chunk + ')')
      + else:
      + self.addChunk('write(' + chunk + ')')

      def addFilteredChunk(self, chunk, filterArgs=None, rawExpr=None,
      lineCol=None):
      if filterArgs is None:

      Well what is your opinion? :-)

      David


      ml napsal(a):
      > Hi!
      >
      > Scenario:
      > Template is in UTF-8, placeholders are unicode -> UnicodeDecodeError
      >
      > I read the "unicode troubles" thread and there is a simple solution for
      > non-compiled templates:
      > s = codecs.open(“input.txt”, “r”, “utf-8”) # will get a unicode
      > t = Template(source=s)
      > out = t.respond()
      >
      > But how about compiled templates?
      > Let's have a simple template x.tmpl:
      >
      > #encoding utf-8
      > ščřž
      > $p
      >
      > and it's compiled version is used in a script:
      >
      > import x
      > tpl = x.x(searchList=[{"p":u"unicode"}])
      > buf = tpl.respond()
      >
      > This will fail. In the compiled template (x.py) there is a command to
      > print the template's second line:
      > write('\xc5\xa1\xc4\x8d\xc5\x99\xc5\xbe\n')
      > and this is the core of the problem. I would expect
      > write(u'ščřž\n')
      > or at least
      > write(u'\xc5\xa1\xc4\x8d\xc5\x99\xc5\xbe\n')
      >
      > Is there any switch for that?
      >
      > Thank you!
      >
      > David
      >
      >
      > -------------------------------------------------------------------------
      > Take Surveys. Earn Cash. Influence the Future of IT
      > Join SourceForge.net's Techsay panel and you'll get the chance to share your
      > opinions on IT & business topics through brief surveys - and earn cash
      > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
      > _______________________________________________
      > Cheetahtemplate-discuss mailing list
      > Cheetahtemplate-discuss@...
      > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


      -------------------------------------------------------------------------
      Take Surveys. Earn Cash. Influence the Future of IT
      Join SourceForge.net's Techsay panel and you'll get the chance to share your
      opinions on IT & business topics through brief surveys - and earn cash
      http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
      _______________________________________________
      Cheetahtemplate-discuss mailing list
      Cheetahtemplate-discuss@...
      https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
    • ml
      I forgot one more neccessary hack: 499c503,505
      Message 2 of 9 , Feb 5, 2007
      View Source
      • 0 Attachment
        I forgot one more neccessary hack:
        499c503,505
        < self.addWriteChunk( repr(strConst).replace('\\012','\\n'))
        ---
        > _str = strConst.replace('\n', '\\n')
        > _str = _str.replace('\'','\\\'')
        > self.addWriteChunk("'" + _str + "'")

        Because the current implementation doesn't preserve characters
        (repr(...)) so #encoding and the solution below is useless.

        David




        Tavis Rudd napsal(a):
        > Hi David,
        > I prefer the switch driven approach, however, addWriteChunk isn't the
        > correct place to add it. It needs to go in commitStrConst(). I'm
        > experimenting with that now and will check something in to the cvs today.
        > Cheers,
        > Tavis
        >
        > On Sat, 27 Jan 2007, ml wrote:
        >
        >> I looked into the compiler's source code and I found no trace of any
        > switch. So I'm suggesting two solutions:
        >
        > 1) automatic
        > --- /tmp/Compiler.py 2007-01-27 09:53:00.000000000 +0100
        > +++ /tmp/CompilerNew.py 2007-01-27 09:54:24.000000000 +0100
        > @@ -437,7 +437,10 @@
        > self._methodBodyChunks[-1] = self._methodBodyChunks[-1] +
        > appendage
        >
        > def addWriteChunk(self, chunk):
        > - self.addChunk('write(' + chunk + ')')
        > + if self._moduleCompiler._moduleEncoding == 'ascii':
        > + self.addChunk('write(' + chunk + ')')
        > + else:
        > + self.addChunk('write(u' + chunk + ')')
        >
        > def addFilteredChunk(self, chunk, filterArgs=None, rawExpr=None,
        > lineCol=None):
        > if filterArgs is None:
        >
        > 2) switch driven
        > --- /tmp/Compiler.py 2007-01-27 09:53:00.000000000 +0100
        > +++ /tmp/CompilerNew.py 2007-01-27 10:00:00.000000000 +0100
        > @@ -53,6 +53,7 @@
        > # builtin, or local var that doesn't need
        > # looking up in the searchlist BUT use
        > # namemapper on the rest of the lookup
        > + 'useUnicode': False, # unicode output is preferred
        > 'allowSearchListAsMethArg': True,
        > 'useAutocalling': True, # detect and call callable()'s, requires
        > NameMapper
        > 'useStackFrames': True, # use NameMapper.valueFromFrameOrSearchList
        > @@ -437,7 +438,10 @@
        > self._methodBodyChunks[-1] = self._methodBodyChunks[-1] +
        > appendage
        >
        > def addWriteChunk(self, chunk):
        > - self.addChunk('write(' + chunk + ')')
        > + if self._moduleCompiler.setting('useUnicode'):
        > + self.addChunk('write(u' + chunk + ')')
        > + else:
        > + self.addChunk('write(' + chunk + ')')
        >
        > def addFilteredChunk(self, chunk, filterArgs=None, rawExpr=None,
        > lineCol=None):
        > if filterArgs is None:
        >
        > Well what is your opinion? :-)
        >
        > David
        >
        >
        > ml napsal(a):
        >> Hi!
        >>
        >> Scenario:
        >> Template is in UTF-8, placeholders are unicode -> UnicodeDecodeError
        >>
        >> I read the "unicode troubles" thread and there is a simple solution for
        >> non-compiled templates:
        >> s = codecs.open(“input.txt”, “r”, “utf-8”) # will get a unicode
        >> t = Template(source=s)
        >> out = t.respond()
        >>
        >> But how about compiled templates?
        >> Let's have a simple template x.tmpl:
        >>
        >> #encoding utf-8
        >> ščřž
        >> $p
        >>
        >> and it's compiled version is used in a script:
        >>
        >> import x
        >> tpl = x.x(searchList=[{"p":u"unicode"}])
        >> buf = tpl.respond()
        >>
        >> This will fail. In the compiled template (x.py) there is a command to
        >> print the template's second line:
        >> write('\xc5\xa1\xc4\x8d\xc5\x99\xc5\xbe\n')
        >> and this is the core of the problem. I would expect
        >> write(u'ščřž\n')
        >> or at least
        >> write(u'\xc5\xa1\xc4\x8d\xc5\x99\xc5\xbe\n')
        >>
        >> Is there any switch for that?
        >>
        >> Thank you!
        >>
        >> David
        >>
        >>
        >> -------------------------------------------------------------------------
        >> Take Surveys. Earn Cash. Influence the Future of IT
        >> Join SourceForge.net's Techsay panel and you'll get the chance to
        >> share your
        >> opinions on IT & business topics through brief surveys - and earn cash
        >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
        >> _______________________________________________
        >> Cheetahtemplate-discuss mailing list
        >> Cheetahtemplate-discuss@...
        >> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
        >
        >
        > -------------------------------------------------------------------------
        > Take Surveys. Earn Cash. Influence the Future of IT
        > Join SourceForge.net's Techsay panel and you'll get the chance to share
        > your
        > opinions on IT & business topics through brief surveys - and earn cash
        > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
        > _______________________________________________
        > Cheetahtemplate-discuss mailing list
        > Cheetahtemplate-discuss@...
        > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


        -------------------------------------------------------------------------
        Using Tomcat but need to do more? Need to support web services, security?
        Get stuff done quickly with pre-integrated technology to make your job easier.
        Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
        http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
        _______________________________________________
        Cheetahtemplate-discuss mailing list
        Cheetahtemplate-discuss@...
        https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
      • Tavis Rudd
        Hi David, I ve just checked in a change that accomplishes the same thing. Thanks, Tavis ... Take Surveys. Earn Cash. Influence the Future of IT Join
        Message 3 of 9 , Mar 28, 2007
        View Source
        • 0 Attachment
          Hi David,
          I've just checked in a change that accomplishes the same thing.
          Thanks,
          Tavis

          On Mon, 5 Feb 2007, ml wrote:

          > I forgot one more neccessary hack:
          > 499c503,505
          > < self.addWriteChunk( repr(strConst).replace('\\012','\\n'))
          > ---
          >> _str = strConst.replace('\n', '\\n')
          >> _str = _str.replace('\'','\\\'')
          >> self.addWriteChunk("'" + _str + "'")
          >
          > Because the current implementation doesn't preserve characters
          > (repr(...)) so #encoding and the solution below is useless.
          >
          > David
          >
          >
          >
          >
          > Tavis Rudd napsal(a):
          >> Hi David,
          >> I prefer the switch driven approach, however, addWriteChunk isn't the
          >> correct place to add it. It needs to go in commitStrConst(). I'm
          >> experimenting with that now and will check something in to the cvs today.
          >> Cheers,
          >> Tavis
          >>
          >> On Sat, 27 Jan 2007, ml wrote:
          >>
          >>> I looked into the compiler's source code and I found no trace of any
          >> switch. So I'm suggesting two solutions:
          >>
          >> 1) automatic
          >> --- /tmp/Compiler.py 2007-01-27 09:53:00.000000000 +0100
          >> +++ /tmp/CompilerNew.py 2007-01-27 09:54:24.000000000 +0100
          >> @@ -437,7 +437,10 @@
          >> self._methodBodyChunks[-1] = self._methodBodyChunks[-1] +
          >> appendage
          >>
          >> def addWriteChunk(self, chunk):
          >> - self.addChunk('write(' + chunk + ')')
          >> + if self._moduleCompiler._moduleEncoding == 'ascii':
          >> + self.addChunk('write(' + chunk + ')')
          >> + else:
          >> + self.addChunk('write(u' + chunk + ')')
          >>
          >> def addFilteredChunk(self, chunk, filterArgs=None, rawExpr=None,
          >> lineCol=None):
          >> if filterArgs is None:
          >>
          >> 2) switch driven
          >> --- /tmp/Compiler.py 2007-01-27 09:53:00.000000000 +0100
          >> +++ /tmp/CompilerNew.py 2007-01-27 10:00:00.000000000 +0100
          >> @@ -53,6 +53,7 @@
          >> # builtin, or local var that doesn't need
          >> # looking up in the searchlist BUT use
          >> # namemapper on the rest of the lookup
          >> + 'useUnicode': False, # unicode output is preferred
          >> 'allowSearchListAsMethArg': True,
          >> 'useAutocalling': True, # detect and call callable()'s, requires
          >> NameMapper
          >> 'useStackFrames': True, # use NameMapper.valueFromFrameOrSearchList
          >> @@ -437,7 +438,10 @@
          >> self._methodBodyChunks[-1] = self._methodBodyChunks[-1] +
          >> appendage
          >>
          >> def addWriteChunk(self, chunk):
          >> - self.addChunk('write(' + chunk + ')')
          >> + if self._moduleCompiler.setting('useUnicode'):
          >> + self.addChunk('write(u' + chunk + ')')
          >> + else:
          >> + self.addChunk('write(' + chunk + ')')
          >>
          >> def addFilteredChunk(self, chunk, filterArgs=None, rawExpr=None,
          >> lineCol=None):
          >> if filterArgs is None:
          >>
          >> Well what is your opinion? :-)
          >>
          >> David
          >>
          >>
          >> ml napsal(a):
          >>> Hi!
          >>>
          >>> Scenario:
          >>> Template is in UTF-8, placeholders are unicode -> UnicodeDecodeError
          >>>
          >>> I read the "unicode troubles" thread and there is a simple solution for
          >>> non-compiled templates:
          >>> s = codecs.open(“input.txt”, “r”, “utf-8”) # will get a unicode
          >>> t = Template(source=s)
          >>> out = t.respond()
          >>>
          >>> But how about compiled templates?
          >>> Let's have a simple template x.tmpl:
          >>>
          >>> #encoding utf-8
          >>> ščřž
          >>> $p
          >>>
          >>> and it's compiled version is used in a script:
          >>>
          >>> import x
          >>> tpl = x.x(searchList=[{"p":u"unicode"}])
          >>> buf = tpl.respond()
          >>>
          >>> This will fail. In the compiled template (x.py) there is a command to
          >>> print the template's second line:
          >>> write('\xc5\xa1\xc4\x8d\xc5\x99\xc5\xbe\n')
          >>> and this is the core of the problem. I would expect
          >>> write(u'ščřž\n')
          >>> or at least
          >>> write(u'\xc5\xa1\xc4\x8d\xc5\x99\xc5\xbe\n')
          >>>
          >>> Is there any switch for that?
          >>>
          >>> Thank you!
          >>>
          >>> David
          >>>
          >>>
          >>> -------------------------------------------------------------------------
          >>> Take Surveys. Earn Cash. Influence the Future of IT
          >>> Join SourceForge.net's Techsay panel and you'll get the chance to
          >>> share your
          >>> opinions on IT & business topics through brief surveys - and earn cash
          >>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
          >>> _______________________________________________
          >>> Cheetahtemplate-discuss mailing list
          >>> Cheetahtemplate-discuss@...
          >>> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
          >>
          >>
          >> -------------------------------------------------------------------------
          >> Take Surveys. Earn Cash. Influence the Future of IT
          >> Join SourceForge.net's Techsay panel and you'll get the chance to share
          >> your
          >> opinions on IT & business topics through brief surveys - and earn cash
          >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
          >> _______________________________________________
          >> Cheetahtemplate-discuss mailing list
          >> Cheetahtemplate-discuss@...
          >> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
          >
          >
        • Miha Vrhovnik
          Hi Travis, any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread? Or at at least the WebFilter not being Unicode enabled? Regards, Miha ...
          Message 4 of 9 , Mar 28, 2007
          View Source
          • 0 Attachment
            Hi Travis,

            any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread?
            Or at at least the WebFilter not being Unicode enabled?

            Regards,
            Miha

            "Tavis Rudd" <tavis@...> wrote on 28.3.2007 18:38:06:
            >Hi David,
            >I've just checked in a change that accomplishes the same thing.
            >Thanks,
            >Tavis
            >



            --
            It's time to get rid of your current e-mail client ...
            ... and start using si.Mail.

            It's small & free. ( http://simail.sourceforge.net/ )

            -------------------------------------------------------------------------
            Take Surveys. Earn Cash. Influence the Future of IT
            Join SourceForge.net's Techsay panel and you'll get the chance to share your
            opinions on IT & business topics through brief surveys-and earn cash
            http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
            _______________________________________________
            Cheetahtemplate-discuss mailing list
            Cheetahtemplate-discuss@...
            https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
          • Miha Vrhovnik
            Hi Travis, any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread? Or at at least the WebFilter not being Unicode enabled? Regards, Miha ...
            Message 5 of 9 , Mar 28, 2007
            View Source
            • 0 Attachment
              Hi Travis,

              any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread?
              Or at at least the WebFilter not being Unicode enabled?

              Regards,
              Miha

              "Tavis Rudd" <tavis@...> wrote on 28.3.2007 18:38:06:
              >Hi David,
              >I've just checked in a change that accomplishes the same thing.
              >Thanks,
              >Tavis
              >



              --
              It's time to get rid of your current e-mail client ...
              ... and start using si.Mail.

              It's small & free. ( http://simail.sourceforge.net/ )

              -------------------------------------------------------------------------
              Take Surveys. Earn Cash. Influence the Future of IT
              Join SourceForge.net's Techsay panel and you'll get the chance to share your
              opinions on IT & business topics through brief surveys-and earn cash
              http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
              _______________________________________________
              Cheetahtemplate-discuss mailing list
              Cheetahtemplate-discuss@...
              https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
            • Miha Vrhovnik
              Hi Travis, any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread? Or at at least the WebFilter not being Unicode enabled? Regards, Miha ...
              Message 6 of 9 , Mar 28, 2007
              View Source
              • 0 Attachment
                Hi Travis,

                any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread?
                Or at at least the WebFilter not being Unicode enabled?

                Regards,
                Miha

                "Tavis Rudd" <tavis@...> wrote on 28.3.2007 18:38:06:
                >Hi David,
                >I've just checked in a change that accomplishes the same thing.
                >Thanks,
                >Tavis
                >



                --
                It's time to get rid of your current e-mail client ...
                ... and start using si.Mail.

                It's small & free. ( http://simail.sourceforge.net/ )

                -------------------------------------------------------------------------
                Take Surveys. Earn Cash. Influence the Future of IT
                Join SourceForge.net's Techsay panel and you'll get the chance to share your
                opinions on IT & business topics through brief surveys-and earn cash
                http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                _______________________________________________
                Cheetahtemplate-discuss mailing list
                Cheetahtemplate-discuss@...
                https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
              • Miha Vrhovnik
                Hi Travis, any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread? Or at at least the WebFilter not being Unicode enabled? Regards, Miha ...
                Message 7 of 9 , Mar 28, 2007
                View Source
                • 0 Attachment
                  Hi Travis,

                  any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread?
                  Or at at least the WebFilter not being Unicode enabled?

                  Regards,
                  Miha

                  "Tavis Rudd" <tavis@...> wrote on 28.3.2007 18:38:06:
                  >Hi David,
                  >I've just checked in a change that accomplishes the same thing.
                  >Thanks,
                  >Tavis
                  >



                  --
                  It's time to get rid of your current e-mail client ...
                  ... and start using si.Mail.

                  It's small & free. ( http://simail.sourceforge.net/ )

                  -------------------------------------------------------------------------
                  Take Surveys. Earn Cash. Influence the Future of IT
                  Join SourceForge.net's Techsay panel and you'll get the chance to share your
                  opinions on IT & business topics through brief surveys-and earn cash
                  http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                  _______________________________________________
                  Cheetahtemplate-discuss mailing list
                  Cheetahtemplate-discuss@...
                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                • Miha Vrhovnik
                  Hi Travis, any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread? Or at at least the WebFilter not being Unicode enabled? Regards, Miha ...
                  Message 8 of 9 , Mar 28, 2007
                  View Source
                  • 0 Attachment
                    Hi Travis,

                    any chances to fix the other 2 bugs. Mentioned in Unicode troubles thread?
                    Or at at least the WebFilter not being Unicode enabled?

                    Regards,
                    Miha

                    "Tavis Rudd" <tavis@...> wrote on 28.3.2007 18:38:06:
                    >Hi David,
                    >I've just checked in a change that accomplishes the same thing.
                    >Thanks,
                    >Tavis
                    >



                    --
                    It's time to get rid of your current e-mail client ...
                    ... and start using si.Mail.

                    It's small & free. ( http://simail.sourceforge.net/ )

                    -------------------------------------------------------------------------
                    Take Surveys. Earn Cash. Influence the Future of IT
                    Join SourceForge.net's Techsay panel and you'll get the chance to share your
                    opinions on IT & business topics through brief surveys-and earn cash
                    http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                    _______________________________________________
                    Cheetahtemplate-discuss mailing list
                    Cheetahtemplate-discuss@...
                    https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                  Your message has been successfully submitted and would be delivered to recipients shortly.