Loading ...
Sorry, an error occurred while loading the content.

[Cheetahtemplate-discuss] [PATCH] Convert unicode compiled template to an utf8 char buffer when writing to a file

Expand Messages
  • Jean-Baptiste Quenot
    Hello community, Please find attached a patch to address a problem when compiling Unicode templates with cheetah compile on the command-line. HTH, --
    Message 1 of 16 , Apr 1, 2009
    • 0 Attachment
      Hello community,

      Please find attached a patch to address a problem when compiling
      Unicode templates with "cheetah compile" on the command-line.

      HTH,
      --
      Jean-Baptiste Quenot
      http://jbq.caraldi.com/
    • R. Tyler Ballance
      ... Can you elaborate a little bit on what the problem is? (maybe even with a test case? ;)) Does this problem just affect writing the output to a file, or
      Message 2 of 16 , Apr 1, 2009
      • 0 Attachment
        On Wed, Apr 01, 2009 at 12:12:54PM +0200, Jean-Baptiste Quenot wrote:
        > Hello community,
        >
        > Please find attached a patch to address a problem when compiling
        > Unicode templates with "cheetah compile" on the command-line.

        Can you elaborate a little bit on what the problem is? (maybe even with
        a test case? ;))

        Does this "problem" just affect writing the output to a file, or dynamically
        compiled templates as well?


        I'll squeeze this patch into the 2.1.1 release planned for the 16th
        regardless.

        > From 3fc83fb2f1d60fff083ff46d1e6295a2f999618b Mon Sep 17 00:00:00 2001
        > From: Jean-Baptiste Quenot <jbq@...>
        > Date: Wed, 1 Apr 2009 12:08:14 +0200
        > Subject: [PATCH] Convert unicode compiled template to an utf8 char buffer when writing to a file
        >
        > ---
        > src/CheetahWrapper.py | 2 +-
        > 1 files changed, 1 insertions(+), 1 deletions(-)
        > mode change 100755 => 100644 src/CheetahWrapper.py
        >
        > diff --git a/src/CheetahWrapper.py b/src/CheetahWrapper.py
        > old mode 100755
        > new mode 100644
        > index 5e3d6dd..75135eb
        > --- a/src/CheetahWrapper.py
        > +++ b/src/CheetahWrapper.py
        > @@ -622,7 +622,7 @@ be named according to the same rules as Python modules.""" % tup)
        > sys.stdout.write(output)
        > else:
        > f = open(dst, 'w')
        > - f.write(output)
        > + f.write(output.encode('utf8'))
        > f.close()
        >
        >
        > --
        > 1.6.0.4
        >

        > ------------------------------------------------------------------------------

        > _______________________________________________
        > Cheetahtemplate-discuss mailing list
        > Cheetahtemplate-discuss@...
        > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


        --
        -R. Tyler Ballance
        Slide, Inc.
      • Jean-Baptiste Quenot
        Oh yeah I should have included a stacktrace at least: Traceback (most recent call last): File /usr/bin/cheetah , line 3, in CheetahWrapper().main()
        Message 3 of 16 , Apr 1, 2009
        • 0 Attachment
          Oh yeah I should have included a stacktrace at least:

          Traceback (most recent call last):
          File "/usr/bin/cheetah", line 3, in <module>
          CheetahWrapper().main()
          File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
          line 186, in main
          meth()
          File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
          line 244, in compile
          self._compileOrFill()
          File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
          line 391, in _compileOrFill
          self._compileOrFillBundle(b)
          File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
          line 625, in _compileOrFillBundle
          f.write(output)
          UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
          position 10318: ordinal not in range(128)
          --
          Jean-Baptiste Quenot
          http://jbq.caraldi.com/

          ------------------------------------------------------------------------------
          _______________________________________________
          Cheetahtemplate-discuss mailing list
          Cheetahtemplate-discuss@...
          https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
        • mobiledreamers@gmail.com
          i keep getting this a lot ... -- Bidegg worlds best auction site http://bidegg.com ... _______________________________________________ Cheetahtemplate-discuss
          Message 4 of 16 , Apr 1, 2009
          • 0 Attachment
            i keep getting this a lot

            On Wed, Apr 1, 2009 at 11:44 AM, Jean-Baptiste Quenot <jbq@...> wrote:
            Oh yeah I should have included a stacktrace at least:

            Traceback (most recent call last):
             File "/usr/bin/cheetah", line 3, in <module>
               CheetahWrapper().main()
             File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
            line 186, in main
               meth()
             File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
            line 244, in compile
               self._compileOrFill()
             File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
            line 391, in _compileOrFill
               self._compileOrFillBundle(b)
             File "/var/lib/python-support/python2.5/Cheetah/CheetahWrapper.py",
            line 625, in _compileOrFillBundle
               f.write(output)
            UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
            position 10318: ordinal not in range(128)
            --
            Jean-Baptiste Quenot
            http://jbq.caraldi.com/

            ------------------------------------------------------------------------------
            _______________________________________________
            Cheetahtemplate-discuss mailing list
            Cheetahtemplate-discuss@...
            https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss



            --
            Bidegg worlds best auction site
            http://bidegg.com
          • R. Tyler Ballance
            ... Wonderful, I ve applied your patch after running the full test suite against it, looks good to me :) This is currently in next and will go out with
            Message 5 of 16 , Apr 1, 2009
            • 0 Attachment
              On Wed, Apr 01, 2009 at 12:12:54PM +0200, Jean-Baptiste Quenot wrote:
              > Hello community,
              >
              > Please find attached a patch to address a problem when compiling
              > Unicode templates with "cheetah compile" on the command-line.

              Wonderful, I've applied your patch after running the full test suite
              against it, looks good to me :)


              This is currently in "next" and will go out with v2.1.1 on the 16th.


              Thanks for providing it as a .patch :)

              > From 3fc83fb2f1d60fff083ff46d1e6295a2f999618b Mon Sep 17 00:00:00 2001
              > From: Jean-Baptiste Quenot <jbq@...>
              > Date: Wed, 1 Apr 2009 12:08:14 +0200
              > Subject: [PATCH] Convert unicode compiled template to an utf8 char buffer when writing to a file
              >
              > ---
              > src/CheetahWrapper.py | 2 +-
              > 1 files changed, 1 insertions(+), 1 deletions(-)
              > mode change 100755 => 100644 src/CheetahWrapper.py
              >
              > diff --git a/src/CheetahWrapper.py b/src/CheetahWrapper.py
              > old mode 100755
              > new mode 100644
              > index 5e3d6dd..75135eb
              > --- a/src/CheetahWrapper.py
              > +++ b/src/CheetahWrapper.py
              > @@ -622,7 +622,7 @@ be named according to the same rules as Python modules.""" % tup)
              > sys.stdout.write(output)
              > else:
              > f = open(dst, 'w')
              > - f.write(output)
              > + f.write(output.encode('utf8'))
              > f.close()
              >
              >
              > --
              > 1.6.0.4
              >

              > ------------------------------------------------------------------------------

              > _______________________________________________
              > Cheetahtemplate-discuss mailing list
              > Cheetahtemplate-discuss@...
              > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


              --
              -R. Tyler Ballance
              Slide, Inc.
            • Jean-Baptiste Quenot
              ... I m using the unicode branch FWIW. I don t remember if every template is unicode is in the next or the unicode branch but anyway this patch
              Message 6 of 16 , Apr 1, 2009
              • 0 Attachment
                2009/4/2 R. Tyler Ballance <tyler@...>:
                > On Wed, Apr 01, 2009 at 12:12:54PM +0200, Jean-Baptiste Quenot wrote:
                >>
                >> Please find attached a patch to address a problem when compiling
                >> Unicode templates with "cheetah compile" on the command-line.
                >
                > Wonderful, I've applied your patch after running the full test suite
                > against it, looks good to me :)
                >
                >
                > This is currently in "next" and will go out with v2.1.1 on the 16th.

                I'm using the "unicode" branch FWIW. I don't remember if "every
                template is unicode" is in the "next" or the "unicode" branch but
                anyway this patch resolves the only problem encountered by switching
                my application to the "unicode" branch.
                --
                Jean-Baptiste Quenot
                http://jbq.caraldi.com/

                ------------------------------------------------------------------------------
                _______________________________________________
                Cheetahtemplate-discuss mailing list
                Cheetahtemplate-discuss@...
                https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
              • R. Tyler Ballance
                Aha, good to know, I ll cherry-pick it over ... From: Jean-Baptiste Quenot Sent: Wednesday, April 01, 2009 23:54 To:
                Message 7 of 16 , Apr 2, 2009
                • 0 Attachment
                  Aha, good to know, I'll cherry-pick it over


                  -----Original Message-----
                  From: Jean-Baptiste Quenot <jbq@...>
                  Sent: Wednesday, April 01, 2009 23:54
                  To: cheetahtemplate-discuss@...
                  Subject: Re: [Cheetahtemplate-discuss] [PATCH] Convert unicode compiled template to an utf8 char buffer when writing to a file

                  2009/4/2 R. Tyler Ballance <tyler@...>:
                  > On Wed, Apr 01, 2009 at 12:12:54PM +0200, Jean-Baptiste Quenot wrote:
                  >>
                  >> Please find attached a patch to address a problem when compiling
                  >> Unicode templates with "cheetah compile" on the command-line.
                  >
                  > Wonderful, I've applied your patch after running the full test suite
                  > against it, looks good to me :)
                  >
                  >
                  > This is currently in "next" and will go out with v2.1.1 on the 16th.

                  I'm using the "unicode" branch FWIW. I don't remember if "every
                  template is unicode" is in the "next" or the "unicode" branch but
                  anyway this patch resolves the only problem encountered by switching
                  my application to the "unicode" branch.
                  --
                  Jean-Baptiste Quenot
                  http://jbq.caraldi.com/

                  ------------------------------------------------------------------------------
                  _______________________________________________
                  Cheetahtemplate-discuss mailing list
                  Cheetahtemplate-discuss@...
                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


                  ------------------------------------------------------------------------------
                  _______________________________________________
                  Cheetahtemplate-discuss mailing list
                  Cheetahtemplate-discuss@...
                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                • Jean-Baptiste Quenot
                  I m starting getting this with the unicode branch: SyntaxError: Non-ASCII character xc3 in file /home/jbq/Template.py on line 255, but no encoding
                  Message 8 of 16 , Apr 2, 2009
                  • 0 Attachment
                    I'm starting getting this with the "unicode" branch:

                    SyntaxError: Non-ASCII character '\xc3' in file /home/jbq/Template.py
                    on line 255, but no encoding declared; see
                    http://www.python.org/peps/pep-0263.html for details (Template.py,
                    line 255)

                    I wonder why there is no encoding declared in the template. Setting
                    #encoding utf8 should make the compiler also declare encoding in the
                    generated module, don't you think?
                    --
                    Jean-Baptiste Quenot
                    http://jbq.caraldi.com/

                    ------------------------------------------------------------------------------
                    _______________________________________________
                    Cheetahtemplate-discuss mailing list
                    Cheetahtemplate-discuss@...
                    https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                  • Jean-Baptiste Quenot
                    Please find attached a unit test patch for this. I get the following result: .....Compiling /tmp/tmp6ErsxQ.tmpl - /tmp/tmp6ErsxQ.py E
                    Message 9 of 16 , Apr 2, 2009
                    • 0 Attachment
                      Please find attached a unit test patch for this.

                      I get the following result:

                      .....Compiling /tmp/tmp6ErsxQ.tmpl -> /tmp/tmp6ErsxQ.py
                      E
                      **********************************************************************
                      ERROR __main__.JBQ_UTF8_Test6.runTest ()
                      ----------------------------------------------------------------------
                      Traceback (most recent call last):
                      File "src/Tests/Unicode.py", line 107, in runTest
                      mod = loadModule(modname, ["/tmp"])
                      File "src/Tests/Unicode.py", line 74, in loadModule
                      mod = imp.load_module(moduleName, fp, pathname, description)
                      File "/tmp/tmp6ErsxQ.py", line 86
                      SyntaxError: Non-ASCII character '\xc3' in file /tmp/tmp6ErsxQ.py on
                      line 86, but no encoding declared; see
                      http://www.python.org/peps/pep-0263.html for details (tmp6ErsxQ.py,
                      line 86)

                      ----------------------------------------------------------------------
                      Ran 6 tests in 0.042s

                      FAILED (errors=1)
                      --
                      Jean-Baptiste Quenot
                      http://jbq.caraldi.com/
                    • R. Tyler Ballance
                      ... Wonderful, thanks for the patch and new test, I see I accidentally hit the P key instead of the B when I added them into there ;) Anyways, you re right,
                      Message 10 of 16 , Apr 2, 2009
                      • 0 Attachment
                        On Thu, Apr 02, 2009 at 12:33:10PM +0200, Jean-Baptiste Quenot wrote:
                        > Please find attached a unit test patch for this.

                        Wonderful, thanks for the patch and new test, I see I accidentally hit
                        the P key instead of the B when I added them into there ;)

                        Anyways, you're right, this is a regression that I introduced into the
                        unicode branch (towards the end of the 2f6f6c9 patch)

                        What basically was occuring is some interesting behavior IMHO in Python
                        with regards to "file encodings".

                        I prematurely return out of the setModuleEncoding() function in
                        Compiler.py because the internal representation of "source" is now a
                        unicode() object instead of a regular old str() object. When dealing with a
                        dynamically compiled template, the code is exec'd (Template.py:772) and
                        for some reason the call would fail to execute a unicode string that had
                        an "coding" directive in them. When writing the compiled template out to
                        a file it's obvious that we'd want to have the "coding" directive there
                        however.

                        I'm still on the fence on how I want to solve this (in a manner that's
                        not hackish), so suggestions/patches welcome ;)



                        Cheers



                        >
                        > I get the following result:
                        >
                        > .....Compiling /tmp/tmp6ErsxQ.tmpl -> /tmp/tmp6ErsxQ.py
                        > E
                        > **********************************************************************
                        > ERROR __main__.JBQ_UTF8_Test6.runTest ()
                        > ----------------------------------------------------------------------
                        > Traceback (most recent call last):
                        > File "src/Tests/Unicode.py", line 107, in runTest
                        > mod = loadModule(modname, ["/tmp"])
                        > File "src/Tests/Unicode.py", line 74, in loadModule
                        > mod = imp.load_module(moduleName, fp, pathname, description)
                        > File "/tmp/tmp6ErsxQ.py", line 86
                        > SyntaxError: Non-ASCII character '\xc3' in file /tmp/tmp6ErsxQ.py on
                        > line 86, but no encoding declared; see
                        > http://www.python.org/peps/pep-0263.html for details (tmp6ErsxQ.py,
                        > line 86)
                        >
                        > ----------------------------------------------------------------------
                        > Ran 6 tests in 0.042s
                        >
                        > FAILED (errors=1)
                        > --
                        > Jean-Baptiste Quenot
                        > http://jbq.caraldi.com/

                        > From 5a5993dfff1655b84dcd95d8c209d3f62ca33c54 Mon Sep 17 00:00:00 2001
                        > From: Jean-Baptiste Quenot <jbq@...>
                        > Date: Thu, 2 Apr 2009 12:28:07 +0200
                        > Subject: [PATCH] Add failing test for compiling a Unicode template
                        >
                        > ---
                        > src/Tests/Unicode.py | 52 ++++++++++++++++++++++++++++++++++++++++++++-----
                        > 1 files changed, 46 insertions(+), 6 deletions(-)
                        >
                        > diff --git a/src/Tests/Unicode.py b/src/Tests/Unicode.py
                        > index 2e9d719..93df928 100644
                        > --- a/src/Tests/Unicode.py
                        > +++ b/src/Tests/Unicode.py
                        > @@ -2,10 +2,11 @@
                        > # -*- encoding: utf8 -*-
                        >
                        > from Cheetah.Template import Template
                        > -import traceback
                        > +from Cheetah import CheetahWrapper
                        > +import traceback, tempfile, sys, imp, os
                        > import unittest_local_copy as unittest # This is stupid
                        >
                        > -class JPQ_UTF8_Test1(unittest.TestCase):
                        > +class JBQ_UTF8_Test1(unittest.TestCase):
                        > def runTest(self):
                        > t = Template.compile(source="""Main file with |$v|
                        >
                        > @@ -20,7 +21,7 @@ class JPQ_UTF8_Test1(unittest.TestCase):
                        >
                        > assert t()
                        >
                        > -class JPQ_UTF8_Test2(unittest.TestCase):
                        > +class JBQ_UTF8_Test2(unittest.TestCase):
                        > def runTest(self):
                        > t = Template.compile(source="""Main file with |$v|
                        >
                        > @@ -36,7 +37,7 @@ class JPQ_UTF8_Test2(unittest.TestCase):
                        > assert t()
                        >
                        >
                        > -class JPQ_UTF8_Test3(unittest.TestCase):
                        > +class JBQ_UTF8_Test3(unittest.TestCase):
                        > def runTest(self):
                        > t = Template.compile(source="""Main file with |$v|
                        >
                        > @@ -51,7 +52,7 @@ class JPQ_UTF8_Test3(unittest.TestCase):
                        >
                        > assert t()
                        >
                        > -class JPQ_UTF8_Test4(unittest.TestCase):
                        > +class JBQ_UTF8_Test4(unittest.TestCase):
                        > def runTest(self):
                        > t = Template.compile(source="""#encoding utf-8
                        > Main file with |$v| and eacute in the template ??""")
                        > @@ -60,7 +61,23 @@ class JPQ_UTF8_Test4(unittest.TestCase):
                        >
                        > assert t()
                        >
                        > -class JPQ_UTF8_Test5(unittest.TestCase):
                        > +def loadModule(moduleName, path=None):
                        > + if path:
                        > + assert isinstance(path, list)
                        > + try:
                        > + mod = sys.modules[moduleName]
                        > + except KeyError:
                        > + fp = None
                        > +
                        > + try:
                        > + fp, pathname, description = imp.find_module(moduleName, path)
                        > + mod = imp.load_module(moduleName, fp, pathname, description)
                        > + finally:
                        > + if fp:
                        > + fp.close()
                        > + return mod
                        > +
                        > +class JBQ_UTF8_Test5(unittest.TestCase):
                        > def runTest(self):
                        > t = Template.compile(source="""#encoding utf-8
                        > Main file with |$v| and eacute in the template ??""")
                        > @@ -69,5 +86,28 @@ class JPQ_UTF8_Test5(unittest.TestCase):
                        >
                        > assert t()
                        >
                        > +class JBQ_UTF8_Test6(unittest.TestCase):
                        > + def runTest(self):
                        > + source = """#encoding utf-8
                        > + #set $someUnicodeString = u"B??b??"
                        > + Main file with |$v| and eacute in the template ??"""
                        > + t = Template.compile(source=source)
                        > +
                        > + t.v = u'Unicode String'
                        > +
                        > + assert t()
                        > +
                        > + sourcefile = tempfile.mktemp()
                        > + f = open("%s.tmpl" % sourcefile, "w")
                        > + f.write(source)
                        > + f.close()
                        > + cw = CheetahWrapper.CheetahWrapper()
                        > + cw.main(["cheetah", "compile", "--nobackup", sourcefile])
                        > + modname = os.path.basename(sourcefile)
                        > + mod = loadModule(modname, ["/tmp"])
                        > + t = eval("mod.%s()" % modname)
                        > + t.v = u'Unicode String'
                        > + print unicode(t)
                        > +
                        > if __name__ == '__main__':
                        > unittest.main()
                        > --
                        > 1.6.0.4
                        >

                        > ------------------------------------------------------------------------------

                        > _______________________________________________
                        > Cheetahtemplate-discuss mailing list
                        > Cheetahtemplate-discuss@...
                        > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


                        --
                        -R. Tyler Ballance
                        Slide, Inc.
                      • Jean-Baptiste Quenot
                        ... Actually I don t think this patch should have ended in the next branch, but only in the unicode branch. I switched back to the next branch and added
                        Message 11 of 16 , Apr 3, 2009
                        • 0 Attachment
                          2009/4/2 R. Tyler Ballance <tyler@...>:

                          > Aha, good to know, I'll cherry-pick it over

                          Actually I don't think this patch should have ended in the 'next'
                          branch, but only in the 'unicode' branch.

                          I switched back to the 'next' branch and added the last unit test of
                          the Unicode branch, which is actually failing:

                          ERROR __main__.JBQ_UTF8_Test7.runTest ()
                          ----------------------------------------------------------------------
                          Traceback (most recent call last):
                          File "src/Tests/Unicode.py", line 111, in runTest
                          cw.main(["cheetah", "compile", "--nobackup", sourcefile])
                          File "/home/jbq/nomao/packages/cheetah/Cheetah/CheetahWrapper.py",
                          line 149, in main
                          meth()
                          File "/home/jbq/nomao/packages/cheetah/Cheetah/CheetahWrapper.py",
                          line 208, in compile
                          self._compileOrFill()
                          File "/home/jbq/nomao/packages/cheetah/Cheetah/CheetahWrapper.py",
                          line 355, in _compileOrFill
                          self._compileOrFillBundle(b)
                          File "/home/jbq/nomao/packages/cheetah/Cheetah/CheetahWrapper.py",
                          line 590, in _compileOrFillBundle
                          f.write(output.encode('utf8'))
                          UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
                          2835: ordinal not in range(128)

                          ----------------------------------------------------------------------

                          See attached patch for unit tests update.

                          To fix test #7 I just issued a git revert
                          158b56c8858f35669ee997ffd08a9e56aff3d75e

                          Well I don't really mean to fix test #7 entirely, because anyway
                          Unicode support is broken in 'next'. I mean to fix test #7 when
                          sitecustomize.py sets default encoding to UTF-8.
                          --
                          Jean-Baptiste Quenot
                          http://jbq.caraldi.com/
                        • Jean-Baptiste Quenot
                          ... If the issue we re facing is this one: http://bugs.python.org/issue4626 http://svn.python.org/view?view=rev&revision=70113 Then we ll have to wait for the
                          Message 12 of 16 , Apr 3, 2009
                          • 0 Attachment
                            2009/4/3 R. Tyler Ballance <tyler@...>:
                            > On Thu, Apr 02, 2009 at 12:33:10PM +0200, Jean-Baptiste Quenot wrote:
                            >
                            > What basically was occuring is some interesting behavior IMHO in Python
                            > with regards to "file encodings".
                            >
                            > I prematurely return out of the setModuleEncoding() function in
                            > Compiler.py because the internal representation of "source" is now a
                            > unicode() object instead of a regular old str() object. When dealing with a
                            > dynamically compiled template, the code is exec'd (Template.py:772) and
                            > for some reason the call would fail to execute a unicode string that had
                            > an "coding" directive in them. When writing the compiled template out to
                            > a file it's obvious that we'd want to have the "coding" directive there
                            > however.
                            >
                            > I'm still on the fence on how I want to solve this (in a manner that's
                            > not hackish), so suggestions/patches welcome ;)

                            If the issue we're facing is this one:

                            http://bugs.python.org/issue4626
                            http://svn.python.org/view?view=rev&revision=70113

                            Then we'll have to wait for the next Python 3.0.x for an official fix.

                            Or we can work around this Python bug by re-encoding the Unicode
                            string to a char buffer. See attached patch. With this patch, the
                            unit tests pass for me now.
                            --
                            Jean-Baptiste Quenot
                            http://jbq.caraldi.com/
                          • R. Tyler Ballance
                            If you wouldn t mind jumping onto #cheetah on Freenode today, I d like to discuss this a bit more with you. I m at an impasse as to which changes to roll into
                            Message 13 of 16 , Apr 3, 2009
                            • 0 Attachment
                              If you wouldn't mind jumping onto #cheetah on Freenode today, I'd like
                              to discuss this a bit more with you.

                              I'm at an impasse as to which changes to roll into next and which to
                              keep in unicode for a v2.2 release

                              On Fri, Apr 03, 2009 at 04:49:57PM +0200, Jean-Baptiste Quenot wrote:
                              > 2009/4/3 R. Tyler Ballance <tyler@...>:
                              > > On Thu, Apr 02, 2009 at 12:33:10PM +0200, Jean-Baptiste Quenot wrote:
                              > >
                              > > What basically was occuring is some interesting behavior IMHO in Python
                              > > with regards to "file encodings".
                              > >
                              > > I prematurely return out of the setModuleEncoding() function in
                              > > Compiler.py because the internal representation of "source" is now a
                              > > unicode() object instead of a regular old str() object. When dealing with a
                              > > dynamically compiled template, the code is exec'd (Template.py:772) and
                              > > for some reason the call would fail to execute a unicode string that had
                              > > an "coding" directive in them. When writing the compiled template out to
                              > > a file it's obvious that we'd want to have the "coding" directive there
                              > > however.
                              > >
                              > > I'm still on the fence on how I want to solve this (in a manner that's
                              > > not hackish), so suggestions/patches welcome ;)
                              >
                              > If the issue we're facing is this one:
                              >
                              > http://bugs.python.org/issue4626
                              > http://svn.python.org/view?view=rev&revision=70113
                              >
                              > Then we'll have to wait for the next Python 3.0.x for an official fix.
                              >
                              > Or we can work around this Python bug by re-encoding the Unicode
                              > string to a char buffer. See attached patch. With this patch, the
                              > unit tests pass for me now.
                              > --
                              > Jean-Baptiste Quenot
                              > http://jbq.caraldi.com/

                              > From 583423d470e538133d604edec8c444faead25d50 Mon Sep 17 00:00:00 2001
                              > From: Jean-Baptiste Quenot <jbq@...>
                              > Date: Fri, 3 Apr 2009 16:46:05 +0200
                              > Subject: [PATCH] Work around Python bug #4626 regarding compile() and exec() with coding cookie
                              >
                              > ---
                              > src/Compiler.py | 2 --
                              > src/Template.py | 2 +-
                              > 2 files changed, 1 insertions(+), 3 deletions(-)
                              >
                              > diff --git a/src/Compiler.py b/src/Compiler.py
                              > index 5d3968c..8377478 100644
                              > --- a/src/Compiler.py
                              > +++ b/src/Compiler.py
                              > @@ -1813,8 +1813,6 @@ class ModuleCompiler(SettingsManager, GenUtils):
                              > self._moduleShBang = shBang
                              >
                              > def setModuleEncoding(self, encoding):
                              > - #print ('setModuleEncoding', locals())
                              > - return
                              > self._moduleEncoding = encoding
                              > self._moduleEncodingStr = '# -*- coding: %s -*-' %encoding
                              >
                              > diff --git a/src/Template.py b/src/Template.py
                              > index a4d07e0..a2bf0b0 100644
                              > --- a/src/Template.py
                              > +++ b/src/Template.py
                              > @@ -768,7 +768,7 @@ class Template(Servlet):
                              > setattr(mod, baseclassName, baseclassValue)
                              > ##
                              > try:
                              > - co = compile(generatedModuleCode, __file__, 'exec')
                              > + co = compile(generatedModuleCode.encode('utf8'), __file__, 'exec')
                              > exec co in mod.__dict__
                              > except SyntaxError, e:
                              > try:
                              > --
                              > 1.6.0.4
                              >

                              > ------------------------------------------------------------------------------

                              > _______________________________________________
                              > Cheetahtemplate-discuss mailing list
                              > Cheetahtemplate-discuss@...
                              > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


                              --
                              -R. Tyler Ballance
                              Slide, Inc.
                            • R. Tyler Ballance
                              ... FWIW this has been pushed up to the unicode branch and is ready for testing :) ... -- -R. Tyler Ballance Slide, Inc. ...
                              Message 14 of 16 , Apr 3, 2009
                              • 0 Attachment
                                On Fri, Apr 03, 2009 at 02:47:45PM -0700, R. Tyler Ballance wrote:
                                > This should resolve the issue reported by jbq@ reflected in src/Tests/Unicode,py:JBQ_UTF8_Test7

                                FWIW this has been pushed up to the unicode branch and is ready for
                                testing :)


                                >
                                > With this commit the #encoding directive will do what you might expect it to do in terms of encoding
                                > the resulting .py file (adding in the "-*- coding: " directive) or encoding the unicode string when
                                > executing a dynamic template
                                >
                                > Signed-off-by: R. Tyler Ballance <tyler@...>
                                > ---
                                > src/CheetahWrapper.py | 2 +-
                                > src/Compiler.py | 3 ---
                                > src/Template.py | 12 ++++++++++--
                                > 3 files changed, 11 insertions(+), 6 deletions(-)
                                >
                                > diff --git a/src/CheetahWrapper.py b/src/CheetahWrapper.py
                                > index 776a394..5a37f5e 100644
                                > --- a/src/CheetahWrapper.py
                                > +++ b/src/CheetahWrapper.py
                                > @@ -587,7 +587,7 @@ be named according to the same rules as Python modules.""" % tup)
                                > sys.stdout.write(output)
                                > else:
                                > f = open(dst, 'w')
                                > - f.write(output.encode('utf8'))
                                > + f.write(output)
                                > f.close()
                                >
                                >
                                > diff --git a/src/Compiler.py b/src/Compiler.py
                                > index 5d3968c..34bc994 100644
                                > --- a/src/Compiler.py
                                > +++ b/src/Compiler.py
                                > @@ -1813,10 +1813,7 @@ class ModuleCompiler(SettingsManager, GenUtils):
                                > self._moduleShBang = shBang
                                >
                                > def setModuleEncoding(self, encoding):
                                > - #print ('setModuleEncoding', locals())
                                > - return
                                > self._moduleEncoding = encoding
                                > - self._moduleEncodingStr = '# -*- coding: %s -*-' %encoding
                                >
                                > def getModuleEncoding(self):
                                > return self._moduleEncoding
                                > diff --git a/src/Template.py b/src/Template.py
                                > index a4d07e0..4958e9f 100644
                                > --- a/src/Template.py
                                > +++ b/src/Template.py
                                > @@ -719,6 +719,7 @@ class Template(Servlet):
                                > except:
                                > #@@TR: should add some logging to this
                                > pass
                                > + outputEncoding = 'ascii'
                                > if useCache and cacheHash and cacheHash in klass._CHEETAH_compileCache:
                                > cacheItem = klass._CHEETAH_compileCache[cacheHash]
                                > generatedModuleCode = cacheItem.code
                                > @@ -733,9 +734,16 @@ class Template(Servlet):
                                > compiler.setShBang(commandlineopts.shbang)
                                > compiler.compile()
                                > generatedModuleCode = compiler.getModuleCode()
                                > + outputEncoding = compiler.getModuleEncoding()
                                >
                                > if not returnAClass:
                                > - return generatedModuleCode
                                > + # This is a bit of a hackish solution to make sure we're setting the proper
                                > + # encoding on generated code that is destined to be written to a file
                                > + if not outputEncoding == 'ascii':
                                > + generatedModuleCode = generatedModuleCode.split('\n')
                                > + generatedModuleCode.insert(1, '# -*- coding: %s -*-' % outputEncoding)
                                > + generatedModuleCode = '\n'.join(generatedModuleCode)
                                > + return generatedModuleCode.encode(outputEncoding)
                                > else:
                                > if cacheItem:
                                > cacheItem.lastCheckoutTime = time.time()
                                > @@ -768,7 +776,7 @@ class Template(Servlet):
                                > setattr(mod, baseclassName, baseclassValue)
                                > ##
                                > try:
                                > - co = compile(generatedModuleCode, __file__, 'exec')
                                > + co = compile(generatedModuleCode.encode(outputEncoding), __file__, 'exec')
                                > exec co in mod.__dict__
                                > except SyntaxError, e:
                                > try:
                                > --
                                >
                                >

                                --
                                -R. Tyler Ballance
                                Slide, Inc.
                              • R. Tyler Ballance
                                This should resolve the issue reported by jbq@ reflected in src/Tests/Unicode,py:JBQ_UTF8_Test7 With this commit the #encoding directive will do what you might
                                Message 15 of 16 , Apr 3, 2009
                                • 0 Attachment
                                  This should resolve the issue reported by jbq@ reflected in src/Tests/Unicode,py:JBQ_UTF8_Test7

                                  With this commit the #encoding directive will do what you might expect it to do in terms of encoding
                                  the resulting .py file (adding in the "-*- coding: " directive) or encoding the unicode string when
                                  executing a dynamic template

                                  Signed-off-by: R. Tyler Ballance <tyler@...>
                                  ---
                                  src/CheetahWrapper.py | 2 +-
                                  src/Compiler.py | 3 ---
                                  src/Template.py | 12 ++++++++++--
                                  3 files changed, 11 insertions(+), 6 deletions(-)

                                  diff --git a/src/CheetahWrapper.py b/src/CheetahWrapper.py
                                  index 776a394..5a37f5e 100644
                                  --- a/src/CheetahWrapper.py
                                  +++ b/src/CheetahWrapper.py
                                  @@ -587,7 +587,7 @@ be named according to the same rules as Python modules.""" % tup)
                                  sys.stdout.write(output)
                                  else:
                                  f = open(dst, 'w')
                                  - f.write(output.encode('utf8'))
                                  + f.write(output)
                                  f.close()


                                  diff --git a/src/Compiler.py b/src/Compiler.py
                                  index 5d3968c..34bc994 100644
                                  --- a/src/Compiler.py
                                  +++ b/src/Compiler.py
                                  @@ -1813,10 +1813,7 @@ class ModuleCompiler(SettingsManager, GenUtils):
                                  self._moduleShBang = shBang

                                  def setModuleEncoding(self, encoding):
                                  - #print ('setModuleEncoding', locals())
                                  - return
                                  self._moduleEncoding = encoding
                                  - self._moduleEncodingStr = '# -*- coding: %s -*-' %encoding

                                  def getModuleEncoding(self):
                                  return self._moduleEncoding
                                  diff --git a/src/Template.py b/src/Template.py
                                  index a4d07e0..4958e9f 100644
                                  --- a/src/Template.py
                                  +++ b/src/Template.py
                                  @@ -719,6 +719,7 @@ class Template(Servlet):
                                  except:
                                  #@@TR: should add some logging to this
                                  pass
                                  + outputEncoding = 'ascii'
                                  if useCache and cacheHash and cacheHash in klass._CHEETAH_compileCache:
                                  cacheItem = klass._CHEETAH_compileCache[cacheHash]
                                  generatedModuleCode = cacheItem.code
                                  @@ -733,9 +734,16 @@ class Template(Servlet):
                                  compiler.setShBang(commandlineopts.shbang)
                                  compiler.compile()
                                  generatedModuleCode = compiler.getModuleCode()
                                  + outputEncoding = compiler.getModuleEncoding()

                                  if not returnAClass:
                                  - return generatedModuleCode
                                  + # This is a bit of a hackish solution to make sure we're setting the proper
                                  + # encoding on generated code that is destined to be written to a file
                                  + if not outputEncoding == 'ascii':
                                  + generatedModuleCode = generatedModuleCode.split('\n')
                                  + generatedModuleCode.insert(1, '# -*- coding: %s -*-' % outputEncoding)
                                  + generatedModuleCode = '\n'.join(generatedModuleCode)
                                  + return generatedModuleCode.encode(outputEncoding)
                                  else:
                                  if cacheItem:
                                  cacheItem.lastCheckoutTime = time.time()
                                  @@ -768,7 +776,7 @@ class Template(Servlet):
                                  setattr(mod, baseclassName, baseclassValue)
                                  ##
                                  try:
                                  - co = compile(generatedModuleCode, __file__, 'exec')
                                  + co = compile(generatedModuleCode.encode(outputEncoding), __file__, 'exec')
                                  exec co in mod.__dict__
                                  except SyntaxError, e:
                                  try:
                                  --



                                  ------------------------------------------------------------------------------
                                  _______________________________________________
                                  Cheetahtemplate-discuss mailing list
                                  Cheetahtemplate-discuss@...
                                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                • R. Tyler Ballance
                                  ... It s also worth mentioning right off the bat that there are currently a number of failing tests, some failing because they need to be upgraded, some
                                  Message 16 of 16 , Apr 3, 2009
                                  • 0 Attachment
                                    On Fri, Apr 03, 2009 at 02:33:24PM -0700, R. Tyler Ballance wrote:
                                    > On Fri, Apr 03, 2009 at 02:47:45PM -0700, R. Tyler Ballance wrote:
                                    > > This should resolve the issue reported by jbq@ reflected in src/Tests/Unicode,py:JBQ_UTF8_Test7
                                    >
                                    > FWIW this has been pushed up to the unicode branch and is ready for
                                    > testing :)

                                    It's also worth mentioning right off the bat that there are currently a
                                    number of failing tests, some failing because they need to be upgraded,
                                    some failing due to other issues.

                                    The full test run output is here: http://gist.github.com/89983


                                    >
                                    >
                                    > >
                                    > > With this commit the #encoding directive will do what you might expect it to do in terms of encoding
                                    > > the resulting .py file (adding in the "-*- coding: " directive) or encoding the unicode string when
                                    > > executing a dynamic template
                                    > >
                                    > > Signed-off-by: R. Tyler Ballance <tyler@...>
                                    > > ---
                                    > > src/CheetahWrapper.py | 2 +-
                                    > > src/Compiler.py | 3 ---
                                    > > src/Template.py | 12 ++++++++++--
                                    > > 3 files changed, 11 insertions(+), 6 deletions(-)
                                    > >
                                    > > diff --git a/src/CheetahWrapper.py b/src/CheetahWrapper.py
                                    > > index 776a394..5a37f5e 100644
                                    > > --- a/src/CheetahWrapper.py
                                    > > +++ b/src/CheetahWrapper.py
                                    > > @@ -587,7 +587,7 @@ be named according to the same rules as Python modules.""" % tup)
                                    > > sys.stdout.write(output)
                                    > > else:
                                    > > f = open(dst, 'w')
                                    > > - f.write(output.encode('utf8'))
                                    > > + f.write(output)
                                    > > f.close()
                                    > >
                                    > >
                                    > > diff --git a/src/Compiler.py b/src/Compiler.py
                                    > > index 5d3968c..34bc994 100644
                                    > > --- a/src/Compiler.py
                                    > > +++ b/src/Compiler.py
                                    > > @@ -1813,10 +1813,7 @@ class ModuleCompiler(SettingsManager, GenUtils):
                                    > > self._moduleShBang = shBang
                                    > >
                                    > > def setModuleEncoding(self, encoding):
                                    > > - #print ('setModuleEncoding', locals())
                                    > > - return
                                    > > self._moduleEncoding = encoding
                                    > > - self._moduleEncodingStr = '# -*- coding: %s -*-' %encoding
                                    > >
                                    > > def getModuleEncoding(self):
                                    > > return self._moduleEncoding
                                    > > diff --git a/src/Template.py b/src/Template.py
                                    > > index a4d07e0..4958e9f 100644
                                    > > --- a/src/Template.py
                                    > > +++ b/src/Template.py
                                    > > @@ -719,6 +719,7 @@ class Template(Servlet):
                                    > > except:
                                    > > #@@TR: should add some logging to this
                                    > > pass
                                    > > + outputEncoding = 'ascii'
                                    > > if useCache and cacheHash and cacheHash in klass._CHEETAH_compileCache:
                                    > > cacheItem = klass._CHEETAH_compileCache[cacheHash]
                                    > > generatedModuleCode = cacheItem.code
                                    > > @@ -733,9 +734,16 @@ class Template(Servlet):
                                    > > compiler.setShBang(commandlineopts.shbang)
                                    > > compiler.compile()
                                    > > generatedModuleCode = compiler.getModuleCode()
                                    > > + outputEncoding = compiler.getModuleEncoding()
                                    > >
                                    > > if not returnAClass:
                                    > > - return generatedModuleCode
                                    > > + # This is a bit of a hackish solution to make sure we're setting the proper
                                    > > + # encoding on generated code that is destined to be written to a file
                                    > > + if not outputEncoding == 'ascii':
                                    > > + generatedModuleCode = generatedModuleCode.split('\n')
                                    > > + generatedModuleCode.insert(1, '# -*- coding: %s -*-' % outputEncoding)
                                    > > + generatedModuleCode = '\n'.join(generatedModuleCode)
                                    > > + return generatedModuleCode.encode(outputEncoding)
                                    > > else:
                                    > > if cacheItem:
                                    > > cacheItem.lastCheckoutTime = time.time()
                                    > > @@ -768,7 +776,7 @@ class Template(Servlet):
                                    > > setattr(mod, baseclassName, baseclassValue)
                                    > > ##
                                    > > try:
                                    > > - co = compile(generatedModuleCode, __file__, 'exec')
                                    > > + co = compile(generatedModuleCode.encode(outputEncoding), __file__, 'exec')
                                    > > exec co in mod.__dict__
                                    > > except SyntaxError, e:
                                    > > try:
                                    > > --
                                    > >
                                    > >
                                    >
                                    > --
                                    > -R. Tyler Ballance
                                    > Slide, Inc.



                                    > ------------------------------------------------------------------------------

                                    > _______________________________________________
                                    > Cheetahtemplate-discuss mailing list
                                    > Cheetahtemplate-discuss@...
                                    > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


                                    --
                                    -R. Tyler Ballance
                                    Slide, Inc.
                                  Your message has been successfully submitted and would be delivered to recipients shortly.