Loading ...
Sorry, an error occurred while loading the content.
 

Re: [Cheetahtemplate-discuss] unicode troubles

Expand Messages
  • Charlie knudsen
    ... t = Template(file= input.txt ) out = t.respond() if type(out) != type(u ): out = out.decode( utf-8 ) print out.encode( utf-8 ) ... While this solves the
    Message 1 of 30 , Jan 7, 2007
      I may have made some progress on my issue.  The below code is able to process the file:
      -----------------------
      t = Template(file="input.txt")
      out = t.respond()
      if type(out) != type(u''):
          out = out.decode('utf-8')
      print out.encode('utf-8')
      ---------------------------
      While this solves the simple example I outlined, I have not yet plugged this fix or variation of this fix into the actual my actual project.  So please let me know if you could let me know if I am doing anything wrong, or why the decode function is needed I would appreciate it.

      Thanks,

      Charlie

      ----- Original Message ----
      From: Miha Vrhovnik <mvrhov@...>
      To: cheetahtemplate-discuss@...
      Sent: Sunday, January 7, 2007 4:20:54 AM
      Subject: Re: [Cheetahtemplate-discuss] unicode troubles

      si.Mail composed message

      To list admin, please deny my previous message, I used wrong e-mail to post it.


      I brought that up about a year ago!

      And still nobody come out with a solution.


       I have template encoded in utf-8 and then I "fill" it with unicode strings, and I would like to get back unicode.

       This fails because TEMPLATE is NOT converted to when reading.


       As I said before template looks like this:


       <?xml version="1.0" encoding="utf-8"?>

       http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

       

       <!--#*

         DO NOT CHANGE NEXT FEW LINES UNLES YOU KNOW WHAT ARE YOU DOING !!!!!

       *#-->

       <!-- #*set output to properly encode all html special characters *#-->

       #filter WebSafe

       <!--#* set template engine to utf-8 *#-->

       #encoding UTF-8

       <head>

       .......

       </body>

       </html>

       #end filter


       Regards,

       Miha


       "Charlie knudsen" <charlie_knudsen@...> wrote on 7.1.2007 10:56:04:

       >Hi All,

       >I am new to Cheetah, am using version 2.0rc7, and am running into unicode troubles.  I am using windows XP.  I am trying to create a Template and then get the content back into a unicode string.  I have tried the below code, I also posted the code and input file at the below link if you want run/view it:

       >http://cknudsen.net/examples/unicode_test.zip

       >

       >--------------------------------------------------

       ># This fails with a UnicodeDecodeError

       ># The type returned by respond is not unicode, is this correct?

       >t = Template(file="input.txt")

       >out = t.respond()

       >print type(out)  # not unicode

       >print out.encode('utf-8') # not able to encode the string

       >----------ERROR----------------------------------------

       >ERROR:

       >Traceback (most recent call last):

       >  File "C:\Documents and Settings\CharlieK\Desktop\unicode_test\test.py", line 18, in ?

       >    print out.encode('utf-8') # not able to encode the string

       >UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

       >-------------------------------------------------------

       >

       ># This fails with a UnicodeDecodeError

       >t = Template(file="input.txt")

       >out = unicode(t)  # fails here

       >print type(out)

       >print out.encode('utf-8')

       >---------ERROR-------------------------------------

       >Traceback (most recent call last):

       >  File "C:\Documents and Settings\CharlieK\Desktop\unicode_test\test.py", line 15, in ?

       >    out = unicode(t)  # fails here

       >UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

       >----------------------------------------------------------

       >

       >------INPUT.TXT----------------------------------

       >#encoding UTF-8

       >Indica que el contenido no está traducido o que está traducido parcialmente.

       >Für février Interopérabilité d'activité d'activité

       >-----------------------------------------

       >

       >I am able to read the file in using codecs.open(...), so I don't think it is the actual input text.

       >

       >If there is any help you could give me, or unicode examples I could look at it would be appreciated.

       >

       >Thanks for your help,

       >

       >Charlie

       >

       >

       >

       >

       >

       >-------------------------------------------------------------------------

       >Take Surveys. Earn Cash. Influence the Future of IT

       >Join SourceForge.net's Techsay panel and you'll get the chance to share your

       >opinions on IT & business topics through brief surveys - and earn cash

       >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

       >_______________________________________________

       >Cheetahtemplate-discuss mailing list

       >Cheetahtemplate-discuss@...

       >https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

       >



       --  

      It's time to get rid of your current e-mail client ... 

      ... and start using si.Mail. 


      It's small & free. ( http://simail.sourceforge.net/ )

      -------------------------------------------------------------------------
      Take Surveys. Earn Cash. Influence the Future of IT
      Join SourceForge.net's Techsay panel and you'll get the chance to share your
      opinions on IT & business topics through brief surveys - and earn cash
      http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
      _______________________________________________
      Cheetahtemplate-discuss mailing list
      Cheetahtemplate-discuss@...
      https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

    • Mike Orr
      ... Yes, this is reasonable. str(t) and unicode(t) break if the string contains non-ASCII characters due to bugs in Python, so use t.respond() instead, which
      Message 2 of 30 , Jan 7, 2007
        On 1/7/07, Charlie knudsen <charlie_knudsen@...> wrote:
        >
        > I may have made some progress on my issue. The below code is able to
        > process the file:
        > -----------------------
        > t = Template(file="input.txt")
        > out = t.respond()
        > if type(out) != type(u''):
        > out = out.decode('utf-8')
        > print out.encode('utf-8')
        > ---------------------------
        > While this solves the simple example I outlined, I have not yet plugged this
        > fix or variation of this fix into the actual my actual project. So please
        > let me know if you could let me know if I am doing anything wrong, or why
        > the decode function is needed I would appreciate it.

        Yes, this is reasonable. str(t) and unicode(t) break if the string
        contains non-ASCII characters due to bugs in Python, so use
        t.respond() instead, which preserves the string-or-unicode status. I
        use precompiled ASCII templates with Unicode placeholder values and
        get Unicode back fine. Perhaps you've discovered a bug in how
        Template(file=) operates re Unicode. Is there a reason you can't use
        precompiled templates?

        The WebSafe filter is not safe for non-ASCII characters because it
        calls str() on the placeholder value. It needs to be based on
        RawOrEncodedUnicode. I'm trying to clear out a couple other Python
        projects so I can fix up the Filters module and a few other things
        that need done in Cheetah.

        Sorry for not replying sooner. I was sick the past few days so I just
        read your message quickly, and didn't realize the problem was so
        similar to one I had had.

        --
        Mike Orr <sluggoster@...>

        -------------------------------------------------------------------------
        Take Surveys. Earn Cash. Influence the Future of IT
        Join SourceForge.net's Techsay panel and you'll get the chance to share your
        opinions on IT & business topics through brief surveys - and earn cash
        http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
        _______________________________________________
        Cheetahtemplate-discuss mailing list
        Cheetahtemplate-discuss@...
        https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
      • Miha Vrhovnik
        ... In my case I cannot use precompiled templates. But still, cheetah should behave like python and convert input template in unicode. ... While at it please
        Message 3 of 30 , Jan 8, 2007
          >Yes, this is reasonable. str(t) and unicode(t) break if the string
          >contains non-ASCII characters due to bugs in Python, so use
          >t.respond() instead, which preserves the string-or-unicode status. I
          >use precompiled ASCII templates with Unicode placeholder values and
          >get Unicode back fine. Perhaps you've discovered a bug in how
          >Template(file=) operates re Unicode. Is there a reason you can't use
          >precompiled templates?
          In my case I cannot use precompiled templates.
          But still, cheetah should behave like python and convert input template in unicode.

          >The WebSafe filter is not safe for non-ASCII characters because it
          >calls str() on the placeholder value. It needs to be based on
          >RawOrEncodedUnicode. I'm trying to clear out a couple other Python
          >projects so I can fix up the Filters module and a few other things
          >that need done in Cheetah.
          While at it please add:
          s = s.replace("\n", "<br />")
          into Websafe filter.

          >Sorry for not replying sooner. I was sick the past few days so I just
          >read your message quickly, and didn't realize the problem was so
          >similar to one I had had.
          >

          Regards,
          Miha

          --
          It's time to get rid of your current e-mail client ...
          ... and start using si.Mail.

          It's small & free. ( http://simail.sourceforge.net/ )

          -------------------------------------------------------------------------
          Take Surveys. Earn Cash. Influence the Future of IT
          Join SourceForge.net's Techsay panel and you'll get the chance to share your
          opinions on IT & business topics through brief surveys - and earn cash
          http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
          _______________________________________________
          Cheetahtemplate-discuss mailing list
          Cheetahtemplate-discuss@...
          https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
        • Brian Bird
          I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify the encoding (perhaps there should be
          Message 4 of 30 , Jan 8, 2007
            si.Mail composed message

            I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah will use codecs.open instead?)

             

            At first glance, the problem seems to be in Compiler.py (L1536 in 2.0rc6)

                elif isinstance(file, types.StringType) or isinstance(file, types.UnicodeType): # it's a filename.

                        f = open(file) # Raises IOError.

                        source = f.read()

                        f.close()

             

            If Compiler.py could somehow know the encoding it could use codecs.open instead of f.read()?

             

            The workaround I use is I manually read in the file myself and then pass the Unicode string to Cheetah as a source.

            This avoids Compiler.py doing f.read() :-

             

            s = codecs.open(“input.txt”, “r”, “utf-8”)

            t = Template(source=s)

            out = t.respond()

            print type(out)  # This time it IS Unicode because the input source is unicode

            print out.encode('utf-8') # Shouldn’t have any problems as long as the terminal can print the characters

             

             

            Brian

             

             


            From: cheetahtemplate-discuss-bounces@... [mailto:cheetahtemplate-discuss-bounces@...] On Behalf Of Charlie knudsen
            Sent: 08 January 2007 00:41
            To: cheetahtemplate-discuss@...
            Subject: Re: [Cheetahtemplate-discuss] unicode troubles

             

            I may have made some progress on my issue.  The below code is able to process the file:
            -----------------------
            t = Template(file="input.txt")
            out = t.respond()
            if type(out) != type(u''):
                out = out.decode('utf-8')
            print out.encode('utf-8')
            ---------------------------
            While this solves the simple example I outlined, I have not yet plugged this fix or variation of this fix into the actual my actual project.  So please let me know if you could let me know if I am doing anything wrong, or why the decode function is needed I would appreciate it.

            Thanks,

            Charlie

            ----- Original Message ----
            From: Miha Vrhovnik <mvrhov@...>
            To: cheetahtemplate-discuss@...
            Sent: Sunday, January 7, 2007 4:20:54 AM
            Subject: Re: [Cheetahtemplate-discuss] unicode troubles

            To list admin, please deny my previous message, I used wrong e-mail to post it.

             

            I brought that up about a year ago!

            And still nobody come out with a solution.

             

             I have template encoded in utf-8 and then I "fill" it with unicode strings, and I would like to get back unicode.

             This fails because TEMPLATE is NOT converted to when reading.

             

             As I said before template looks like this:

             

             <?xml version="1.0" encoding="utf-8"?>

             http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

             

             <!--#*

               DO NOT CHANGE NEXT FEW LINES UNLES YOU KNOW WHAT ARE YOU DOING !!!!!

             *#-->

             <!-- #*set output to properly encode all html special characters *#-->

             #filter WebSafe

             <!--#* set template engine to utf-8 *#-->

             #encoding UTF-8

             <head>

             .......

             </body>

             </html>

             #end filter

             

             Regards,

             Miha

             

             "Charlie knudsen" <charlie_knudsen@...> wrote on 7.1.2007 10:56:04:

             >Hi All,

             >I am new to Cheetah, am using version 2.0rc7, and am running into unicode troubles.  I am using windows XP.  I am trying to create a Template and then get the content back into a unicode string.  I have tried the below code, I also posted the code and input file at the below link if you want run/view it:

             >http://cknudsen.net/examples/unicode_test.zip

             >

             >--------------------------------------------------

             ># This fails with a UnicodeDecodeError

             ># The type returned by respond is not unicode, is this correct?

             >t = Template(file="input.txt")

             >out = t.respond()

             >print type(out)  # not unicode

             >print out.encode('utf-8') # not able to encode the string

             >----------ERROR----------------------------------------

             >ERROR:

             >Traceback (most recent call last):

             >  File "C:\Documents and Settings\CharlieK\Desktop\unicode_test\test.py", line 18, in ?

             >    print out.encode('utf-8') # not able to encode the string

             >UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

             >-------------------------------------------------------

             >

             ># This fails with a UnicodeDecodeError

             >t = Template(file="input.txt")

             >out = unicode(t)  # fails here

             >print type(out)

             >print out.encode('utf-8')

             >---------ERROR-------------------------------------

             >Traceback (most recent call last):

             >  File "C:\Documents and Settings\CharlieK\Desktop\unicode_test\test.py", line 15, in ?

             >    out = unicode(t)  # fails here

             >UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

             >----------------------------------------------------------

             >

             >------INPUT.TXT----------------------------------

             >#encoding UTF-8

             >Indica que el contenido no está traducido o que está traducido parcialmente.

             >Für février Interopérabilité d'activité d'activité

             >-----------------------------------------

             >

             >I am able to read the file in using codecs.open(...), so I don't think it is the actual input text.

             >

             >If there is any help you could give me, or unicode examples I could look at it would be appreciated.

             >

             >Thanks for your help,

             >

             >Charlie

             >

             >

             >

             >

             >

             >-------------------------------------------------------------------------

             >Take Surveys. Earn Cash. Influence the Future of IT

             >Join SourceForge.net's Techsay panel and you'll get the chance to share your

             >opinions on IT & business topics through brief surveys - and earn cash

             >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

             >_______________________________________________

             >Cheetahtemplate-discuss mailing list

             >Cheetahtemplate-discuss@...

             >https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

             >

             

             

             --  

            It's time to get rid of your current e-mail client ... 

            ... and start using si.Mail. 

             

            It's small & free. ( http://simail.sourceforge.net/ )

            -------------------------------------------------------------------------
            Take Surveys. Earn Cash. Influence the Future of IT
            Join SourceForge.net's Techsay panel and you'll get the chance to share your
            opinions on IT & business topics through brief surveys - and earn cash
            http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

            _______________________________________________
            Cheetahtemplate-discuss mailing list
            Cheetahtemplate-discuss@...
            https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

             

          • Miha Vrhovnik
            ... ... cut... Brian you are missing something. I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that s why I m expecting it to
            Message 5 of 30 , Jan 8, 2007
              "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
              >I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
              > the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
              ... cut...

              Brian you are missing something.
              I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.

              Regards,
              Miha

              --
              It's time to get rid of your current e-mail client ...
              ... and start using si.Mail.

              It's small & free. ( http://simail.sourceforge.net/ )

              -------------------------------------------------------------------------
              Take Surveys. Earn Cash. Influence the Future of IT
              Join SourceForge.net's Techsay panel and you'll get the chance to share your
              opinions on IT & business topics through brief surveys - and earn cash
              http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
              _______________________________________________
              Cheetahtemplate-discuss mailing list
              Cheetahtemplate-discuss@...
              https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
            • Brian Bird
              ... UTF-8 ... Ah - I didn t see that in your Xml example, but my solution should work for the original problem (by Charlie Knudson) I don t understand how
              Message 6 of 30 , Jan 8, 2007
                > Brian you are missing something.
                > I told Cheettah that file encoding is utf-8 by this line: #encoding
                UTF-8
                > and that's why I'm expecting it to behave like python e.g convert
                > everything to unicode.

                Ah - I didn't see that in your Xml example, but my 'solution' should
                work for the original problem (by Charlie Knudson)

                I don't understand how using #encoding could possibly help anyway, since
                according to the docs it just adds " # -*- coding: UTF-8 -*-" to the top
                of the compiled .py file. It doesn't say it actually does anything with
                the encoding. See:
                http://www.cheetahtemplate.org/docs/users_guide_html_multipage/moduleFor
                matting.encoding.html

                It wouldn't make sense anyway - Cheetah has to read in the file before
                it could know the encoding, and when it got to the #encoding line it
                would need to re-read it all in again (ok, xml parsers do this to a
                certain extent, but they know the encoding will be on the first line).
                And what if there were two #encoding lines in the template?

                That's why I think you should have to explicitly tell Cheetah the
                encoding when instantiating the Template with a file= argument. Hence
                the workaround I use is to use the source= argument and determine the
                encoding myself (using codecs.open) before Cheetah gets it's hands on it
                :-) There may be a more efficient way to do this, but it's always worked
                for me.

                Brian

                -------------------------------------------------------------------------
                Take Surveys. Earn Cash. Influence the Future of IT
                Join SourceForge.net's Techsay panel and you'll get the chance to share your
                opinions on IT & business topics through brief surveys - and earn cash
                http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                _______________________________________________
                Cheetahtemplate-discuss mailing list
                Cheetahtemplate-discuss@...
                https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
              • Mike Orr
                ... That s outside the scope of WebSafe. WebSafe s job is to escape HTML markup characters, not to do formatting. None of the functions WebSafe is modeled
                Message 7 of 30 , Jan 8, 2007
                  On 1/8/07, Miha Vrhovnik <mvrhov@...> wrote:
                  > >Yes, this is reasonable. str(t) and unicode(t) break if the string
                  > >contains non-ASCII characters due to bugs in Python, so use
                  > >t.respond() instead, which preserves the string-or-unicode status. I
                  > >use precompiled ASCII templates with Unicode placeholder values and
                  > >get Unicode back fine. Perhaps you've discovered a bug in how
                  > >Template(file=) operates re Unicode. Is there a reason you can't use
                  > >precompiled templates?
                  > In my case I cannot use precompiled templates.
                  > But still, cheetah should behave like python and convert input template in unicode.
                  >
                  > >The WebSafe filter is not safe for non-ASCII characters because it
                  > >calls str() on the placeholder value. It needs to be based on
                  > >RawOrEncodedUnicode. I'm trying to clear out a couple other Python
                  > >projects so I can fix up the Filters module and a few other things
                  > >that need done in Cheetah.
                  > While at it please add:
                  > s = s.replace("\n", "<br />")
                  > into Websafe filter.

                  That's outside the scope of WebSafe. WebSafe's job is to escape HTML
                  markup characters, not to do formatting. None of the functions
                  WebSafe is modeled after do that. You can either do this:

                  #from cgi import escape as e
                  #from my_functions import nl_to_br
                  $nl_to_br($e($var))

                  and not use WebSafe, or write a custom filter that does all of it.

                  --
                  Mike Orr <sluggoster@...>

                  -------------------------------------------------------------------------
                  Take Surveys. Earn Cash. Influence the Future of IT
                  Join SourceForge.net's Techsay panel and you'll get the chance to share your
                  opinions on IT & business topics through brief surveys - and earn cash
                  http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                  _______________________________________________
                  Cheetahtemplate-discuss mailing list
                  Cheetahtemplate-discuss@...
                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                • Mike Orr
                  ... Nevertheless it should work, because Template() is supposed to mimic the behavior of writing and importing a template module, including supporting Python s
                  Message 8 of 30 , Jan 8, 2007
                    On 1/8/07, Brian Bird <Brian.Bird@...> wrote:
                    > > Brian you are missing something.
                    > > I told Cheettah that file encoding is utf-8 by this line: #encoding
                    > UTF-8
                    > > and that's why I'm expecting it to behave like python e.g convert
                    > > everything to unicode.
                    >
                    > Ah - I didn't see that in your Xml example, but my 'solution' should
                    > work for the original problem (by Charlie Knudson)
                    >
                    > I don't understand how using #encoding could possibly help anyway, since
                    > according to the docs it just adds " # -*- coding: UTF-8 -*-" to the top
                    > of the compiled .py file. It doesn't say it actually does anything with
                    > the encoding. See:
                    > http://www.cheetahtemplate.org/docs/users_guide_html_multipage/moduleFor
                    > matting.encoding.html
                    >
                    > It wouldn't make sense anyway - Cheetah has to read in the file before
                    > it could know the encoding, and when it got to the #encoding line it
                    > would need to re-read it all in again (ok, xml parsers do this to a
                    > certain extent, but they know the encoding will be on the first line).
                    > And what if there were two #encoding lines in the template?

                    Nevertheless it should work, because Template() is supposed to mimic
                    the behavior of writing and importing a template module, including
                    supporting Python's '# -*- coding:' feature. I suspect #encoding was
                    added without realizing the ramifications to Template(file=). Without
                    analyzing the compiler I can't say whether the module string needs to
                    be converted to Unicode internally to properly mimic Python's 'import'
                    routine, or whether the compiler bypasses the module step and creates
                    the template class directly (which might bypass '# -*- coding:'), or
                    something else.

                    Reading the encoding from '#encoding' should not be an issue because
                    Cheetah has to parse '#encoding' anyway so it can save the value
                    somewhere. And Cheetah already selects one #encoding if multiple are
                    provided: it selects the last one encountered (according to my test
                    just now).

                    Although I'd recommend just making this a syntax error. Why should
                    Cheetah allow multiple #encoding's in a file when it ignores all but
                    the last? Either the encodings are the same, in which case it's a
                    trivial clean-up, or they are different, in which case the template
                    definition is fubar'd. There shouldn't be any effect on
                    inheritance/includes since it's a file-based directive. (By the way,
                    we should also check whether #includes handles #encoding properly,
                    both from a file and from a string.)

                    > That's why I think you should have to explicitly tell Cheetah the
                    > encoding when instantiating the Template with a file= argument.

                    I'm not quite comfortable with:

                    Template(file="foo.tmpl", encoding="utf-8")

                    The Template arguments are already complicated to document, and this
                    would introduce an argument that applies only to file=, and not to
                    src= or a precompiled template. It should not apply to source=
                    because it's more straightforward for users to do the conversion
                    themselves.

                    > Hence
                    > the workaround I use is to use the source= argument and determine the
                    > encoding myself (using codecs.open) before Cheetah gets it's hands on it
                    > :-) There may be a more efficient way to do this, but it's always worked
                    > for me.

                    I would just do this for now.

                    It sounds like we need an analysis of what exactly the parser and
                    compiler do. Added to the virtual TODO, but don't hold your breath.

                    --
                    Mike Orr <sluggoster@...>

                    -------------------------------------------------------------------------
                    Take Surveys. Earn Cash. Influence the Future of IT
                    Join SourceForge.net's Techsay panel and you'll get the chance to share your
                    opinions on IT & business topics through brief surveys - and earn cash
                    http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                    _______________________________________________
                    Cheetahtemplate-discuss mailing list
                    Cheetahtemplate-discuss@...
                    https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                  • Tavis Rudd
                    You re misunderstanding how Python handles file encodings. Python doesn t convert everything to unicode. If you have a utf-8 encoded Python source file with
                    Message 9 of 30 , Mar 28, 2007
                      You're misunderstanding how Python handles file encodings. Python doesn't
                      convert everything to unicode. If you have a utf-8 encoded Python
                      source file with a normal str object in it, the str remains a str
                      regardless of what encoded characters may be in it.

                      If you want Cheetah to return a unicode object as output you must feed it
                      a unicode source parameter. Otherwise you can call .encode('...') on its
                      output to get a unicode object.

                      On Mon, 8 Jan 2007, Miha Vrhovnik wrote:

                      > "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
                      >> I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
                      >> the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
                      > ... cut...
                      >
                      > Brian you are missing something.
                      > I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.
                      >
                      > Regards,
                      > Miha
                      >
                      > --
                      > It's time to get rid of your current e-mail client ...
                      > ... and start using si.Mail.
                      >
                      > It's small & free. ( http://simail.sourceforge.net/ )
                      >
                      > -------------------------------------------------------------------------
                      > Take Surveys. Earn Cash. Influence the Future of IT
                      > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                      > opinions on IT & business topics through brief surveys - and earn cash
                      > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                      > _______________________________________________
                      > Cheetahtemplate-discuss mailing list
                      > Cheetahtemplate-discuss@...
                      > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                      >

                      -------------------------------------------------------------------------
                      Take Surveys. Earn Cash. Influence the Future of IT
                      Join SourceForge.net's Techsay panel and you'll get the chance to share your
                      opinions on IT & business topics through brief surveys-and earn cash
                      http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                      _______________________________________________
                      Cheetahtemplate-discuss mailing list
                      Cheetahtemplate-discuss@...
                      https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                    • Tavis Rudd
                      You re misunderstanding how Python handles file encodings. Python doesn t convert everything to unicode. If you have a utf-8 encoded Python source file with
                      Message 10 of 30 , Mar 28, 2007
                        You're misunderstanding how Python handles file encodings. Python doesn't
                        convert everything to unicode. If you have a utf-8 encoded Python
                        source file with a normal str object in it, the str remains a str
                        regardless of what encoded characters may be in it.

                        If you want Cheetah to return a unicode object as output you must feed it
                        a unicode source parameter. Otherwise you can call .encode('...') on its
                        output to get a unicode object.

                        On Mon, 8 Jan 2007, Miha Vrhovnik wrote:

                        > "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
                        >> I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
                        >> the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
                        > ... cut...
                        >
                        > Brian you are missing something.
                        > I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.
                        >
                        > Regards,
                        > Miha
                        >
                        > --
                        > It's time to get rid of your current e-mail client ...
                        > ... and start using si.Mail.
                        >
                        > It's small & free. ( http://simail.sourceforge.net/ )
                        >
                        > -------------------------------------------------------------------------
                        > Take Surveys. Earn Cash. Influence the Future of IT
                        > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                        > opinions on IT & business topics through brief surveys - and earn cash
                        > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                        > _______________________________________________
                        > Cheetahtemplate-discuss mailing list
                        > Cheetahtemplate-discuss@...
                        > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                        >

                        -------------------------------------------------------------------------
                        Take Surveys. Earn Cash. Influence the Future of IT
                        Join SourceForge.net's Techsay panel and you'll get the chance to share your
                        opinions on IT & business topics through brief surveys-and earn cash
                        http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                        _______________________________________________
                        Cheetahtemplate-discuss mailing list
                        Cheetahtemplate-discuss@...
                        https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                      • Tavis Rudd
                        You re misunderstanding how Python handles file encodings. Python doesn t convert everything to unicode. If you have a utf-8 encoded Python source file with
                        Message 11 of 30 , Mar 28, 2007
                          You're misunderstanding how Python handles file encodings. Python doesn't
                          convert everything to unicode. If you have a utf-8 encoded Python
                          source file with a normal str object in it, the str remains a str
                          regardless of what encoded characters may be in it.

                          If you want Cheetah to return a unicode object as output you must feed it
                          a unicode source parameter. Otherwise you can call .encode('...') on its
                          output to get a unicode object.

                          On Mon, 8 Jan 2007, Miha Vrhovnik wrote:

                          > "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
                          >> I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
                          >> the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
                          > ... cut...
                          >
                          > Brian you are missing something.
                          > I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.
                          >
                          > Regards,
                          > Miha
                          >
                          > --
                          > It's time to get rid of your current e-mail client ...
                          > ... and start using si.Mail.
                          >
                          > It's small & free. ( http://simail.sourceforge.net/ )
                          >
                          > -------------------------------------------------------------------------
                          > Take Surveys. Earn Cash. Influence the Future of IT
                          > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                          > opinions on IT & business topics through brief surveys - and earn cash
                          > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                          > _______________________________________________
                          > Cheetahtemplate-discuss mailing list
                          > Cheetahtemplate-discuss@...
                          > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                          >

                          -------------------------------------------------------------------------
                          Take Surveys. Earn Cash. Influence the Future of IT
                          Join SourceForge.net's Techsay panel and you'll get the chance to share your
                          opinions on IT & business topics through brief surveys-and earn cash
                          http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                          _______________________________________________
                          Cheetahtemplate-discuss mailing list
                          Cheetahtemplate-discuss@...
                          https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                        • Tavis Rudd
                          You re misunderstanding how Python handles file encodings. Python doesn t convert everything to unicode. If you have a utf-8 encoded Python source file with
                          Message 12 of 30 , Mar 28, 2007
                            You're misunderstanding how Python handles file encodings. Python doesn't
                            convert everything to unicode. If you have a utf-8 encoded Python
                            source file with a normal str object in it, the str remains a str
                            regardless of what encoded characters may be in it.

                            If you want Cheetah to return a unicode object as output you must feed it
                            a unicode source parameter. Otherwise you can call .encode('...') on its
                            output to get a unicode object.

                            On Mon, 8 Jan 2007, Miha Vrhovnik wrote:

                            > "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
                            >> I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
                            >> the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
                            > ... cut...
                            >
                            > Brian you are missing something.
                            > I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.
                            >
                            > Regards,
                            > Miha
                            >
                            > --
                            > It's time to get rid of your current e-mail client ...
                            > ... and start using si.Mail.
                            >
                            > It's small & free. ( http://simail.sourceforge.net/ )
                            >
                            > -------------------------------------------------------------------------
                            > Take Surveys. Earn Cash. Influence the Future of IT
                            > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                            > opinions on IT & business topics through brief surveys - and earn cash
                            > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                            > _______________________________________________
                            > Cheetahtemplate-discuss mailing list
                            > Cheetahtemplate-discuss@...
                            > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                            >

                            -------------------------------------------------------------------------
                            Take Surveys. Earn Cash. Influence the Future of IT
                            Join SourceForge.net's Techsay panel and you'll get the chance to share your
                            opinions on IT & business topics through brief surveys-and earn cash
                            http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                            _______________________________________________
                            Cheetahtemplate-discuss mailing list
                            Cheetahtemplate-discuss@...
                            https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                          • Tavis Rudd
                            Good explanation, Brian. I don t think there s anything wrong with the way Cheetah handles encoding. Btw, Cheetah is currently being used, without problems,
                            Message 13 of 30 , Mar 28, 2007
                              Good explanation, Brian. I don't think there's anything wrong with the
                              way Cheetah handles encoding.

                              Btw, Cheetah is currently being used, without problems, for a large number
                              of non-English, non-ascii sites (see the bottom of
                              http://cheetahtemplate.org/whouses.html for some examples).

                              On Mon, 8 Jan 2007, Brian Bird wrote:

                              >> Brian you are missing something.
                              >> I told Cheettah that file encoding is utf-8 by this line: #encoding
                              > UTF-8
                              >> and that's why I'm expecting it to behave like python e.g convert
                              >> everything to unicode.
                              >
                              > Ah - I didn't see that in your Xml example, but my 'solution' should
                              > work for the original problem (by Charlie Knudson)
                              >
                              > I don't understand how using #encoding could possibly help anyway, since
                              > according to the docs it just adds " # -*- coding: UTF-8 -*-" to the top
                              > of the compiled .py file. It doesn't say it actually does anything with
                              > the encoding. See:
                              > http://www.cheetahtemplate.org/docs/users_guide_html_multipage/moduleFor
                              > matting.encoding.html
                              >
                              > It wouldn't make sense anyway - Cheetah has to read in the file before
                              > it could know the encoding, and when it got to the #encoding line it
                              > would need to re-read it all in again (ok, xml parsers do this to a
                              > certain extent, but they know the encoding will be on the first line).
                              > And what if there were two #encoding lines in the template?
                              >
                              > That's why I think you should have to explicitly tell Cheetah the
                              > encoding when instantiating the Template with a file= argument. Hence
                              > the workaround I use is to use the source= argument and determine the
                              > encoding myself (using codecs.open) before Cheetah gets it's hands on it
                              > :-) There may be a more efficient way to do this, but it's always worked
                              > for me.
                              >
                              > Brian
                              >
                              > -------------------------------------------------------------------------
                              > Take Surveys. Earn Cash. Influence the Future of IT
                              > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                              > opinions on IT & business topics through brief surveys - and earn cash
                              > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                              > _______________________________________________
                              > Cheetahtemplate-discuss mailing list
                              > Cheetahtemplate-discuss@...
                              > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                              >

                              -------------------------------------------------------------------------
                              Take Surveys. Earn Cash. Influence the Future of IT
                              Join SourceForge.net's Techsay panel and you'll get the chance to share your
                              opinions on IT & business topics through brief surveys-and earn cash
                              http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                              _______________________________________________
                              Cheetahtemplate-discuss mailing list
                              Cheetahtemplate-discuss@...
                              https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                            • Tavis Rudd
                              Good explanation, Brian. I don t think there s anything wrong with the way Cheetah handles encoding. Btw, Cheetah is currently being used, without problems,
                              Message 14 of 30 , Mar 28, 2007
                                Good explanation, Brian. I don't think there's anything wrong with the
                                way Cheetah handles encoding.

                                Btw, Cheetah is currently being used, without problems, for a large number
                                of non-English, non-ascii sites (see the bottom of
                                http://cheetahtemplate.org/whouses.html for some examples).

                                On Mon, 8 Jan 2007, Brian Bird wrote:

                                >> Brian you are missing something.
                                >> I told Cheettah that file encoding is utf-8 by this line: #encoding
                                > UTF-8
                                >> and that's why I'm expecting it to behave like python e.g convert
                                >> everything to unicode.
                                >
                                > Ah - I didn't see that in your Xml example, but my 'solution' should
                                > work for the original problem (by Charlie Knudson)
                                >
                                > I don't understand how using #encoding could possibly help anyway, since
                                > according to the docs it just adds " # -*- coding: UTF-8 -*-" to the top
                                > of the compiled .py file. It doesn't say it actually does anything with
                                > the encoding. See:
                                > http://www.cheetahtemplate.org/docs/users_guide_html_multipage/moduleFor
                                > matting.encoding.html
                                >
                                > It wouldn't make sense anyway - Cheetah has to read in the file before
                                > it could know the encoding, and when it got to the #encoding line it
                                > would need to re-read it all in again (ok, xml parsers do this to a
                                > certain extent, but they know the encoding will be on the first line).
                                > And what if there were two #encoding lines in the template?
                                >
                                > That's why I think you should have to explicitly tell Cheetah the
                                > encoding when instantiating the Template with a file= argument. Hence
                                > the workaround I use is to use the source= argument and determine the
                                > encoding myself (using codecs.open) before Cheetah gets it's hands on it
                                > :-) There may be a more efficient way to do this, but it's always worked
                                > for me.
                                >
                                > Brian
                                >
                                > -------------------------------------------------------------------------
                                > Take Surveys. Earn Cash. Influence the Future of IT
                                > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                > opinions on IT & business topics through brief surveys - and earn cash
                                > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                > _______________________________________________
                                > Cheetahtemplate-discuss mailing list
                                > Cheetahtemplate-discuss@...
                                > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                >

                                -------------------------------------------------------------------------
                                Take Surveys. Earn Cash. Influence the Future of IT
                                Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                opinions on IT & business topics through brief surveys-and earn cash
                                http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                _______________________________________________
                                Cheetahtemplate-discuss mailing list
                                Cheetahtemplate-discuss@...
                                https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                              • Tavis Rudd
                                Good explanation, Brian. I don t think there s anything wrong with the way Cheetah handles encoding. Btw, Cheetah is currently being used, without problems,
                                Message 15 of 30 , Mar 28, 2007
                                  Good explanation, Brian. I don't think there's anything wrong with the
                                  way Cheetah handles encoding.

                                  Btw, Cheetah is currently being used, without problems, for a large number
                                  of non-English, non-ascii sites (see the bottom of
                                  http://cheetahtemplate.org/whouses.html for some examples).

                                  On Mon, 8 Jan 2007, Brian Bird wrote:

                                  >> Brian you are missing something.
                                  >> I told Cheettah that file encoding is utf-8 by this line: #encoding
                                  > UTF-8
                                  >> and that's why I'm expecting it to behave like python e.g convert
                                  >> everything to unicode.
                                  >
                                  > Ah - I didn't see that in your Xml example, but my 'solution' should
                                  > work for the original problem (by Charlie Knudson)
                                  >
                                  > I don't understand how using #encoding could possibly help anyway, since
                                  > according to the docs it just adds " # -*- coding: UTF-8 -*-" to the top
                                  > of the compiled .py file. It doesn't say it actually does anything with
                                  > the encoding. See:
                                  > http://www.cheetahtemplate.org/docs/users_guide_html_multipage/moduleFor
                                  > matting.encoding.html
                                  >
                                  > It wouldn't make sense anyway - Cheetah has to read in the file before
                                  > it could know the encoding, and when it got to the #encoding line it
                                  > would need to re-read it all in again (ok, xml parsers do this to a
                                  > certain extent, but they know the encoding will be on the first line).
                                  > And what if there were two #encoding lines in the template?
                                  >
                                  > That's why I think you should have to explicitly tell Cheetah the
                                  > encoding when instantiating the Template with a file= argument. Hence
                                  > the workaround I use is to use the source= argument and determine the
                                  > encoding myself (using codecs.open) before Cheetah gets it's hands on it
                                  > :-) There may be a more efficient way to do this, but it's always worked
                                  > for me.
                                  >
                                  > Brian
                                  >
                                  > -------------------------------------------------------------------------
                                  > Take Surveys. Earn Cash. Influence the Future of IT
                                  > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                  > opinions on IT & business topics through brief surveys - and earn cash
                                  > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                  > _______________________________________________
                                  > Cheetahtemplate-discuss mailing list
                                  > Cheetahtemplate-discuss@...
                                  > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                  >

                                  -------------------------------------------------------------------------
                                  Take Surveys. Earn Cash. Influence the Future of IT
                                  Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                  opinions on IT & business topics through brief surveys-and earn cash
                                  http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                  _______________________________________________
                                  Cheetahtemplate-discuss mailing list
                                  Cheetahtemplate-discuss@...
                                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                • Miha Vrhovnik
                                  - I feed cheetah with unicode strings - result MUST be unicode - files are supposed to be utf-8 encoded - I don t have control over template files as I m using
                                  Message 16 of 30 , Mar 29, 2007
                                    - I feed cheetah with unicode strings
                                    - result MUST be unicode
                                    - files are supposed to be utf-8 encoded
                                    - I don't have control over template files as I'm using python as scripting language in my app. and everything I put to him is in unicode so that's what I expect when getting strings from it.

                                    Any recommendations?

                                    Mike said, that WebEncode filter is not unicode enabled and this is supposed to be anther reason for getting unicode errors.

                                    Regards,
                                    Miha

                                    "Tavis Rudd" <tavis@...> wrote on 29.3.2007 0:46:43:
                                    >You're misunderstanding how Python handles file encodings. Python doesn't
                                    >convert everything to unicode. If you have a utf-8 encoded Python
                                    >source file with a normal str object in it, the str remains a str
                                    >regardless of what encoded characters may be in it.
                                    >
                                    >If you want Cheetah to return a unicode object as output you must feed it
                                    >a unicode source parameter. Otherwise you can call .encode('...') on its
                                    >output to get a unicode object.
                                    >
                                    >On Mon, 8 Jan 2007, Miha Vrhovnik wrote:
                                    >
                                    >> "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
                                    >>> I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
                                    >>> the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
                                    >> ... cut...
                                    >>
                                    >> Brian you are missing something.
                                    >> I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.
                                    >>
                                    >> Regards,
                                    >> Miha
                                    >>
                                    >> --
                                    >> It's time to get rid of your current e-mail client ...
                                    >> ... and start using si.Mail.
                                    >>
                                    >> It's small & free. ( http://simail.sourceforge.net/ )
                                    >>
                                    >> -------------------------------------------------------------------------
                                    >> Take Surveys. Earn Cash. Influence the Future of IT
                                    >> Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                    >> opinions on IT & business topics through brief surveys - and earn cash
                                    >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                    >> _______________________________________________
                                    >> Cheetahtemplate-discuss mailing list
                                    >> Cheetahtemplate-discuss@...
                                    >> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                    >>
                                    >
                                    >-------------------------------------------------------------------------
                                    >Take Surveys. Earn Cash. Influence the Future of IT
                                    >Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                    >opinions on IT & business topics through brief surveys-and earn cash
                                    >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                    >_______________________________________________
                                    >Cheetahtemplate-discuss mailing list
                                    >Cheetahtemplate-discuss@...
                                    >https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                    >


                                    --
                                    It's time to get rid of your current e-mail client ...
                                    ... and start using si.Mail.

                                    It's small & free. ( http://simail.sourceforge.net/ )

                                    -------------------------------------------------------------------------
                                    Take Surveys. Earn Cash. Influence the Future of IT
                                    Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                    opinions on IT & business topics through brief surveys-and earn cash
                                    http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                    _______________________________________________
                                    Cheetahtemplate-discuss mailing list
                                    Cheetahtemplate-discuss@...
                                    https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                  • Tavis Rudd
                                    Hi Miha, I d put the template source as unicode strings in .py files instead of .tmpl files, import them, then give them to Cheetah with the source argument
                                    Message 17 of 30 , Mar 29, 2007
                                      Hi Miha,
                                      I'd put the template source as unicode strings in .py files instead of
                                      .tmpl files, import them, then give them to Cheetah with the source
                                      argument rather than the file argument.

                                      Unfortunately I know nothing about that WebEncode filter.

                                      Cheers,
                                      Tavis

                                      On Thu, 29 Mar 2007, Miha Vrhovnik wrote:

                                      > - I feed cheetah with unicode strings
                                      > - result MUST be unicode
                                      > - files are supposed to be utf-8 encoded
                                      > - I don't have control over template files as I'm using python as scripting language in my app. and everything I put to him is in unicode so that's what I expect when getting strings from it.
                                      >
                                      > Any recommendations?
                                      >
                                      > Mike said, that WebEncode filter is not unicode enabled and this is supposed to be anther reason for getting unicode errors.
                                      >
                                      > Regards,
                                      > Miha
                                      >
                                      > "Tavis Rudd" <tavis@...> wrote on 29.3.2007 0:46:43:
                                      >> You're misunderstanding how Python handles file encodings. Python doesn't
                                      >> convert everything to unicode. If you have a utf-8 encoded Python
                                      >> source file with a normal str object in it, the str remains a str
                                      >> regardless of what encoded characters may be in it.
                                      >>
                                      >> If you want Cheetah to return a unicode object as output you must feed it
                                      >> a unicode source parameter. Otherwise you can call .encode('...') on its
                                      >> output to get a unicode object.
                                      >>
                                      >> On Mon, 8 Jan 2007, Miha Vrhovnik wrote:
                                      >>
                                      >>> "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
                                      >>>> I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
                                      >>>> the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
                                      >>> ... cut...
                                      >>>
                                      >>> Brian you are missing something.
                                      >>> I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.
                                      >>>
                                      >>> Regards,
                                      >>> Miha
                                      >>>
                                      >>> --
                                      >>> It's time to get rid of your current e-mail client ...
                                      >>> ... and start using si.Mail.
                                      >>>
                                      >>> It's small & free. ( http://simail.sourceforge.net/ )
                                      >>>
                                      >>> -------------------------------------------------------------------------
                                      >>> Take Surveys. Earn Cash. Influence the Future of IT
                                      >>> Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                      >>> opinions on IT & business topics through brief surveys - and earn cash
                                      >>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                      >>> _______________________________________________
                                      >>> Cheetahtemplate-discuss mailing list
                                      >>> Cheetahtemplate-discuss@...
                                      >>> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                      >>>
                                      >>
                                      >> -------------------------------------------------------------------------
                                      >> Take Surveys. Earn Cash. Influence the Future of IT
                                      >> Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                      >> opinions on IT & business topics through brief surveys-and earn cash
                                      >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                      >> _______________________________________________
                                      >> Cheetahtemplate-discuss mailing list
                                      >> Cheetahtemplate-discuss@...
                                      >> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                      >>
                                      >
                                      >
                                      > --
                                      > It's time to get rid of your current e-mail client ...
                                      > ... and start using si.Mail.
                                      >
                                      > It's small & free. ( http://simail.sourceforge.net/ )
                                      >
                                      > -------------------------------------------------------------------------
                                      > Take Surveys. Earn Cash. Influence the Future of IT
                                      > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                      > opinions on IT & business topics through brief surveys-and earn cash
                                      > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                      > _______________________________________________
                                      > Cheetahtemplate-discuss mailing list
                                      > Cheetahtemplate-discuss@...
                                      > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                      >

                                      -------------------------------------------------------------------------
                                      Take Surveys. Earn Cash. Influence the Future of IT
                                      Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                      opinions on IT & business topics through brief surveys-and earn cash
                                      http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                      _______________________________________________
                                      Cheetahtemplate-discuss mailing list
                                      Cheetahtemplate-discuss@...
                                      https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                    • Mike Orr
                                      ... The problem is Cheetah/Filters.py line 213: s = str(val) This raises UnicodeEncodeError if it s a unicode string containing non-ASCII characters. It
                                      Message 18 of 30 , Mar 29, 2007
                                        On 3/29/07, Tavis Rudd <tavis@...> wrote:
                                        > Hi Miha,
                                        > I'd put the template source as unicode strings in .py files instead of
                                        > .tmpl files, import them, then give them to Cheetah with the source
                                        > argument rather than the file argument.
                                        >
                                        > Unfortunately I know nothing about that WebEncode filter.'

                                        The problem is Cheetah/Filters.py line 213:

                                        s = str(val)

                                        This raises UnicodeEncodeError if it's a unicode string containing
                                        non-ASCII characters. It should avoid converting it if it's already
                                        unicode, as RawOrEncodedUnicode.filter does.

                                        The Filter, ReplaceNone, MaxLen, Pager, Strip, and StripSqueeze
                                        filters are also broken the same way.

                                        Tavis, how about we rename RawOrEncodedUnicode to Filter, and make
                                        every subclass call its .filter() method instead of str(). I'd also
                                        change

                                        if type(val)==unicodeType:

                                        to

                                        if isinstance(val, unicode)

                                        This will make it work with unicode subclasses as PTY uses and Quixote
                                        will soon use.

                                        I've been trying to make these changes myself but SourceForge's CVS
                                        hasn't been cooperating.

                                        --
                                        Mike Orr <sluggoster@...>

                                        -------------------------------------------------------------------------
                                        Take Surveys. Earn Cash. Influence the Future of IT
                                        Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                        opinions on IT & business topics through brief surveys-and earn cash
                                        http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                        _______________________________________________
                                        Cheetahtemplate-discuss mailing list
                                        Cheetahtemplate-discuss@...
                                        https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                      • Tavis Rudd
                                        Sure. I ve just checked it in. I also got rid of that old Pager filter, which was a quick demonstration hack I added years ago, and the now redundant
                                        Message 19 of 30 , Mar 29, 2007
                                          Sure. I've just checked it in.

                                          I also got rid of that old Pager filter, which was a quick demonstration
                                          hack I added years ago, and the now redundant ReplaceNone filter. If
                                          anyone is relying on this, please say so now.

                                          I think we could get rid of that setting/settings stuff in
                                          Filter.__init__ which uses the DummyTemplate. I'm not aware of it ever
                                          having been used. Any objections?

                                          On Thu, 29 Mar 2007, Mike Orr wrote:

                                          > On 3/29/07, Tavis Rudd <tavis@...> wrote:
                                          >> Hi Miha,
                                          >> I'd put the template source as unicode strings in .py files instead of
                                          >> .tmpl files, import them, then give them to Cheetah with the source
                                          >> argument rather than the file argument.
                                          >>
                                          >> Unfortunately I know nothing about that WebEncode filter.'
                                          >
                                          > The problem is Cheetah/Filters.py line 213:
                                          >
                                          > s = str(val)
                                          >
                                          > This raises UnicodeEncodeError if it's a unicode string containing
                                          > non-ASCII characters. It should avoid converting it if it's already
                                          > unicode, as RawOrEncodedUnicode.filter does.
                                          >
                                          > The Filter, ReplaceNone, MaxLen, Pager, Strip, and StripSqueeze
                                          > filters are also broken the same way.
                                          >
                                          > Tavis, how about we rename RawOrEncodedUnicode to Filter, and make
                                          > every subclass call its .filter() method instead of str(). I'd also
                                          > change
                                          >
                                          > if type(val)==unicodeType:
                                          >
                                          > to
                                          >
                                          > if isinstance(val, unicode)
                                          >
                                          > This will make it work with unicode subclasses as PTY uses and Quixote
                                          > will soon use.
                                          >
                                          > I've been trying to make these changes myself but SourceForge's CVS
                                          > hasn't been cooperating.
                                          >
                                          > --
                                          > Mike Orr <sluggoster@...>
                                          >

                                          -------------------------------------------------------------------------
                                          Take Surveys. Earn Cash. Influence the Future of IT
                                          Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                          opinions on IT & business topics through brief surveys-and earn cash
                                          http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                          _______________________________________________
                                          Cheetahtemplate-discuss mailing list
                                          Cheetahtemplate-discuss@...
                                          https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                        • Tavis Rudd
                                          One thing I d consider is adding a #unicode directive for use with .tmpl source files. It would indicate that Cheetah should output unicode strings for the
                                          Message 20 of 30 , Mar 29, 2007
                                            One thing I'd consider is adding a #unicode directive for use with .tmpl
                                            source files. It would indicate that Cheetah should output unicode
                                            strings for the template in all situations.
                                            The template objects created by all of these idioms
                                            Template.compile(file='foo.tmpl') # with #unicode in foo.tmpl
                                            Template.compile(source='''#unicode\n ...''')
                                            Template.compile(source=u'''some unicode string''')
                                            would return unicode strings from their render methods.

                                            I don't want to make unicode output the default behaviour as it
                                            will break existing applications, my own included, that expect encoded
                                            strings rather than decoded unicode objects as output.

                                            Tavis

                                            On Thu, 29 Mar 2007, Tavis Rudd wrote:

                                            > Hi Miha,
                                            > I'd put the template source as unicode strings in .py files instead of
                                            > .tmpl files, import them, then give them to Cheetah with the source
                                            > argument rather than the file argument.
                                            >
                                            > Unfortunately I know nothing about that WebEncode filter.
                                            >
                                            > Cheers,
                                            > Tavis
                                            >
                                            > On Thu, 29 Mar 2007, Miha Vrhovnik wrote:
                                            >
                                            >> - I feed cheetah with unicode strings
                                            >> - result MUST be unicode
                                            >> - files are supposed to be utf-8 encoded
                                            >> - I don't have control over template files as I'm using python as scripting language in my app. and everything I put to him is in unicode so that's what I expect when getting strings from it.
                                            >>
                                            >> Any recommendations?
                                            >>
                                            >> Mike said, that WebEncode filter is not unicode enabled and this is supposed to be anther reason for getting unicode errors.
                                            >>
                                            >> Regards,
                                            >> Miha
                                            >>
                                            >> "Tavis Rudd" <tavis@...> wrote on 29.3.2007 0:46:43:
                                            >>> You're misunderstanding how Python handles file encodings. Python doesn't
                                            >>> convert everything to unicode. If you have a utf-8 encoded Python
                                            >>> source file with a normal str object in it, the str remains a str
                                            >>> regardless of what encoded characters may be in it.
                                            >>>
                                            >>> If you want Cheetah to return a unicode object as output you must feed it
                                            >>> a unicode source parameter. Otherwise you can call .encode('...') on its
                                            >>> output to get a unicode object.
                                            >>>
                                            >>> On Mon, 8 Jan 2007, Miha Vrhovnik wrote:
                                            >>>
                                            >>>> "Brian Bird" <brian.bird@...> wrote on 8.1.2007 9:16:41:
                                            >>>>> I think the problem is that the file= argument expects to read in the file as bytes because there is nowhere to specify
                                            >>>>> the encoding (perhaps there should be an extra keyword argument called encoding= and if specified then Cheetah
                                            >>>> ... cut...
                                            >>>>
                                            >>>> Brian you are missing something.
                                            >>>> I told Cheettah that file encoding is utf-8 by this line: #encoding UTF-8 and that's why I'm expecting it to behave like python e.g convert everything to unicode.
                                            >>>>
                                            >>>> Regards,
                                            >>>> Miha
                                            >>>>
                                            >>>> --
                                            >>>> It's time to get rid of your current e-mail client ...
                                            >>>> ... and start using si.Mail.
                                            >>>>
                                            >>>> It's small & free. ( http://simail.sourceforge.net/ )
                                            >>>>
                                            >>>> -------------------------------------------------------------------------
                                            >>>> Take Surveys. Earn Cash. Influence the Future of IT
                                            >>>> Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                            >>>> opinions on IT & business topics through brief surveys - and earn cash
                                            >>>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                            >>>> _______________________________________________
                                            >>>> Cheetahtemplate-discuss mailing list
                                            >>>> Cheetahtemplate-discuss@...
                                            >>>> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                            >>>>
                                            >>>
                                            >>> -------------------------------------------------------------------------
                                            >>> Take Surveys. Earn Cash. Influence the Future of IT
                                            >>> Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                            >>> opinions on IT & business topics through brief surveys-and earn cash
                                            >>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                            >>> _______________________________________________
                                            >>> Cheetahtemplate-discuss mailing list
                                            >>> Cheetahtemplate-discuss@...
                                            >>> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                            >>>
                                            >>
                                            >>
                                            >> --
                                            >> It's time to get rid of your current e-mail client ...
                                            >> ... and start using si.Mail.
                                            >>
                                            >> It's small & free. ( http://simail.sourceforge.net/ )
                                            >>
                                            >> -------------------------------------------------------------------------
                                            >> Take Surveys. Earn Cash. Influence the Future of IT
                                            >> Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                            >> opinions on IT & business topics through brief surveys-and earn cash
                                            >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                            >> _______________________________________________
                                            >> Cheetahtemplate-discuss mailing list
                                            >> Cheetahtemplate-discuss@...
                                            >> https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                            >>
                                            >
                                            > -------------------------------------------------------------------------
                                            > Take Surveys. Earn Cash. Influence the Future of IT
                                            > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                            > opinions on IT & business topics through brief surveys-and earn cash
                                            > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                            > _______________________________________________
                                            > Cheetahtemplate-discuss mailing list
                                            > Cheetahtemplate-discuss@...
                                            > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                            >

                                            -------------------------------------------------------------------------
                                            Take Surveys. Earn Cash. Influence the Future of IT
                                            Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                            opinions on IT & business topics through brief surveys-and earn cash
                                            http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                            _______________________________________________
                                            Cheetahtemplate-discuss mailing list
                                            Cheetahtemplate-discuss@...
                                            https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                          • Miha Vrhovnik
                                            Travis this would be very very good solution especially if it would take the #encoding line as input encoding to convert everything to unicode. Regards, Miha
                                            Message 21 of 30 , Mar 29, 2007
                                              Travis this would be very very good solution especially if it would take the #encoding line as input encoding to convert everything to unicode.

                                              Regards,
                                              Miha

                                              "Tavis Rudd" <tavis@...> wrote on 29.3.2007 20:40:49:
                                              >One thing I'd consider is adding a #unicode directive for use with .tmpl
                                              >source files. It would indicate that Cheetah should output unicode
                                              >strings for the template in all situations.
                                              >The template objects created by all of these idioms
                                              > Template.compile(file='foo.tmpl') # with #unicode in foo.tmpl
                                              > Template.compile(source='''#unicode\n ...''')
                                              > Template.compile(source=u'''some unicode string''')
                                              >would return unicode strings from their render methods.
                                              >
                                              >I don't want to make unicode output the default behaviour as it
                                              >will break existing applications, my own included, that expect encoded
                                              >strings rather than decoded unicode objects as output.
                                              >
                                              >Tavis
                                              >

                                              -------------------------------------------------------------------------
                                              Take Surveys. Earn Cash. Influence the Future of IT
                                              Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                              opinions on IT & business topics through brief surveys-and earn cash
                                              http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                              _______________________________________________
                                              Cheetahtemplate-discuss mailing list
                                              Cheetahtemplate-discuss@...
                                              https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                            • Mike Orr
                                              ... Agreed, both Filter.__init__ and DummyTemplate can go. They should have gone when #settings was eliminated. Filter.generateAutoArgs can go too. Does
                                              Message 22 of 30 , Mar 29, 2007
                                                On 3/29/07, Tavis Rudd <tavis@...> wrote:
                                                > I think we could get rid of that setting/settings stuff in
                                                > Filter.__init__ which uses the DummyTemplate. I'm not aware of it ever
                                                > having been used. Any objections?

                                                Agreed, both Filter.__init__ and DummyTemplate can go. They should
                                                have gone when #settings was eliminated.

                                                Filter.generateAutoArgs can go too. Does Cheetah ever call it?
                                                Apparently no filter except Pager has ever needed it.

                                                --
                                                Mike Orr <sluggoster@...>

                                                -------------------------------------------------------------------------
                                                Take Surveys. Earn Cash. Influence the Future of IT
                                                Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                opinions on IT & business topics through brief surveys-and earn cash
                                                http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                _______________________________________________
                                                Cheetahtemplate-discuss mailing list
                                                Cheetahtemplate-discuss@...
                                                https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                              • Tavis Rudd
                                                The #unicode and #encoding directives could be combined in the form: #unicode utf-8 #unicode latin-1 etc. ... Take Surveys. Earn Cash. Influence the Future of
                                                Message 23 of 30 , Mar 29, 2007
                                                  The #unicode and #encoding directives could be combined in the form:
                                                  #unicode utf-8
                                                  #unicode latin-1
                                                  etc.

                                                  On Thu, 29 Mar 2007, Miha Vrhovnik wrote:

                                                  > Travis this would be very very good solution especially if it would take the #encoding line as input encoding to convert everything to unicode.
                                                  >
                                                  > Regards,
                                                  > Miha
                                                  >
                                                  > "Tavis Rudd" <tavis@...> wrote on 29.3.2007 20:40:49:
                                                  >> One thing I'd consider is adding a #unicode directive for use with .tmpl
                                                  >> source files. It would indicate that Cheetah should output unicode
                                                  >> strings for the template in all situations.
                                                  >> The template objects created by all of these idioms
                                                  >> Template.compile(file='foo.tmpl') # with #unicode in foo.tmpl
                                                  >> Template.compile(source='''#unicode\n ...''')
                                                  >> Template.compile(source=u'''some unicode string''')
                                                  >> would return unicode strings from their render methods.
                                                  >>
                                                  >> I don't want to make unicode output the default behaviour as it
                                                  >> will break existing applications, my own included, that expect encoded
                                                  >> strings rather than decoded unicode objects as output.
                                                  >>
                                                  >> Tavis
                                                  >>
                                                  >
                                                  > -------------------------------------------------------------------------
                                                  > Take Surveys. Earn Cash. Influence the Future of IT
                                                  > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                  > opinions on IT & business topics through brief surveys-and earn cash
                                                  > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                  > _______________________________________________
                                                  > Cheetahtemplate-discuss mailing list
                                                  > Cheetahtemplate-discuss@...
                                                  > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                  >

                                                  -------------------------------------------------------------------------
                                                  Take Surveys. Earn Cash. Influence the Future of IT
                                                  Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                  opinions on IT & business topics through brief surveys-and earn cash
                                                  http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                  _______________________________________________
                                                  Cheetahtemplate-discuss mailing list
                                                  Cheetahtemplate-discuss@...
                                                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                • Mike Orr
                                                  ... Hmm. What about this? #unicode = Make output unicode. #unicode FOO = Make output unicode, and set input encoding. (Does this set the
                                                  Message 24 of 30 , Mar 29, 2007
                                                    On 3/29/07, Tavis Rudd <tavis@...> wrote:
                                                    > The #unicode and #encoding directives could be combined in the form:
                                                    > #unicode utf-8
                                                    > #unicode latin-1
                                                    > etc.

                                                    Hmm. What about this?

                                                    #unicode => Make output unicode.
                                                    #unicode FOO => Make output unicode, and set input encoding.
                                                    (Does this set the placeholder values' encoding too?)
                                                    no directive => Template must be ASCII. Output will be unicode if
                                                    any placeholder value is unicode. (Possible exception if another value is a
                                                    non-ASCII str.)


                                                    #encoding sets the input encoding but does not make the output unicode.
                                                    Its original purpose was to provide an "# -*- encoding" for non-ASCII
                                                    templates, something that Python 2.4 suddenly required. I guess it
                                                    should have made the output unicode too. But now there are lots of
                                                    sites using it as is. Would they break if it started outputting
                                                    unicode?

                                                    --
                                                    Mike Orr <sluggoster@...>

                                                    -------------------------------------------------------------------------
                                                    Take Surveys. Earn Cash. Influence the Future of IT
                                                    Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                    opinions on IT & business topics through brief surveys-and earn cash
                                                    http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                    _______________________________________________
                                                    Cheetahtemplate-discuss mailing list
                                                    Cheetahtemplate-discuss@...
                                                    https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                  • Tavis Rudd
                                                    ... using the system default input encoding ... No, it wouldn t do anything to the the encoding of placeholder values by default. However, we could provide a
                                                    Message 25 of 30 , Mar 29, 2007
                                                      On Thu, 29 Mar 2007, Mike Orr wrote:
                                                      > On 3/29/07, Tavis Rudd <tavis@...> wrote:
                                                      >> The #unicode and #encoding directives could be combined in the form:
                                                      >> # unicode utf-8
                                                      >> # unicode latin-1
                                                      >> etc.
                                                      >
                                                      > Hmm. What about this?
                                                      >
                                                      > # unicode => Make output unicode.
                                                      using the system default input encoding

                                                      > # unicode FOO => Make output unicode, and set input encoding.
                                                      > (Does this set the placeholder values' encoding too?)
                                                      No, it wouldn't do anything to the the encoding of placeholder values by
                                                      default. However, we could provide a filter that does that.

                                                      > no directive => Template must be ASCII. Output will be unicode if
                                                      > any placeholder value is unicode. (Possible exception if another value
                                                      > is a non-ASCII str.)
                                                      Well, technically the requirement isn't always ASCII, it's whatever the
                                                      default encoding is set to.

                                                      > #encoding sets the input encoding but does not make the output unicode.
                                                      > Its original purpose was to provide an "# -*- encoding" for non-ASCII
                                                      > templates, something that Python 2.4 suddenly required. I guess it
                                                      > should have made the output unicode too. But now there are lots of
                                                      > sites using it as is. Would they break if it started outputting
                                                      > unicode?

                                                      Yes, it would break things.



                                                      -------------------------------------------------------------------------
                                                      Take Surveys. Earn Cash. Influence the Future of IT
                                                      Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                      opinions on IT & business topics through brief surveys-and earn cash
                                                      http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                      _______________________________________________
                                                      Cheetahtemplate-discuss mailing list
                                                      Cheetahtemplate-discuss@...
                                                      https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                    • Mike Orr
                                                      ... The default encoding *is* ASCII for all practical purposes. Applications can t change it. The only way to change it is in site.py. -- Mike Orr
                                                      Message 26 of 30 , Mar 29, 2007
                                                        On 3/29/07, Tavis Rudd <tavis@...> wrote:
                                                        > > no directive => Template must be ASCII. Output will be unicode if
                                                        > > any placeholder value is unicode. (Possible exception if another value
                                                        > > is a non-ASCII str.)
                                                        > Well, technically the requirement isn't always ASCII, it's whatever the
                                                        > default encoding is set to.

                                                        The default encoding *is* ASCII for all practical purposes.
                                                        Applications can't change it. The only way to change it is in
                                                        site.py.

                                                        --
                                                        Mike Orr <sluggoster@...>

                                                        -------------------------------------------------------------------------
                                                        Take Surveys. Earn Cash. Influence the Future of IT
                                                        Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                        opinions on IT & business topics through brief surveys-and earn cash
                                                        http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                        _______________________________________________
                                                        Cheetahtemplate-discuss mailing list
                                                        Cheetahtemplate-discuss@...
                                                        https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                      • Tavis Rudd
                                                        I believe this issue with WebSafe is fixed in the cvs now. ... Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net s Techsay panel and
                                                        Message 27 of 30 , Apr 2 7:22 PM
                                                          I believe this issue with WebSafe is fixed in the cvs now.

                                                          On Sun, 7 Jan 2007, Mike Orr wrote:

                                                          > On 1/7/07, Charlie knudsen <charlie_knudsen@...> wrote:
                                                          >>
                                                          >> I may have made some progress on my issue. The below code is able to
                                                          >> process the file:
                                                          >> -----------------------
                                                          >> t = Template(file="input.txt")
                                                          >> out = t.respond()
                                                          >> if type(out) != type(u''):
                                                          >> out = out.decode('utf-8')
                                                          >> print out.encode('utf-8')
                                                          >> ---------------------------
                                                          >> While this solves the simple example I outlined, I have not yet plugged this
                                                          >> fix or variation of this fix into the actual my actual project. So please
                                                          >> let me know if you could let me know if I am doing anything wrong, or why
                                                          >> the decode function is needed I would appreciate it.
                                                          >
                                                          > Yes, this is reasonable. str(t) and unicode(t) break if the string
                                                          > contains non-ASCII characters due to bugs in Python, so use
                                                          > t.respond() instead, which preserves the string-or-unicode status. I
                                                          > use precompiled ASCII templates with Unicode placeholder values and
                                                          > get Unicode back fine. Perhaps you've discovered a bug in how
                                                          > Template(file=) operates re Unicode. Is there a reason you can't use
                                                          > precompiled templates?
                                                          >
                                                          > The WebSafe filter is not safe for non-ASCII characters because it
                                                          > calls str() on the placeholder value. It needs to be based on
                                                          > RawOrEncodedUnicode. I'm trying to clear out a couple other Python
                                                          > projects so I can fix up the Filters module and a few other things
                                                          > that need done in Cheetah.
                                                          >
                                                          > Sorry for not replying sooner. I was sick the past few days so I just
                                                          > read your message quickly, and didn't realize the problem was so
                                                          > similar to one I had had.
                                                          >
                                                          > --
                                                          > Mike Orr <sluggoster@...>
                                                          >
                                                          > -------------------------------------------------------------------------
                                                          > Take Surveys. Earn Cash. Influence the Future of IT
                                                          > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                          > opinions on IT & business topics through brief surveys - and earn cash
                                                          > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                          > _______________________________________________
                                                          > Cheetahtemplate-discuss mailing list
                                                          > Cheetahtemplate-discuss@...
                                                          > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                          >

                                                          -------------------------------------------------------------------------
                                                          Take Surveys. Earn Cash. Influence the Future of IT
                                                          Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                          opinions on IT & business topics through brief surveys-and earn cash
                                                          http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                          _______________________________________________
                                                          Cheetahtemplate-discuss mailing list
                                                          Cheetahtemplate-discuss@...
                                                          https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                        • Tavis Rudd
                                                          The #unicode directive is implemented in the cvs now. Please note: - #unicode and #encoding are mutually exclusive. Use one or the other. - #unicode must be
                                                          Message 28 of 30 , Apr 2 8:06 PM
                                                            The #unicode directive is implemented in the cvs now.
                                                            Please note:
                                                            - #unicode and #encoding are mutually exclusive. Use one or the other.
                                                            - #unicode must be on a line by itself.
                                                            - Strings in embedded code must be explictly marked as unicode if they
                                                            contain non-ascii chars:

                                                            #unicode latin-1
                                                            $f("<some non-ascii char>") ## wrong
                                                            $f(u"<some non-ascii char>") ## right

                                                            However, this works fine:

                                                            #unicode latin-1
                                                            blah blah <some non-ascii char> blah blah

                                                            Cheers,
                                                            Tavis




                                                            On Thu, 29 Mar 2007, Tavis Rudd wrote:
                                                            >
                                                            >
                                                            > On Thu, 29 Mar 2007, Mike Orr wrote:
                                                            >> On 3/29/07, Tavis Rudd <tavis@...> wrote:
                                                            >>> The #unicode and #encoding directives could be combined in the form:
                                                            >>> # unicode utf-8
                                                            >>> # unicode latin-1
                                                            >>> etc.
                                                            >>
                                                            >> Hmm. What about this?
                                                            >>
                                                            >> # unicode => Make output unicode.
                                                            > using the system default input encoding
                                                            >
                                                            >> # unicode FOO => Make output unicode, and set input encoding.
                                                            >> (Does this set the placeholder values' encoding too?)
                                                            > No, it wouldn't do anything to the the encoding of placeholder values by
                                                            > default. However, we could provide a filter that does that.
                                                            >
                                                            >> no directive => Template must be ASCII. Output will be unicode if
                                                            >> any placeholder value is unicode. (Possible exception if another value
                                                            >> is a non-ASCII str.)
                                                            > Well, technically the requirement isn't always ASCII, it's whatever the
                                                            > default encoding is set to.
                                                            >
                                                            >> #encoding sets the input encoding but does not make the output unicode.
                                                            >> Its original purpose was to provide an "# -*- encoding" for non-ASCII
                                                            >> templates, something that Python 2.4 suddenly required. I guess it
                                                            >> should have made the output unicode too. But now there are lots of
                                                            >> sites using it as is. Would they break if it started outputting
                                                            >> unicode?
                                                            >
                                                            > Yes, it would break things.
                                                            >
                                                            >
                                                            >
                                                            > -------------------------------------------------------------------------
                                                            > Take Surveys. Earn Cash. Influence the Future of IT
                                                            > Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                            > opinions on IT & business topics through brief surveys-and earn cash
                                                            > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                            > _______________________________________________
                                                            > Cheetahtemplate-discuss mailing list
                                                            > Cheetahtemplate-discuss@...
                                                            > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                            >

                                                            -------------------------------------------------------------------------
                                                            Take Surveys. Earn Cash. Influence the Future of IT
                                                            Join SourceForge.net's Techsay panel and you'll get the chance to share your
                                                            opinions on IT & business topics through brief surveys-and earn cash
                                                            http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
                                                            _______________________________________________
                                                            Cheetahtemplate-discuss mailing list
                                                            Cheetahtemplate-discuss@...
                                                            https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                                                          Your message has been successfully submitted and would be delivered to recipients shortly.