Loading ...
Sorry, an error occurred while loading the content.

Re: [Zope-dev] zope.index 3.5.2 broken

Expand Messages
  • Shane Hathaway
    ... Actually, those tests were plain insane and I ve fixed them on the trunk. I intend to make a new zope.index release today. How insane were these tests?
    Message 1 of 18 , Aug 3, 2009
    View Source
    • 0 Attachment
      Marius Gedminas wrote:
      > On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
      >> Hi,
      >>
      >> the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
      >>
      >> File
      >> "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt",
      >> line 143, in textindex.txt
      >> Failed example:
      >> [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result
      >
      > I'm assuming items() returns a plain Python dictionary with string keys.
      >
      > Python's string hashes return different valuesfor half of all the strings
      > on 64-bit machines. This influences the ordering of dictionary keys and
      > some other things too (such as the sequence of random numbers you get if
      > you use a string as the seed).
      >
      > Add a sorted() on both sides and the test should pass.

      Actually, those tests were plain insane and I've fixed them on the
      trunk. I intend to make a new zope.index release today.

      How insane were these tests? Well, the author of the tests noticed that
      the C optimization produces different scores than the Python version,
      and compensated for that in a way that dramatically reduced readability.
      Furthermore, the C code was producing incorrect results due to an
      unsafe optimization that accidentally took 32 bits of a Python float and
      pretended it was an integer. The breakage only became obvious when we
      started taking 64 bits instead, leading to completely different results.

      Shane
      _______________________________________________
      Zope-Dev maillist - Zope-Dev@...
      http://mail.zope.org/mailman/listinfo/zope-dev
      ** No cross posts or HTML encoding! **
      (Related lists -
      http://mail.zope.org/mailman/listinfo/zope-announce
      http://mail.zope.org/mailman/listinfo/zope )
    • Chris McDonough
      ... /me hangs head in shame. - C _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org
      Message 2 of 18 , Aug 3, 2009
      View Source
      • 0 Attachment
        On 8/3/09 1:07 PM, Shane Hathaway wrote:
        > Marius Gedminas wrote:
        >> On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
        >>> Hi,
        >>>
        >>> the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
        >>>
        >>> File
        >>> "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt",
        >>> line 143, in textindex.txt
        >>> Failed example:
        >>> [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result
        >>
        >> I'm assuming items() returns a plain Python dictionary with string keys.
        >>
        >> Python's string hashes return different valuesfor half of all the strings
        >> on 64-bit machines. This influences the ordering of dictionary keys and
        >> some other things too (such as the sequence of random numbers you get if
        >> you use a string as the seed).
        >>
        >> Add a sorted() on both sides and the test should pass.
        >
        > Actually, those tests were plain insane and I've fixed them on the
        > trunk. I intend to make a new zope.index release today.
        >
        > How insane were these tests? Well, the author of the tests noticed that
        > the C optimization produces different scores than the Python version,
        > and compensated for that in a way that dramatically reduced readability.

        /me hangs head in shame.

        - C
        _______________________________________________
        Zope-Dev maillist - Zope-Dev@...
        http://mail.zope.org/mailman/listinfo/zope-dev
        ** No cross posts or HTML encoding! **
        (Related lists -
        http://mail.zope.org/mailman/listinfo/zope-announce
        http://mail.zope.org/mailman/listinfo/zope )
      • Andreas Jung
        ... Is there a buildbot for the zope.* or ZTK packages testing them under Linux 32bit and 64 bit? Andreas _______________________________________________
        Message 3 of 18 , Aug 3, 2009
        View Source
        • 0 Attachment
          On 03.08.09 20:15, Chris McDonough wrote:
          > On 8/3/09 1:07 PM, Shane Hathaway wrote:
          >
          >> Marius Gedminas wrote:
          >>
          >>> On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
          >>>
          >>>> Hi,
          >>>>
          >>>> the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
          >>>>
          >>>> File
          >>>> "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt",
          >>>> line 143, in textindex.txt
          >>>> Failed example:
          >>>> [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result
          >>>>
          >>> I'm assuming items() returns a plain Python dictionary with string keys.
          >>>
          >>> Python's string hashes return different valuesfor half of all the strings
          >>> on 64-bit machines. This influences the ordering of dictionary keys and
          >>> some other things too (such as the sequence of random numbers you get if
          >>> you use a string as the seed).
          >>>
          >>> Add a sorted() on both sides and the test should pass.
          >>>
          >> Actually, those tests were plain insane and I've fixed them on the
          >> trunk. I intend to make a new zope.index release today.
          >>
          >> How insane were these tests? Well, the author of the tests noticed that
          >> the C optimization produces different scores than the Python version,
          >> and compensated for that in a way that dramatically reduced readability.
          >>
          Is there a buildbot for the zope.* or ZTK packages testing them under Linux
          32bit and 64 bit?

          Andreas
        • Shane Hathaway
          ... That was you? Well, it could have been worse. :-) Shane _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org
          Message 4 of 18 , Aug 3, 2009
          View Source
          • 0 Attachment
            Chris McDonough wrote:
            > On 8/3/09 1:07 PM, Shane Hathaway wrote:
            >> How insane were these tests? Well, the author of the tests noticed that
            >> the C optimization produces different scores than the Python version,
            >> and compensated for that in a way that dramatically reduced readability.
            >
            > /me hangs head in shame.

            That was you? Well, it could have been worse. :-)

            Shane
            _______________________________________________
            Zope-Dev maillist - Zope-Dev@...
            http://mail.zope.org/mailman/listinfo/zope-dev
            ** No cross posts or HTML encoding! **
            (Related lists -
            http://mail.zope.org/mailman/listinfo/zope-announce
            http://mail.zope.org/mailman/listinfo/zope )
          • Chris McDonough
            ... I didn t write the original tests but I did paper over the okascore result differences by making the tests compare true or false vs. the expected output.
            Message 5 of 18 , Aug 3, 2009
            View Source
            • 0 Attachment
              On 8/3/09 2:34 PM, Shane Hathaway wrote:
              > Chris McDonough wrote:
              >> On 8/3/09 1:07 PM, Shane Hathaway wrote:
              >>> How insane were these tests? Well, the author of the tests noticed that
              >>> the C optimization produces different scores than the Python version,
              >>> and compensated for that in a way that dramatically reduced readability.
              >>
              >> /me hangs head in shame.
              >
              > That was you? Well, it could have been worse. :-)

              I didn't write the original tests but I did paper over the okascore result
              differences by making the tests compare true or false vs. the expected output.
              Ten lashes.

              - C

              _______________________________________________
              Zope-Dev maillist - Zope-Dev@...
              http://mail.zope.org/mailman/listinfo/zope-dev
              ** No cross posts or HTML encoding! **
              (Related lists -
              http://mail.zope.org/mailman/listinfo/zope-announce
              http://mail.zope.org/mailman/listinfo/zope )
            • Wolfgang Schnerring
              ... AFAICS all three buildbots listed on http://docs.zope.org/zopetoolkit/process/tools.html do. Wolfgang _______________________________________________
              Message 6 of 18 , Aug 3, 2009
              View Source
              • 0 Attachment
                * Andreas Jung <lists@...> [2009-08-03 20:21]:
                > On 03.08.09 20:15, Chris McDonough wrote:
                >> On 8/3/09 1:07 PM, Shane Hathaway wrote:
                >>> Marius Gedminas wrote:
                >>>> On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
                >>>>> the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
                >>>>
                >>>> Python's string hashes return different valuesfor half of all the strings
                >>>> on 64-bit machines. This influences the ordering of dictionary keys and
                >>>> some other things too (such as the sequence of random numbers you get if
                >>>> you use a string as the seed).
                >
                > Is there a buildbot for the zope.* or ZTK packages testing them under Linux
                > 32bit and 64 bit?

                AFAICS all three buildbots listed on
                http://docs.zope.org/zopetoolkit/process/tools.html do.

                Wolfgang

                _______________________________________________
                Zope-Dev maillist - Zope-Dev@...
                http://mail.zope.org/mailman/listinfo/zope-dev
                ** No cross posts or HTML encoding! **
                (Related lists -
                http://mail.zope.org/mailman/listinfo/zope-announce
                http://mail.zope.org/mailman/listinfo/zope )
              • Martijn Faassen
                Hey, Andreas Jung wrote: [snip] ... From what you say, agreed. Regards, Martijn _______________________________________________ Zope-Dev maillist -
                Message 7 of 18 , Aug 6, 2009
                View Source
                • 0 Attachment
                  Hey,

                  Andreas Jung wrote:
                  [snip]

                  > The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt
                  > that such a major change us ok as a bugfix release. I should have become
                  > a new major release.

                  From what you say, agreed.

                  Regards,

                  Martijn

                  _______________________________________________
                  Zope-Dev maillist - Zope-Dev@...
                  http://mail.zope.org/mailman/listinfo/zope-dev
                  ** No cross posts or HTML encoding! **
                  (Related lists -
                  http://mail.zope.org/mailman/listinfo/zope-announce
                  http://mail.zope.org/mailman/listinfo/zope )
                Your message has been successfully submitted and would be delivered to recipients shortly.