Loading ...
Sorry, an error occurred while loading the content.

Re: Problem with perl and large files

Expand Messages
  • Brian
    Sorry, by user space I mean it s at level above syscall. I think I ve found the problem, sources/perl/5.10/config.sh-cs08q1armel got transformed into
    Message 1 of 6 , Dec 1, 2009
    • 0 Attachment
      Sorry, by "user space" I mean it's at level above syscall.

      I think I've found the problem, sources/perl/5.10/config.sh-cs08q1armel got transformed into builds/perl/config.sh by
      Cross/generate_config_sh. And the transformation will use CFLAGS if it is set. For some reason CFLAGS is set, so ccflags supplied in config.sh-cs08q1armel never got to builds/perl/config.sh. This results that ccflags does not contain '-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', these options never got applied during build.

      A patch is on the way (for all optware platforms).

      -Brian

      --- In nslu2-linux@yahoogroups.com, "linuxnutzer1" <joerg_kuehne@...> wrote:
      >
      >
      >
      >
      >
      >
      > Hi Brian,
      >
      > thank you for the very fast answer!
      > Unfortunately it seems that the issue remains, so can you explain what you mean with "so the problem is in user space"?
      > Is it a perl bug or a glibc issue from the qnap firmware?
      > I found an old discussion by googling this issue. See
      > http://www.justskins.com/forums/about-stat-problem-49666.html
      > A "perl -V" did not show any -D_FILE_OFFSET_BITS=64 string on my QNAP, but "uselargefiles=define". On all my other linux boxes this command shows -D_FILE_OFFSET_BITS=64 as ccflags. But if stat64() is called from perl binary than it seems that this option have applied.
      > Very mysterious.
      >
      > Thank you in advance
      >
      > Jörg
      >
      > --- In nslu2-linux@yahoogroups.com, "Brian" <b88zhou@> wrote:
      > >
      > > Doing a strace on a file > 2GB clearly shows that stat64 call has succeeded, so the problem is in user space.
      > >
      > > stat64("big-file", {st_mode=S_IFREG|0777, st_size=2200331748, ...}) = 0
      > >
      > > -Brian
      > >
      > > --- In nslu2-linux@yahoogroups.com, "joerg_kuehne" <joerg_kuehne@> wrote:
      > > >
      > > > Hi,
      > > >
      > > > I hope, that anyone in this group can give me a hint about the maintainer of the perl ipkg-package of the cs08q1armel branch.
      > > > I found an issue with the pert stat() function, that fails for files larger than 2GByte on my QNAP TS119. This function is used from the rsnapshot-diff script to get all informations about a certain file for comparing different backup snapshots. It seems that perl is compiled without 64Bit support for file sizes, with 32Bit one can only store size numbers up to 2GByte. Here a little test script to verify this issue.
      > > >
      > > > test.pl
      > > > --- cut here ---
      > > > #! /opt/bin/perl
      > > >
      > > > @mystat = stat($ARGV[0]);
      > > > print "$ARGV[0]: file size: $mystat[7]\n";
      > > > --- cut here ---
      > > >
      > > > Run test.pl FILE
      > > >
      > > > If file is smaller than 2GByte it will display the right file size, if it is larger then the file size will be empty caused by the failed stat().
      > > > Can anyone tell me why perl is compiled without the
      > > > -D_FILE_OFFSET_BITS=64 option? Is it a restriction of the ARM architecture or glibc? Or has the usage of this option unwanted side effects or was it simply forgotten?
      > > > Thank you in advance
      > > >
      > > > Jörg
      > > >
      > >
      >
    • linuxnutzer1
      Hi Brian, excellent work!!! Thank you very, very much. I upgraded the perl package and now my test script and the rsnapshot-diff script are working perfectly.
      Message 2 of 6 , Dec 1, 2009
      • 0 Attachment
        Hi Brian,

        excellent work!!! Thank you very, very much. I upgraded the perl package and now my test script and the rsnapshot-diff script are working perfectly.

        Best wishes,

        Jörg

        --- In nslu2-linux@yahoogroups.com, "Brian" <b88zhou@...> wrote:
        >
        > Sorry, by "user space" I mean it's at level above syscall.
        >
        > I think I've found the problem, sources/perl/5.10/config.sh-cs08q1armel got transformed into builds/perl/config.sh by
        > Cross/generate_config_sh. And the transformation will use CFLAGS if it is set. For some reason CFLAGS is set, so ccflags supplied in config.sh-cs08q1armel never got to builds/perl/config.sh. This results that ccflags does not contain '-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', these options never got applied during build.
        >
        > A patch is on the way (for all optware platforms).
        >
        > -Brian
        >
        > --- In nslu2-linux@yahoogroups.com, "linuxnutzer1" <joerg_kuehne@> wrote:
        > >
        > >
        > >
        > >
        > >
        > >
        > > Hi Brian,
        > >
        > > thank you for the very fast answer!
        > > Unfortunately it seems that the issue remains, so can you explain what you mean with "so the problem is in user space"?
        > > Is it a perl bug or a glibc issue from the qnap firmware?
        > > I found an old discussion by googling this issue. See
        > > http://www.justskins.com/forums/about-stat-problem-49666.html
        > > A "perl -V" did not show any -D_FILE_OFFSET_BITS=64 string on my QNAP, but "uselargefiles=define". On all my other linux boxes this command shows -D_FILE_OFFSET_BITS=64 as ccflags. But if stat64() is called from perl binary than it seems that this option have applied.
        > > Very mysterious.
        > >
        > > Thank you in advance
        > >
        > > Jörg
        > >
        > > --- In nslu2-linux@yahoogroups.com, "Brian" <b88zhou@> wrote:
        > > >
        > > > Doing a strace on a file > 2GB clearly shows that stat64 call has succeeded, so the problem is in user space.
        > > >
        > > > stat64("big-file", {st_mode=S_IFREG|0777, st_size=2200331748, ...}) = 0
        > > >
        > > > -Brian
        > > >
        > > > --- In nslu2-linux@yahoogroups.com, "joerg_kuehne" <joerg_kuehne@> wrote:
        > > > >
        > > > > Hi,
        > > > >
        > > > > I hope, that anyone in this group can give me a hint about the maintainer of the perl ipkg-package of the cs08q1armel branch.
        > > > > I found an issue with the pert stat() function, that fails for files larger than 2GByte on my QNAP TS119. This function is used from the rsnapshot-diff script to get all informations about a certain file for comparing different backup snapshots. It seems that perl is compiled without 64Bit support for file sizes, with 32Bit one can only store size numbers up to 2GByte. Here a little test script to verify this issue.
        > > > >
        > > > > test.pl
        > > > > --- cut here ---
        > > > > #! /opt/bin/perl
        > > > >
        > > > > @mystat = stat($ARGV[0]);
        > > > > print "$ARGV[0]: file size: $mystat[7]\n";
        > > > > --- cut here ---
        > > > >
        > > > > Run test.pl FILE
        > > > >
        > > > > If file is smaller than 2GByte it will display the right file size, if it is larger then the file size will be empty caused by the failed stat().
        > > > > Can anyone tell me why perl is compiled without the
        > > > > -D_FILE_OFFSET_BITS=64 option? Is it a restriction of the ARM architecture or glibc? Or has the usage of this option unwanted side effects or was it simply forgotten?
        > > > > Thank you in advance
        > > > >
        > > > > Jörg
        > > > >
        > > >
        > >
        >
      Your message has been successfully submitted and would be delivered to recipients shortly.