Loading ...
Sorry, an error occurred while loading the content.

Re: [PBML] Help!!!!

Expand Messages
  • Sudarshan Raghavan
    ... Instead of storing all the lines into an array and processing them why don t you loop through the filehandle like this. while ( ) { # Do your
    Message 1 of 7 , Aug 5, 2002
    • 0 Attachment
      folkartkits wrote:

      > Hi everyone,
      > I am very new at all this but I need help. This is my problem:
      > Here is the code:
      >
      > open(PRDLOG,$sc_prd_log_path)||die "cant open $sc_prd_log_path";
      >
      > @lines=<PRDLOG>;
      > close(PRDLOG);

      Instead of storing all the lines into an array and processing them why
      don't you loop through the
      filehandle like this.

      while (<PRDLOG>) {
      # Do your processing here, $_ contains the line
      }
      close (PRDLOG);

      Slurping the entire file into an array at once is not a good idea,
      especially if it is a huge file

      >
      >
      >
      > foreach $prd(@lines)
      > {
      > chomp($prd);
      > ($phrase,$number)=split(/\|/,$prd);

      Taking the first line of your sample log as an example, $phrase will
      contain 945 and $number
      will contain 'Delta ....'. Is this what you want. The split here will
      return a 3 element array and
      you are ignoring the 3'rd element, in this case 1.

      >
      >
      > $phrase_array{$phrase}=$number;
      > }
      >
      > $phrase_array{$product} += $quantity;

      You are performing a numeric operation on a string.

      >
      > {@keys=sort {$phrase_array{$a} <=> $phrase_array{$b}} keys(%
      > phrase_array)}

      You are sorting strings using a numeric operator <=>. Are you sure this
      works, if so what sort
      of output does it give.

      >
      >
      > open(PRDLOG,">$sc_prd_log_path");
      >
      > foreach $key(@keys)
      > {
      >
      > print PRDLOG qq~$key|$phrase_array{$key}\n~;
      > }
      > close(PRDLOG);
      > --------------------------------------------------------------------
      > SAMPLE PRDLOG
      >
      > 945|Delta Ceramcoat 2oz Toffee Brown D2086|1
      > 946|Winter Garden Letter Box PPDA-WGLB|2
      > --------------------------------------------------------------------
      > The current code works to write $phrase,$number to the log , but
      > misses the beginning number field. What needs to be added or changed
      > to allow this to happen??? Note numbered field will stay the same
      >
      > Thanks
      >
      > ps $product and $quantity are from other parts of the script.
      >
      >


      [Non-text portions of this message have been removed]
    • Dragon
      ... Why not? Memory s cheap, and it s considerably faster to process things in memory than it is to process them from a disk... especially if it s a huge
      Message 2 of 7 , Aug 6, 2002
      • 0 Attachment
        --- Sudarshan Raghavan <sraghav@...> wrote:
        > Instead of storing all the lines into an array and
        > processing them why don't you loop through the
        > filehandle like this.
        >
        > while (<PRDLOG>) {
        > # Do your processing here, $_ contains the line
        > }
        > close (PRDLOG);
        >
        > Slurping the entire file into an array at once is
        > not a good idea, especially if it is a huge file


        Why not? Memory's cheap, and it's considerably faster
        to process things in memory than it is to process them
        from a disk... especially if it's a huge file.

        And yes, I realize that you would have to have enough
        available memory to hold the file. But so long as you
        stay within that constraint, why wouldn't you want to
        do so?


        __________________________________________________
        Do You Yahoo!?
        Yahoo! Health - Feel better, live better
        http://health.yahoo.com
      • Sudarshan Raghavan
        ... When you make a IO request on a file the OS reads the entire block into a in memory buffer and maybe also the next block. The next read request say is
        Message 3 of 7 , Aug 6, 2002
        • 0 Attachment
          Dragon wrote:

          > --- Sudarshan Raghavan <sraghav@...> wrote:
          > > Instead of storing all the lines into an array and
          > > processing them why don't you loop through the
          > > filehandle like this.
          > >
          > > while (<PRDLOG>) {
          > > # Do your processing here, $_ contains the line
          > > }
          > > close (PRDLOG);
          > >
          > > Slurping the entire file into an array at once is
          > > not a good idea, especially if it is a huge file
          >
          >
          > Why not? Memory's cheap, and it's considerably faster
          > to process things in memory than it is to process them
          > from a disk... especially if it's a huge file.

          When you make a IO request on a file the OS reads the entire block
          into a in memory buffer and maybe also the next block.
          The next read request say is within the same block the contents are
          returned
          from the in memory buffer, the OS does not go back to the disk.
          The internal OS buffering is designed to minimise disk I/O.

          By slurping in the entire file you are duplicating the work that the OS
          is already doing for you. Why would you want to do that?

          The number of disk I/O's made in both these cases is the same, the
          memory
          consumption differs (in the case of huge files the difference is also
          huge)
          What makes you think that slurping in huge files improves effeciency?

          >
          >
          > And yes, I realize that you would have to have enough
          > available memory to hold the file. But so long as you
          > stay within that constraint, why wouldn't you want to
          > do so?

          How do you decide that when your script goes to a customer?

          >
          >
          >
          > __________________________________________________


          [Non-text portions of this message have been removed]
        • Octavian Rasnita
          I ve made a test and if I read all the file at once, the script works faster than reading and checking line by line. But if the file is too big, the only
          Message 4 of 7 , Aug 6, 2002
          • 0 Attachment
            I've made a test and if I read all the file at once, the script works faster
            than reading and checking line by line.
            But if the file is too big, the only solution is to read it as you
            described.
            The problem sometimes, is that if you need to sort a very big text file, you
            can't do it reading it line by line.

            Teddy Center: http://teddy.fcc.ro/
            Mail: orasnita@...

            ----- Original Message -----
            From: "Sudarshan Raghavan" <sraghav@...>
            To: <perl-beginner@yahoogroups.com>
            Sent: Tuesday, August 06, 2002 4:36 PM
            Subject: Re: [PBML] Help!!!!


            > Dragon wrote:
            >
            > > --- Sudarshan Raghavan <sraghav@...> wrote:
            > > > Instead of storing all the lines into an array and
            > > > processing them why don't you loop through the
            > > > filehandle like this.
            > > >
            > > > while (<PRDLOG>) {
            > > > # Do your processing here, $_ contains the line
            > > > }
            > > > close (PRDLOG);
            > > >
            > > > Slurping the entire file into an array at once is
            > > > not a good idea, especially if it is a huge file
            > >
            > >
            > > Why not? Memory's cheap, and it's considerably faster
            > > to process things in memory than it is to process them
            > > from a disk... especially if it's a huge file.
            >
            > When you make a IO request on a file the OS reads the entire block
            > into a in memory buffer and maybe also the next block.
            > The next read request say is within the same block the contents are
            > returned
            > from the in memory buffer, the OS does not go back to the disk.
            > The internal OS buffering is designed to minimise disk I/O.
            >
            > By slurping in the entire file you are duplicating the work that the OS
            > is already doing for you. Why would you want to do that?
            >
            > The number of disk I/O's made in both these cases is the same, the
            > memory
            > consumption differs (in the case of huge files the difference is also
            > huge)
            > What makes you think that slurping in huge files improves effeciency?
            >
            > >
            > >
            > > And yes, I realize that you would have to have enough
            > > available memory to hold the file. But so long as you
            > > stay within that constraint, why wouldn't you want to
            > > do so?
            >
            > How do you decide that when your script goes to a customer?
            >
            > >
            > >
            > >
            > > __________________________________________________
            >
            >
            > [Non-text portions of this message have been removed]
            >
            >
            >
            > Unsubscribing info is here:
            http://help.yahoo.com/help/us/groups/groups-32.html
            >
            > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
            >
            >
            >
          • Dragon
            ... I never said that slurping in huge files improves efficiency. What I said was that it was faster to work with data in memory than to do work with data on
            Message 5 of 7 , Aug 6, 2002
            • 0 Attachment
              --- Sudarshan Raghavan <sraghav@...> wrote:
              > Dragon wrote:
              >
              > > --- Sudarshan Raghavan <sraghav@...>
              > wrote:
              > The number of disk I/O's made in both these cases is
              > the same, the memory
              > consumption differs (in the case of huge files the
              > difference is also huge)
              > What makes you think that slurping in huge files
              > improves effeciency?


              I never said that slurping in huge files improves
              efficiency. What I said was that it was faster to
              work with data in memory than to do work with data on
              a disk (i.e. to do disk I/O's).


              > > And yes, I realize that you would have to have
              > > enough
              > > available memory to hold the file. But so long as
              > > you
              > > stay within that constraint, why wouldn't you want
              > > to do so?
              >
              > How do you decide that when your script goes to a
              > customer?


              Maybe by doing something radical like asking the
              customer for his requirements? Or if it's not
              something being developed for a specific customer, by
              indicating the memory requirements / max file size in
              the script documentation?


              __________________________________________________
              Do You Yahoo!?
              Yahoo! Health - Feel better, live better
              http://health.yahoo.com
            • Sudarshan Raghavan
              ... I think you are missing the point. The block that contains the data you are reading is brought into memory. In effect you are doing a memory I/O not always
              Message 6 of 7 , Aug 6, 2002
              • 0 Attachment
                Dragon wrote:

                > --- Sudarshan Raghavan <sraghav@...> wrote:
                > > Dragon wrote:
                > >
                > > > --- Sudarshan Raghavan <sraghav@...>
                > > wrote:
                > > The number of disk I/O's made in both these cases is
                > > the same, the memory
                > > consumption differs (in the case of huge files the
                > > difference is also huge)
                > > What makes you think that slurping in huge files
                > > improves effeciency?
                >
                >
                > I never said that slurping in huge files improves
                > efficiency. What I said was that it was faster to
                > work with data in memory than to do work with data on
                > a disk (i.e. to do disk I/O's).

                I think you are missing the point. The block that contains the
                data you are reading is brought into memory. In effect you
                are doing a memory I/O not always a disk I/O.


                [Non-text portions of this message have been removed]
              Your message has been successfully submitted and would be delivered to recipients shortly.