Loading ...
Sorry, an error occurred while loading the content.

Minimising the memory used by arrays.

Expand Messages
  • kenshail2003
    I am populating an array from disk. Each value is a single byte. A file of around 4Mb appears to gobble about 60Mb of RAM. I have to keep the whole array in
    Message 1 of 1 , Aug 2, 2006
    • 0 Attachment
      I am populating an array from disk. Each value is a single byte.
      A file of around 4Mb appears to gobble about 60Mb of RAM.
      I have to keep the whole array in RAM for sequential and random
      access during later processing.
      This may not sound serious but much larger files are possible.
      Using Activestate 5.8.4 on a 686.
      Does anyone know of a method of minimising an array's RAM useage
      without having to cache the aray/file in/out during processing.

      Here is a cut-down version of the sub I use to load the file.
      I show a Tk progress-bar while loading. That is why I have used read
      to load the file so that the process can break-off to refresh the bar.

      # Get a binary file in to an array with a progress bar
      # $_[0] is the file path,
      # $_[1] is a reference to the array that the binary should be loaded
      to
      # $_[2] is the text to be dispayed in the progress bar,
      # $_[3] is the block size in bytes
      # The larger the block size the quicker it will load, but the steps
      of the progress bar will be coarser
      # returns 1 if file was opened OK

      sub getBinaryFileWithProgressBar
      {
      my $temp;
      my $blockCount=0; # The actual count of save blocks that gets used
      to drive the progress bar.
      my $fileSize;
      my $ok=0;
      my @buf;

      if(open INPUT, "<".$_[0])
      {
      binmode(INPUT);
      $fileSize=(stat(INPUT))[7];
      &MakeProgressPopUp($_[2]); # this just instantates the progress bar
      @{$_[1]}=();
      while (read(INPUT, $temp,$_[3]) > 0)
      {
      @buf=split(//,$temp);
      push @{$_[1]},@buf;;
      # do the progress bar
      $blockCount+=$_[3];
      $progress=$barWidth * $blockCount/$fileSize; # some global values
      $prog->update(); # Refresh the progess-bar object
      }
      close INPUT;
      $tl->withdraw; # close the progress bar parent dialog
      $ok=1;
      }
      return $ok;
      }


      The subsequent processing of the array is too complex to include here
      but breaks down to simple assignments and tests. Since the array is
      fully populated there would, I assume be no advantage to loading the
      file in to a hash.
    Your message has been successfully submitted and would be delivered to recipients shortly.