Loading ...
Sorry, an error occurred while loading the content.

memory usage problem

Expand Messages
  • Jeremy
    Hi, Sorry if this has been asked a lot but I wasn t able to find a clear solution to this yet. I m working on some scripts to upload base64 encoded files to a
    Message 1 of 1 , Jul 9 10:36 AM
    • 0 Attachment
      Hi,

      Sorry if this has been asked a lot but I wasn't able to find a clear solution to this yet. I'm working on some scripts to upload base64 encoded files to a web service. Due to initial ballooning memory problems I split out the uploading code into a separate script all by itself just to isolate it. Multiple files need to be uploaded in a row, so the upload script is called for each one, to help free up 100% of the memory used once the individual upload finishes.

      The problem is sending a 5MB file, base64 encoded, causes the script to use over 200 MB of memory. A 10 MB file causes it to use at least 375 MB of memory -- maybe more but it keeps crashing my virtual machine only set to 512MB total ;-). At 20MB I start getting "Out of Memory!" errors from SOAP::Lite.

      I threw a bunch of lines like this into the script to track the memory usage as it does various things like:
      print `pmap $$ | grep mapped`;

      memory before reading in file: mapped: 13180K    writeable/private: 8156K    shared: 0K
      memory after reading in file and base64 encoding it: mapped: 51400K    writeable/private: 46376K    shared: 0K
      memory after SOAP request has been built but not ran yet: mapped: 60748K    writeable/private: 55724K    shared: 0K
      memory before SOAP call: mapped: 60748K    writeable/private: 55724K    shared: 0K
      memory after SOAP call: mapped: 210808K    writeable/private: 205320K    shared: 0K
      memory after undef'ing/deleting variables: mapped: 201480K    writeable/private: 195976K    shared: 0K

      So, splitting the uploads out to a script script has helped a little bit at least --- before when I was doing multiple uploads in a loop the memory just kept getting bigger and bigger --- but surely something weird is going on when a 5-6MB SOAP request ramps up memory by 150MB, and adding another 5MB to the request uses an extra 175MB. Is there some option I can set to make SOAP::Lite not burn so much memory internally?

      Thanks!
    Your message has been successfully submitted and would be delivered to recipients shortly.