memory usage problem
Sorry if this has been asked a lot but I wasn't able to find a clear solution to this yet. I'm working on some scripts to upload base64 encoded files to a web service. Due to initial ballooning memory problems I split out the uploading code into a separate script all by itself just to isolate it. Multiple files need to be uploaded in a row, so the upload script is called for each one, to help free up 100% of the memory used once the individual upload finishes.
The problem is sending a 5MB file, base64 encoded, causes the script to use over 200 MB of memory. A 10 MB file causes it to use at least 375 MB of memory -- maybe more but it keeps crashing my virtual machine only set to 512MB total ;-). At 20MB I start getting "Out of Memory!" errors from SOAP::Lite.
I threw a bunch of lines like this into the script to track the memory usage as it does various things like:
print `pmap $$ | grep mapped`;
memory before reading in file: mapped: 13180K writeable/private: 8156K shared: 0K
memory after reading in file and base64 encoding it: mapped: 51400K writeable/private: 46376K shared: 0K
memory after SOAP request has been built but not ran yet: mapped: 60748K writeable/private: 55724K shared: 0K
memory before SOAP call: mapped: 60748K writeable/private: 55724K shared: 0K
memory after SOAP call: mapped: 210808K writeable/private: 205320K shared: 0K
memory after undef'ing/deleting variables: mapped: 201480K writeable/private: 195976K shared: 0K
So, splitting the uploads out to a script script has helped a little bit at least --- before when I was doing multiple uploads in a loop the memory just kept getting bigger and bigger --- but surely something weird is going on when a 5-6MB SOAP request ramps up memory by 150MB, and adding another 5MB to the request uses an extra 175MB. Is there some option I can set to make SOAP::Lite not burn so much memory internally?