Java Object Serialization memory problems
- Hi Craig,
I'm the author of JSX, I've just approved your memship to the mailing list.
I'm answering your comment directly:
> I'm writing a distributed artificial life program. I have been having outof memory problems during serialization, and wondered whether XML might
provide a less memory intense solution.
Probably, JSX is going to be the same, as it uses the same mechanisms.
However, with XML you can see what is happening, and hopefully detect where
all that memory is going. You can probably write writeObject() and
readObject() methods in the problematic class to fix it - or use a memento.
For example, the java.util.HashMap has a huge array for the hashtable, much
of which is empty - so that table is transient, and only the actual entries
You can test it out JSX very easily - download, add to classpath, change
calls to *constructors*:
new ObjectOutputWriter(xxx) to new JSX.ObjectWriter(xxx)
new ObjectInputReader(xxx) to new JSX.ObjectReader(xxx)
You *only* need to change the constructor (the class of the variable holding
the object can stay the same etc).
See http://www.jsx.org/download.html for all the details.
I will look into it. I am an academic/hobbyist programmer, with too little time to
research the matter properly, but it seems to me that too little attention has been
paid to memory consumption in the default java serialization methods. I have made
some progress by serializing some large child components within memory and then
compressing them (also in memory) before finally serializing the parent object but
that method is slow. It would be nice to serialize and write small sections of my large
objects to the hard disc one section at a time, instead of facing a huge memory
blowout all at once. Maybe JSX will enable me to do some sort of sequential
serialization; I'm not sure. I could, of course, write some of my sub-objects to
separate files, one at a time, and then recombine them manually but I was not
wanting to write a lot of my own code to do it, and wanted everything saved in a
If I find a nice solution I will let you know.
PS. To give you an idea of the scale of the problem, the final file (even with some
compression) is about 200 megabytes on the hard disc.