You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by "Peter Molettiere (JIRA)" <ax...@ws.apache.org> on 2005/01/19 01:05:17 UTC
[jira] Created: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Excessive Memory Use During Serialization/Deserialization
---------------------------------------------------------
Key: AXIS-1771
URL: http://issues.apache.org/jira/browse/AXIS-1771
Project: Axis
Type: Bug
Components: Basic Architecture
Versions: 1.2RC2
Environment: JDK 1.4.2, Mac OS X, Linux, Windows
Reporter: Peter Molettiere
Attachments: memory-use-test.tgz
Axis uses pathological amounts of memory during the serialization/deserialization process.
We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
The memory used does seem to be released once serialization is done, so this isn't a leak.
Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
As provided, it generates the following output:
Buildfile: build.xml
build:
[javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
run:
[java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
[java] Created tree with 5 levels and 3 children at each level
[java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
[java] GC freed 13 MBytes
[java] Created tree with 5 levels and 4 children at each level
[java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
[java] GC freed 71 MBytes
[java] Created tree with 5 levels and 5 children at each level
[java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
[java] GC freed 287 MBytes
[java] Created tree with 5 levels and 6 children at each level
[java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
[java] GC freed 675 MBytes
[java] Created tree with 5 levels and 7 children at each level
[java] Out of Memory serializing 28 MBytes tree.
[java] Java Result: 1
BUILD SUCCESSFUL
Total time: 2 minutes 51 seconds
Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_58098 ]
Peter Molettiere commented on AXIS-1771:
----------------------------------------
Hmm... I've spent the morning looking at your revised test, Venkat, but it doesn't seem to hold up.
First, let me say that yes, your test as written, for the one small message size that you test, does produce the number you're claiming. But for larger message sizes, I'm seeing deserialization memory usage ratios as high as 42 to 1.
However, when I pull your reporting and garbage collection changes back into my test, and run over a range of message sizes, your ratio doesn't remain constant, even for the same message size that you test.
In answer to your question about what ratio is good enough, I'll say this. If you could in fact get the ratio down to 15 to 1 it would be an improvement over the current situation, and would be much appreciated, but O(15n) performance is not ideal.
Here's what I'm seeing, for a few cases. You can run my code (will attach in a minute) to see all cases.
[java] Created tree with 5 levels and 3 children at each level
[java] Total memory usage: Created tree = 912 KBytes (934232)
[java] Increase in allocation since last check = 229 KBytes (234736)
[java] Total memory usage: Serialized tree = 4 MBytes (4958272)
[java] Increase in allocation since last check = 3 MBytes (4024040)
[java] Total memory usage: Deserialized tree. = 13 MBytes (13692168)
[java] Increase in allocation since last check = 8 MBytes (8733896)
The above test represents a ratio of 13 to serialize, and 35 to deserialize.
[java] Created tree with 5 levels and 4 children at each level
[java] Total memory usage: Created tree = 1 MBytes (1908112)
[java] Increase in allocation since last check = 920 KBytes (942464)
[java] Total memory usage: Serialized tree = 27 MBytes (28685800)
[java] Increase in allocation since last check = 25 MBytes (26777688)
[java] Total memory usage: Deserialized tree. = 66 MBytes (69915592)
[java] Increase in allocation since last check = 39 MBytes (41229792)
The above test represents a ratio of 27 to serialize, and 42 to deserialize.
[java] Created tree with 5 levels and 5 children at each level
[java] Total memory usage: Created tree = 5 MBytes (6065264)
[java] Increase in allocation since last check = 4 MBytes (4247184)
[java] Total memory usage: Serialized tree = 95 MBytes (100226664)
[java] Increase in allocation since last check = 89 MBytes (94161400)
[java] Total memory usage: Deserialized tree. = 246 MBytes (258236984)
[java] Increase in allocation since last check = 150 MBytes (158010320)
The above test represents a ratio of 22 to serialize, and 38 to deserialize.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "tony salerno (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420930 ]
tony salerno commented on AXIS-1771:
------------------------------------
Thanks for the update Peter. I'm afraid we are not in the 90% use case, but I'm just beginning to delve into this issue. I think we are tied Axis right now.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Davanum Srinivas <da...@gmail.com>.
Here's what i get....Also, I cleaned up my old attachments to the bug
report and uploaded the latest (just to be on the safe side). Here's
the URL - http://issues.apache.org/jira/secure/attachment/19322/MemoryTesterClient.java
-- dims
C:\junk>java com.truereq.test.axis.client.MemoryTesterClient
Total memory usage: Begin test: 3, 3 = 965 KBytes (988344)
Increase in allocation since last check = 965 KBytes (988344)
Created tree with 3 levels and 3 children at each level, total of 13
Total memory usage: Created tree = 912 KBytes (934256)
Increase in allocation since last check = -52 KBytes (-54088)
Total memory usage: Axis Serialization Start = 913 KBytes (935344)
Increase in allocation since last check = 1 KBytes (1088)
Total memory usage: Axis Serialization End - Deserialization Start = 1
MBytes (1466096)
Increase in allocation since last check = 518 KBytes (530752)
Total memory usage: Axis Deserialization End = 2 MBytes (2469576)
Increase in allocation since last check = 979 KBytes (1003480)
Total memory usage: Axis Serialization/Deserialization finished = 2
MBytes (2470664)
Increase in allocation since last check = 1 KBytes (1088)
Total memory usage: Axis Serialization/Deserialization cleanup = 2
MBytes (2149432)
Increase in allocation since last check = -313 KBytes (-321232)
Total memory usage: ObjectOutputStream Serialization. = 2 MBytes (2385776)
Increase in allocation since last check = 230 KBytes (236344)
Total memory usage: Deserialization cleanup. = 2 MBytes (2172608)
Increase in allocation since last check = -208 KBytes (-213168)
===========================================================
Created tree with 3 levels and 3 children at each level, total of 13
running axis : .621ms
running stream: .60ms
Streaming is 561ms faster than axis, or 10 times faster.
===========================================================
Total memory usage: Begin test: 4, 3 = 2 MBytes (2141440)
Increase in allocation since last check = -30 KBytes (-31168)
Created tree with 4 levels and 3 children at each level, total of 40
Total memory usage: Created tree = 2 MBytes (2219624)
Increase in allocation since last check = 76 KBytes (78184)
Total memory usage: Axis Serialization Start = 2 MBytes (2220712)
Increase in allocation since last check = 1 KBytes (1088)
Total memory usage: Axis Serialization End - Deserialization Start = 2
MBytes (2798432)
Increase in allocation since last check = 564 KBytes (577720)
Total memory usage: Axis Deserialization End = 5 MBytes (5348680)
Increase in allocation since last check = 2 MBytes (2550248)
Total memory usage: Axis Serialization/Deserialization finished = 5
MBytes (5349768)
Increase in allocation since last check = 1 KBytes (1088)
Total memory usage: Axis Serialization/Deserialization cleanup = 4
MBytes (4655280)
Increase in allocation since last check = -678 KBytes (-694488)
Total memory usage: ObjectOutputStream Serialization. = 4 MBytes (5071512)
Increase in allocation since last check = 406 KBytes (416232)
Total memory usage: Deserialization cleanup. = 4 MBytes (4673928)
Increase in allocation since last check = -388 KBytes (-397584)
===========================================================
Created tree with 4 levels and 3 children at each level, total of 40
running axis : .1061ms
running stream: .60ms
Streaming is 1001ms faster than axis, or 17 times faster.
===========================================================
On Thu, 17 Mar 2005 12:07:24 -0800, Peter Molettiere
<pi...@axonstudios.net> wrote:
>
> On Mar 17, 2005, at 11:41 AM, Davanum Srinivas wrote:
> > Can you please post the full results?
>
> Sure, here you go:
>
> > Created tree with 5 levels and 3 children at each level
> >
> > Total memory usage: Created tree = 896 KBytes (917920)
> > Increase in allocation since last check = 896 KBytes (917920)
> >
> > Total memory usage: Serialized tree = 5 MBytes (5729784)
> > Increase in allocation since last check = 4 MBytes (4811864)
> >
> > Total memory usage: Deserialized tree. = 18 MBytes (19443224)
> > Increase in allocation since last check = 13 MBytes (13713440)
> >
> >
> > ===========================================================
> >
> >
> > Created tree with 5 levels and 4 children at each level
> >
> > Total memory usage: Created tree = 1 MBytes (1840416)
> > Increase in allocation since last check = -16 MBytes (-17602808)
> > - Out of Memory serializing.
> >
> > Process finished with exit code 1
>
> --
> Peter Molettiere
> Senior Engineer
> Truereq, Inc.
> http://www.truereq.com/
>
>
--
Davanum Srinivas - http://webservices.apache.org/~dims/
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
On Mar 17, 2005, at 11:41 AM, Davanum Srinivas wrote:
> Can you please post the full results?
Sure, here you go:
> Created tree with 5 levels and 3 children at each level
>
> Total memory usage: Created tree = 896 KBytes (917920)
> Increase in allocation since last check = 896 KBytes (917920)
>
> Total memory usage: Serialized tree = 5 MBytes (5729784)
> Increase in allocation since last check = 4 MBytes (4811864)
>
> Total memory usage: Deserialized tree. = 18 MBytes (19443224)
> Increase in allocation since last check = 13 MBytes (13713440)
>
>
> ===========================================================
>
>
> Created tree with 5 levels and 4 children at each level
>
> Total memory usage: Created tree = 1 MBytes (1840416)
> Increase in allocation since last check = -16 MBytes (-17602808)
> - Out of Memory serializing.
>
> Process finished with exit code 1
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Davanum Srinivas <da...@gmail.com>.
Can you please post the full results?
-- dims
On Thu, 17 Mar 2005 11:09:24 -0800, Peter Molettiere
<pi...@axonstudios.net> wrote:
>
> On Mar 17, 2005, at 10:22 AM, Davanum Srinivas wrote:
> > Are u using this one with LATEST CVS of axis?
> > http://issues.apache.org/jira/secure/attachment/18746/
> > MemoryTesterClient.java
>
> No, I wasn't. I was using the other version you put up, since 18746 was
> causing an NPE. I just figured the other one was a fix to the NPE. I
> just checked out the LATEST CVS, and used the class you linked above to
> run the test, and got the same results.
>
> --
> Peter Molettiere
> Senior Engineer
> Truereq, Inc.
> http://www.truereq.com/
>
>
--
Davanum Srinivas - http://webservices.apache.org/~dims/
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
On Mar 17, 2005, at 10:22 AM, Davanum Srinivas wrote:
> Are u using this one with LATEST CVS of axis?
> http://issues.apache.org/jira/secure/attachment/18746/
> MemoryTesterClient.java
No, I wasn't. I was using the other version you put up, since 18746 was
causing an NPE. I just figured the other one was a fix to the NPE. I
just checked out the LATEST CVS, and used the class you linked above to
run the test, and got the same results.
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Davanum Srinivas <da...@gmail.com>.
Are u using this one with LATEST CVS of axis?
http://issues.apache.org/jira/secure/attachment/18746/MemoryTesterClient.java
thanks,
dims
On Thu, 17 Mar 2005 10:16:30 -0800, Peter Molettiere
<pi...@axonstudios.net> wrote:
>
> On Mar 17, 2005, at 5:07 AM, Davanum Srinivas (JIRA) wrote:
> > Try this out...tweaked options (including the streaming). Axis takes 8
> > times slower consistently as compared to the ObjectOutputStream
> > option. Memory has reduced too. Let me know if you see something
> > screwy. I will try a bit more and report later. BUT am eager to hear
> > from you.
>
> I downloaded and ran your updated code, with a 32M max heap. It looks
> like it still takes more than 20M to serialize and deserialize a tree
> that takes less than a single megabyte in memory. It ran out of memory
> trying to serialize a 1844656 byte tree. Am I missing something here,
> or does this reflect the improvement you think your tweaks made?
>
> > Created tree with 5 levels and 3 children at each level
> >
> > Total memory usage: Created tree = 905 KBytes (926744)
> > Increase in allocation since last check = 905 KBytes (926744)
> >
> > Total memory usage: Serialized tree = 5 MBytes (5737504)
> > Increase in allocation since last check = 4 MBytes (4810760)
> >
> > Total memory usage: Deserialized tree. = 18 MBytes (19448264)
> > Increase in allocation since last check = 13 MBytes (13710760)
> >
> >
> > ===========================================================
> >
> >
> > Created tree with 5 levels and 4 children at each level
> >
> > Total memory usage: Created tree = 1 MBytes (1844656)
> > Increase in allocation since last check = -16 MBytes (-17603608)
> > - Out of Memory serializing.
> >
> > Process finished with exit code 1
>
> --
> Peter Molettiere
> Senior Engineer
> Truereq, Inc.
> http://www.truereq.com/
>
>
--
Davanum Srinivas - http://webservices.apache.org/~dims/
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
On Mar 17, 2005, at 5:07 AM, Davanum Srinivas (JIRA) wrote:
> Try this out...tweaked options (including the streaming). Axis takes 8
> times slower consistently as compared to the ObjectOutputStream
> option. Memory has reduced too. Let me know if you see something
> screwy. I will try a bit more and report later. BUT am eager to hear
> from you.
I downloaded and ran your updated code, with a 32M max heap. It looks
like it still takes more than 20M to serialize and deserialize a tree
that takes less than a single megabyte in memory. It ran out of memory
trying to serialize a 1844656 byte tree. Am I missing something here,
or does this reflect the improvement you think your tweaks made?
> Created tree with 5 levels and 3 children at each level
>
> Total memory usage: Created tree = 905 KBytes (926744)
> Increase in allocation since last check = 905 KBytes (926744)
>
> Total memory usage: Serialized tree = 5 MBytes (5737504)
> Increase in allocation since last check = 4 MBytes (4810760)
>
> Total memory usage: Deserialized tree. = 18 MBytes (19448264)
> Increase in allocation since last check = 13 MBytes (13710760)
>
>
> ===========================================================
>
>
> Created tree with 5 levels and 4 children at each level
>
> Total memory usage: Created tree = 1 MBytes (1844656)
> Increase in allocation since last check = -16 MBytes (-17603608)
> - Out of Memory serializing.
>
> Process finished with exit code 1
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Davanum Srinivas updated AXIS-1771:
-----------------------------------
Attachment: MemoryTesterClient.java
Peter,
Try this out...tweaked options (including the streaming). Axis takes 8 times slower consistently as compared to the ObjectOutputStream option. Memory has reduced too. Let me know if you see something screwy. I will try a bit more and report later. BUT am eager to hear from you.
thanks,
dims
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Venkat Reddy (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_58205 ]
Venkat Reddy commented on AXIS-1771:
------------------------------------
Peter,
The ratio i reported was obtained by using the XML SOAP message (not the tree size) for comparison with the increase in memroy allocation during handling that SOAP message (deser). I did this to seperate the inefficiencies in the areas of SOAP message creation and its handling, so that we can set seperate goals for these areas.
The ratio was almost consistent for a tree range of (2.7) through (5,4). Again this is using the SOAP Message size as the yard stick.
Venkat
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Venkat Reddy (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57830 ]
Venkat Reddy commented on AXIS-1771:
------------------------------------
Peter,
Here are more observations from me.
1. The tree depth reported by the MemoryTestClient is slightly incorrect. createTree(5,3) actually constructs a tree of 6 levels with 3 children at each level.
2. Since the treesize reported is the memory footprint of the binary data, when we convert it to SOAP message, I'm not sure how many times the size will increase. Axis is taking about 10-12 times (with multiref=false), which I think is not bad.
3. When multi-ref is true, however, we might need some fixing, because of the following -
3.1 The number of elements in the generated SOAPMessage is 4 times of that when multi-ref is false. For example, a (2,7) tree will have 400 elements (343+49+7+1) with no multiref, and the same will have 1600 elements with multi-ref is true. Can't say this is an issue, but it will surely increase memory consumption.
3.2 The recursion I have reported yesterday happens only when multiref is true. The recursion call depth is about 2*d + 3 where d is real depth of tree. So, createTree (5,3) results in recursive calls with depth of 15 (because d is actually 6). This may be linked to the observation 3.1.
3.3 Also attached is a snapshot of recursive calls as an image (recursion.png), and SOAPMessages generated with tree of (2,7).
So i think we may have some fixing to do only when multi-ref'ing is true. May be i'm wrong, becuase i really don't know a thing about multi-ref'ing code, except through these test results :)
Venkat
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Davanum Srinivas updated AXIS-1771:
-----------------------------------
Attachment: MemoryTesterClient.java
Please try this one!!!
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57767 ]
nishant kumar commented on AXIS-1771:
-------------------------------------
a small query again.
in the registerElementByID method of DeserializationContext one created LocalIDResolver which maintains a map of id to object mappings.
then we call registerResolverForID which stores the id, LocalIDResolver pair.
note that registerResolverForID is called from registerElementByID only. so it can be made private.
the LocalIDResolver instance being passed into registerResolverForID is the same localIDs member variable. so the idMap contains many ids pointing to the same localIDs object. this wastes memory. removing idMap and using localIDs alone saved some 300Kb for testTree(5,3)
so what's the point in having idMap object.
OR
am i missing something.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SerializationContext.java.diff, memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_58235 ]
Peter Molettiere commented on AXIS-1771:
----------------------------------------
Ah, I see.
The 15 to 1 (memory use to message size) you're talking about is roughly equivalent to the 30 to 1 (memory use to in-memory representation) that I'm talking about, and really doesn't represent any improvement in the memory use of axis.
You asked earlier whether 15 to 1 is good enough, and I thought you had been saying you had improved axis' memory use to that level. I didn't realize that you changed the units.
Have there been any commits to the axis cvs to improve the situation with this bug?
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Jongjin Choi (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_58411 ]
Jongjin Choi commented on AXIS-1771:
------------------------------------
The point of the comment, I think, is that which schema type the java type is mapped to.
If the mapped schema type's namespace is XML Schema, we can turn off multi-ref. If it is SOAP encoding, we can't.
Thought?
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
First, thanks for looking into this. I appreciate your efforts to
improve the situation.
On Jan 19, 2005, at 9:14 PM, Nishant Kumar wrote:
> yes I was using testTree(5,3).
>
> serialization cannot be improved much as the generated xml was itself
> 1.1 Mb. which is roughly the same amount of memory being used by the
> server during serialization.
I don't see the same amount of memory you do. I'm seeing 4M used during
serialization. I'd like to understand why we're seeing different
numbers for the same test. For the 5,3 test, here's what I'm seeing for
serialization:
>> From my new test, replicating your separation of ser and deser:
>>> Created tree with 5 levels and 3 children at each level
>>> Axis used 4 MBytes to serialize 230 KBytes, a ratio of 20.0
> some more bad news. the amount you have measured is just the amount of
> memory that axis server holds on till the end while de-serializing.
> during thing process it creates may local objects that is GCed while
> deserialization process.
This is exactly the problem I'm reporting. The memory axis uses during
the deserialization process is an extraordinary multiple of the size of
the object graph being deserialized. If my numbers are to be believed,
it's more than 60 to 1. While this memory is released at the end of the
deserialization process, this multiple creates what I believe to be an
artificially low limit on the size of message axis can deal with for
any given max heap size. For a 4 GIG heap (the largest you could
possibly get under any 32 bit OS) the largest message axis could
potentially deserialize is only 71 MEGs. In a default JVM running with
-server, the heap is usually only 128M, which means you can't
deserialize anything larger than about 2M. Adding an additional 384M
of heap only nets you an extra 6M in max message size.
Doesn't the amount of memory used to deserialize things seem a little
out of hand to you?
> If the -Xloggc of sun jvm is to be believed
> then the amount of memory wasted this way is 23MB for testTree(5,3).
> This 23MB only includes the memory churned by short lived objects. This
> excludes the 11MB that you finally measure.
Whatever it is that I'm measuring, it's preventing the use of axis for
the serialization of large messages.
> In this respect what venkat suggested will be good to look at. If he
> can
> explain the usage of SAX2EventRecorder may be I can also have a look.
>
>> From a glance it seemed to me that message is first parsed as DOM,
> creating MessageElements (RPCElement in your case) and then they are
> converted to objects. If this is what happens (which I suspect) then
> how
> is SAX involved.
Certainly any improvement will be greatly appreciated.
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
Re: [jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by Nishant Kumar <ni...@itellix.com>.
Peter,
yes I was using testTree(5,3).
serialization cannot be improved much as the generated xml was itself
1.1 Mb. which is roughly the same amount of memory being used by the
server during serialization.
what can be improved in serialization is the way it is serialized so
that the deserialization process is more efficient. I have already added
a patch for one such thing in your bug report.
some more bad news. the amount you have measured is just the amount of
memory that axis server holds on till the end while de-serializing.
during thing process it creates may local objects that is GCed while
deserialization process. If the -Xloggc of sun jvm is to be believed
then the amount of memory wasted this way is 23MB for testTree(5,3).
This 23MB only includes the memory churned by short lived objects. This
excludes the 11MB that you finally measure.
In this respect what venkat suggested will be good to look at. If he can
explain the usage of SAX2EventRecorder may be I can also have a look.
>>From a glance it seemed to me that message is first parsed as DOM,
creating MessageElements (RPCElement in your case) and then they are
converted to objects. If this is what happens (which I suspect) then how
is SAX involved.
thanks,
nishant
On Thu, 2005-01-20 at 00:59, Peter Molettiere wrote:
> Yes, I've double checked, and you're right, deserialization does take
> less memory than serialization. I'd suggest though that the numbers for
> serialization could be improved as well, though. I'm not sure what tree
> size you used to generate your 1M ser to 12M deser numbers... I'm
> guessing it was smaller than 5,3. And it doesn't seem to scale any
> differently than deserialization.
>
> From my new test, replicating your separation of ser and deser:
> > Created tree with 5 levels and 3 children at each level
> > Axis used 4 MBytes to serialize 230 KBytes, a ratio of 20.0
> > Axis used 14 MBytes to deserialize 230 KBytes, a ratio of 64.0
> > Created tree with 5 levels and 4 children at each level
> > Axis used 33 MBytes to serialize 1 MBytes, a ratio of 29.0
> > Axis used 70 MBytes to deserialize 1 MBytes, a ratio of 63.0
> > Created tree with 5 levels and 5 children at each level
> > Axis used 138 MBytes to serialize 2 MBytes, a ratio of 47.0
> > Axis used 269 MBytes to deserialize 2 MBytes, a ratio of 92.0
> > Created tree with 5 levels and 6 children at each level
> > Axis used 277 MBytes to serialize 7 MBytes, a ratio of 36.0
> > Axis used 675 MBytes to deserialize 7 MBytes, a ratio of 89.0
> > Created tree with 5 levels and 7 children at each level
> > Axis used 570 MBytes to serialize 28 MBytes, a ratio of 20.0
> > Out of Memory deserializing 28 MBytes tree.
>
> Imagine a server trying to handle multiple simultaneous requests to
> serialize a 28M tree -- say, 8. That would require over 4.5G of heap,
> something not supportable on any 32 bit JVM. And 8 simultaneous
> requests is not a very high scalability target. Granted there are
> things to do to work around this, like trying to break your request
> into smaller units, etc., but that doesn't change the scaling factor
> for using axis.
>
> If there's anything I can do to help, please let me know!
>
> --
> Peter Molettiere
> Senior Engineer
> Truereq, Inc.
> http://www.truereq.com/
>
Re: [jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
Yes, I've double checked, and you're right, deserialization does take
less memory than serialization. I'd suggest though that the numbers for
serialization could be improved as well, though. I'm not sure what tree
size you used to generate your 1M ser to 12M deser numbers... I'm
guessing it was smaller than 5,3. And it doesn't seem to scale any
differently than deserialization.
From my new test, replicating your separation of ser and deser:
> Created tree with 5 levels and 3 children at each level
> Axis used 4 MBytes to serialize 230 KBytes, a ratio of 20.0
> Axis used 14 MBytes to deserialize 230 KBytes, a ratio of 64.0
> Created tree with 5 levels and 4 children at each level
> Axis used 33 MBytes to serialize 1 MBytes, a ratio of 29.0
> Axis used 70 MBytes to deserialize 1 MBytes, a ratio of 63.0
> Created tree with 5 levels and 5 children at each level
> Axis used 138 MBytes to serialize 2 MBytes, a ratio of 47.0
> Axis used 269 MBytes to deserialize 2 MBytes, a ratio of 92.0
> Created tree with 5 levels and 6 children at each level
> Axis used 277 MBytes to serialize 7 MBytes, a ratio of 36.0
> Axis used 675 MBytes to deserialize 7 MBytes, a ratio of 89.0
> Created tree with 5 levels and 7 children at each level
> Axis used 570 MBytes to serialize 28 MBytes, a ratio of 20.0
> Out of Memory deserializing 28 MBytes tree.
Imagine a server trying to handle multiple simultaneous requests to
serialize a 28M tree -- say, 8. That would require over 4.5G of heap,
something not supportable on any 32 bit JVM. And 8 simultaneous
requests is not a very high scalability target. Granted there are
things to do to work around this, like trying to break your request
into smaller units, etc., but that doesn't change the scaling factor
for using axis.
If there's anything I can do to help, please let me know!
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57756 ]
nishant kumar commented on AXIS-1771:
-------------------------------------
firstly i would like to congratulate you on writing such a good test which was so easy to run.
secondly serialization doesnot take half the memory. i modified your test, to do encode and decode separately.
I found that for the first case serialization was taking just 1MB where as deserialization was taking 12 MB.
so any one trying to fix this should concentrate on deserialization.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by Nishant Kumar <ni...@itellix.com>.
you are right the identity of Integers won't be maintained.
but this object sharing is generally meant for beans not for integers.
it is important for beans because they can be modified. so when you
change a bean property you will want it to reflect everywhere.
but Integers, Strings are immutable so for all business purpose
indentical and equal imply the same.
but the reason i suggested this is that in the code string were treated
as primitives. ie identity of strings were not being maintained even
before my suggested change. so why not apply the same for primitive
wrappers.
thanks,
nishant.
On Wed, 2005-01-19 at 13:08, David Jencks wrote:
> I'm a beginner at this, but..
>
> given the class
>
> public class Foo {
> private Integer bar1;
> private Integer bar2;
>
> public Foo(Integer bar1, Integer bar2) {
> this.bar1 = bar1;
> this.bar2 = bar2;
> }
>
> //getters and setters elided
>
> public boolean degenerate() {
> return bar1 == bar2;
> }
>
> and the instance
>
> Integer one = new Integer(1);
> Foo foo1 = new Foo(one, one);
>
> assert foo1.degenerate();
>
> after serialization and deserialization, resulting in foo2,
>
> assert !foo2.degenerate();
>
> Is this a problem?
>
> thanks
> david jencks
>
>
> On Jan 18, 2005, at 10:40 PM, nishant kumar (JIRA) wrote:
>
> > [
> > http://issues.apache.org/jira/browse/AXIS-1771?
> > page=comments#action_57757 ]
> >
> > nishant kumar commented on AXIS-1771:
> > -------------------------------------
> >
> > deserialization is taking so much memory because serialization has
> > created too many multirefs. even for ints. this happens because the
> > int is wrapped as Integer and the code thinks that it is not a
> > primitive. so just added a method isPrimitiveWrapper which is used to
> > prevent creation of multirefs for primitive wrappers.
> >
> > using this the size of the encoded string was reduced from 1170kb to
> > 966kb and the memory utilization during deserialization was also
> > reduced by 2MB.
> >
> > here is a solution for this.
> > just create
> >
> > Index: src/org/apache/axis/encoding/SerializationContext.java
> > @@ -566,6 +566,22 @@
> > return getTypeMapping().getTypeQName(cls);
> > }
> >
> > +
> > + private boolean isPrimitiveWrapper(Object value)
> > + {
> > + if (value == null) return true;
> > + Class javaType = value.getClass();
> > + if (javaType == Integer.class) return true;
> > + if (javaType == Long.class) return true;
> > + if (javaType == Double.class) return true;
> > + if (javaType == Float.class) return true;
> > + if (javaType == Boolean.class) return true;
> > + if (javaType == Short.class) return true;
> > + if (javaType == Character.class) return true;
> > + if (javaType == Byte.class) return true;
> > + return false;
> > + }
> > +
> > /**
> > * Indicates whether the object should be interpretted as a
> > primitive
> > * for the purposes of multi-ref processing. A primitive value
> > @@ -800,7 +816,7 @@
> > // Objects appear equal.
> >
> > if (doMultiRefs && isEncoded() &&
> > - (value != forceSer) && !isPrimitive(value)) {
> > + (value != forceSer) && !isPrimitive(value) &&
> > !isPrimitiveWrapper(value)) {
> > if (multiRefIndex == -1)
> > multiRefValues = new HashMap();
> >
> >
> >> Excessive Memory Use During Serialization/Deserialization
> >> ---------------------------------------------------------
> >>
> >> Key: AXIS-1771
> >> URL: http://issues.apache.org/jira/browse/AXIS-1771
> >> Project: Axis
> >> Type: Bug
> >> Components: Basic Architecture
> >> Versions: 1.2RC2
> >> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> >> Reporter: Peter Molettiere
> >> Assignee: Venkat Reddy
> >> Attachments: SerializationContext.java.diff, memory-use-test.tgz
> >>
> >> Axis uses pathological amounts of memory during the
> >> serialization/deserialization process.
> >> We see about a 30 to 1 ratio of memory used during (de)serialization
> >> to in-memory representation of the objects being (de)serialized. So
> >> ser/deser in axis of a 2M graph of objects uses 288M of memory!
> >> Further, the memory used seems to scale linearly with the size of the
> >> object graph being serialized.
> >> The memory used does seem to be released once serialization is done,
> >> so this isn't a leak.
> >> Using the attached example code, (based on the code used to
> >> demonstrate AXIS-1423) you can see this behavior. The test
> >> automatically runs with a max heap size of 1024M, and runs out of
> >> memory serializing a 28M object graph.
> >> As provided, it generates the following output:
> >> Buildfile: build.xml
> >> build:
> >> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory
> >> Test/build/classes
> >> run:
> >> [java] - Unable to find required classes
> >> (javax.activation.DataHandler and javax.mail.internet.MimeMultipart).
> >> Attachment support is disabled.
> >> [java] Created tree with 5 levels and 3 children at each level
> >> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of
> >> 30.0
> >> [java] GC freed 13 MBytes
> >> [java] Created tree with 5 levels and 4 children at each level
> >> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> >> [java] GC freed 71 MBytes
> >> [java] Created tree with 5 levels and 5 children at each level
> >> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of
> >> 51.0
> >> [java] GC freed 287 MBytes
> >> [java] Created tree with 5 levels and 6 children at each level
> >> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of
> >> 29.0
> >> [java] GC freed 675 MBytes
> >> [java] Created tree with 5 levels and 7 children at each level
> >> [java] Out of Memory serializing 28 MBytes tree.
> >> [java] Java Result: 1
> >> BUILD SUCCESSFUL
> >> Total time: 2 minutes 51 seconds
> >> Note that the ratios are halved from the reported values, since it
> >> includes both serialization and deserialization of the object graph.
> >> So axis uses 30.5M to serialize a 1M message, and another 30.5M to
> >> deserialize it, resulting in the reported 71M reported above. Also,
> >> notice that the ratio stays close to 30 to 1 regardless of object
> >> graph size. This is the linear scaling I mention above.
> >> Note also, that if you tweak the code to generate very small object
> >> graphs, you see extremely high ratios, but I would expect this due to
> >> simple one-time overhead to operate on very small amounts of data.
> >> That's why I start with the graph size that I do.
> >>
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > If you think it was sent incorrectly contact one of the administrators:
> > http://issues.apache.org/jira/secure/Administrators.jspa
> > -
> > If you want more information on JIRA, or have a bug to report see:
> > http://www.atlassian.com/software/jira
> >
>
Re: [jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by David Jencks <dj...@apache.org>.
I'm a beginner at this, but..
given the class
public class Foo {
private Integer bar1;
private Integer bar2;
public Foo(Integer bar1, Integer bar2) {
this.bar1 = bar1;
this.bar2 = bar2;
}
//getters and setters elided
public boolean degenerate() {
return bar1 == bar2;
}
and the instance
Integer one = new Integer(1);
Foo foo1 = new Foo(one, one);
assert foo1.degenerate();
after serialization and deserialization, resulting in foo2,
assert !foo2.degenerate();
Is this a problem?
thanks
david jencks
On Jan 18, 2005, at 10:40 PM, nishant kumar (JIRA) wrote:
> [
> http://issues.apache.org/jira/browse/AXIS-1771?
> page=comments#action_57757 ]
>
> nishant kumar commented on AXIS-1771:
> -------------------------------------
>
> deserialization is taking so much memory because serialization has
> created too many multirefs. even for ints. this happens because the
> int is wrapped as Integer and the code thinks that it is not a
> primitive. so just added a method isPrimitiveWrapper which is used to
> prevent creation of multirefs for primitive wrappers.
>
> using this the size of the encoded string was reduced from 1170kb to
> 966kb and the memory utilization during deserialization was also
> reduced by 2MB.
>
> here is a solution for this.
> just create
>
> Index: src/org/apache/axis/encoding/SerializationContext.java
> @@ -566,6 +566,22 @@
> return getTypeMapping().getTypeQName(cls);
> }
>
> +
> + private boolean isPrimitiveWrapper(Object value)
> + {
> + if (value == null) return true;
> + Class javaType = value.getClass();
> + if (javaType == Integer.class) return true;
> + if (javaType == Long.class) return true;
> + if (javaType == Double.class) return true;
> + if (javaType == Float.class) return true;
> + if (javaType == Boolean.class) return true;
> + if (javaType == Short.class) return true;
> + if (javaType == Character.class) return true;
> + if (javaType == Byte.class) return true;
> + return false;
> + }
> +
> /**
> * Indicates whether the object should be interpretted as a
> primitive
> * for the purposes of multi-ref processing. A primitive value
> @@ -800,7 +816,7 @@
> // Objects appear equal.
>
> if (doMultiRefs && isEncoded() &&
> - (value != forceSer) && !isPrimitive(value)) {
> + (value != forceSer) && !isPrimitive(value) &&
> !isPrimitiveWrapper(value)) {
> if (multiRefIndex == -1)
> multiRefValues = new HashMap();
>
>
>> Excessive Memory Use During Serialization/Deserialization
>> ---------------------------------------------------------
>>
>> Key: AXIS-1771
>> URL: http://issues.apache.org/jira/browse/AXIS-1771
>> Project: Axis
>> Type: Bug
>> Components: Basic Architecture
>> Versions: 1.2RC2
>> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
>> Reporter: Peter Molettiere
>> Assignee: Venkat Reddy
>> Attachments: SerializationContext.java.diff, memory-use-test.tgz
>>
>> Axis uses pathological amounts of memory during the
>> serialization/deserialization process.
>> We see about a 30 to 1 ratio of memory used during (de)serialization
>> to in-memory representation of the objects being (de)serialized. So
>> ser/deser in axis of a 2M graph of objects uses 288M of memory!
>> Further, the memory used seems to scale linearly with the size of the
>> object graph being serialized.
>> The memory used does seem to be released once serialization is done,
>> so this isn't a leak.
>> Using the attached example code, (based on the code used to
>> demonstrate AXIS-1423) you can see this behavior. The test
>> automatically runs with a max heap size of 1024M, and runs out of
>> memory serializing a 28M object graph.
>> As provided, it generates the following output:
>> Buildfile: build.xml
>> build:
>> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory
>> Test/build/classes
>> run:
>> [java] - Unable to find required classes
>> (javax.activation.DataHandler and javax.mail.internet.MimeMultipart).
>> Attachment support is disabled.
>> [java] Created tree with 5 levels and 3 children at each level
>> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of
>> 30.0
>> [java] GC freed 13 MBytes
>> [java] Created tree with 5 levels and 4 children at each level
>> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
>> [java] GC freed 71 MBytes
>> [java] Created tree with 5 levels and 5 children at each level
>> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of
>> 51.0
>> [java] GC freed 287 MBytes
>> [java] Created tree with 5 levels and 6 children at each level
>> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of
>> 29.0
>> [java] GC freed 675 MBytes
>> [java] Created tree with 5 levels and 7 children at each level
>> [java] Out of Memory serializing 28 MBytes tree.
>> [java] Java Result: 1
>> BUILD SUCCESSFUL
>> Total time: 2 minutes 51 seconds
>> Note that the ratios are halved from the reported values, since it
>> includes both serialization and deserialization of the object graph.
>> So axis uses 30.5M to serialize a 1M message, and another 30.5M to
>> deserialize it, resulting in the reported 71M reported above. Also,
>> notice that the ratio stays close to 30 to 1 regardless of object
>> graph size. This is the linear scaling I mention above.
>> Note also, that if you tweak the code to generate very small object
>> graphs, you see extremely high ratios, but I would expect this due to
>> simple one-time overhead to operate on very small amounts of data.
>> That's why I start with the graph size that I do.
>>
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> If you want more information on JIRA, or have a bug to report see:
> http://www.atlassian.com/software/jira
>
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57757 ]
nishant kumar commented on AXIS-1771:
-------------------------------------
deserialization is taking so much memory because serialization has created too many multirefs. even for ints. this happens because the int is wrapped as Integer and the code thinks that it is not a primitive. so just added a method isPrimitiveWrapper which is used to prevent creation of multirefs for primitive wrappers.
using this the size of the encoded string was reduced from 1170kb to 966kb and the memory utilization during deserialization was also reduced by 2MB.
here is a solution for this.
just create
Index: src/org/apache/axis/encoding/SerializationContext.java
@@ -566,6 +566,22 @@
return getTypeMapping().getTypeQName(cls);
}
+
+ private boolean isPrimitiveWrapper(Object value)
+ {
+ if (value == null) return true;
+ Class javaType = value.getClass();
+ if (javaType == Integer.class) return true;
+ if (javaType == Long.class) return true;
+ if (javaType == Double.class) return true;
+ if (javaType == Float.class) return true;
+ if (javaType == Boolean.class) return true;
+ if (javaType == Short.class) return true;
+ if (javaType == Character.class) return true;
+ if (javaType == Byte.class) return true;
+ return false;
+ }
+
/**
* Indicates whether the object should be interpretted as a primitive
* for the purposes of multi-ref processing. A primitive value
@@ -800,7 +816,7 @@
// Objects appear equal.
if (doMultiRefs && isEncoded() &&
- (value != forceSer) && !isPrimitive(value)) {
+ (value != forceSer) && !isPrimitive(value) && !isPrimitiveWrapper(value)) {
if (multiRefIndex == -1)
multiRefValues = new HashMap();
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SerializationContext.java.diff, memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
nishant kumar updated AXIS-1771:
--------------------------------
Attachment: SerializationContext.java.diff
as mentioned in my last comment
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SerializationContext.java.diff, memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420932 ]
Davanum Srinivas commented on AXIS-1771:
----------------------------------------
Peter,
did you forget this messsage? (http://marc.theaimsgroup.com/?l=axis-dev&m=111109719701313&w=2)
Tony,
I'd recommend opening a new issue with *YOUR* test case / problem info.
thanks,
dims
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57829 ]
nishant kumar commented on AXIS-1771:
-------------------------------------
in methods like startElement etc wherever Attributes is passed it is always copied into another instance of AttributesImpl.
for example this is from startElement method of Deserialization.
attributes = new AttributesImpl(attributes);
Is this necessary if the attributes was already and instance of AttributesImpl.
Are we creating a new instance so that the previous instance is unchanged or because what we have is an Attributes implementation object which can be different from AttributesImpl.
If it is the second can we have something like.
attributes = (attributes instanceof AttributesImpl) ? attributes : attributes = new AttributesImpl(attributes);
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SerializationContext.java.diff, memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Davanum Srinivas updated AXIS-1771:
-----------------------------------
Priority: Blocker (was: Major)
blocker??
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Venkat Reddy (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Venkat Reddy updated AXIS-1771:
-------------------------------
Attachment: MTC_venkat.java
Attaching MTC_venkat.java. Rename it to MemoryTesterClient.java for running.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57832 ]
Davanum Srinivas commented on AXIS-1771:
----------------------------------------
Nishant,
all these feedback about axis code is good. unfortunately trying to track each suggestion is difficult (kind of wild goose chase, since we don't know if it will fix the problem). My suggestion is since you have a good handle on the code, you should come up with a patch that fixes the problem AND then you should run "ant clean all-tests" and make sure that nothing breaks AND then upload the patch for review.
thanks,
dims
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "tony salerno (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420934 ]
tony salerno commented on AXIS-1771:
------------------------------------
Do you know in which release the latest change was included?
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "tony salerno (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420946 ]
tony salerno commented on AXIS-1771:
------------------------------------
Can you let me know what the flags and settings are and I will try with 1.4? Or, you can just point me to some documentation on this fix.
Thanks,
Tony
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "tony salerno (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420918 ]
tony salerno commented on AXIS-1771:
------------------------------------
Was this issue ever resolved? If so, in what release?
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_58385 ]
nishant kumar commented on AXIS-1771:
-------------------------------------
If strings, calendars etc can be considered primitives then i guess primitive wrapper should also be. strings and calendars can also be null. their references can also be shared by different objects so why treat them as primitivs.
And I could not follow the comment exactly.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Davanum Srinivas <da...@gmail.com>.
Should i mark the bug as resolved? or take it off the blocker list? :)
-- dims
On Thu, 17 Mar 2005 14:06:25 -0800, Peter Molettiere
<pi...@axonstudios.net> wrote:
>
> On Mar 17, 2005, at 12:48 PM, Davanum Srinivas (JIRA) wrote:
> > Try this one!!!!!
>
> That is better... you've gotten it down to about a 4 to 1 ratio, which
> means the largest object graph you can de/ser in a 32M heap is about
> 4M. Execution speed seems largely unchanged though. A quick run (about
> 8 minutes) in JProfiler, just serializing and deserializing the same
> tree, doesn't reveal any leaks.
>
> Lots better than before -- from a 20:1 to a 4:1 ratio -- a 5:1
> improvement. I'd take this bug off the blocker list.
>
> --
> Peter Molettiere
> Senior Engineer
> Truereq, Inc.
> http://www.truereq.com/
>
>
--
Davanum Srinivas - http://webservices.apache.org/~dims/
Re: [jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
On Mar 17, 2005, at 12:48 PM, Davanum Srinivas (JIRA) wrote:
> Try this one!!!!!
That is better... you've gotten it down to about a 4 to 1 ratio, which
means the largest object graph you can de/ser in a 32M heap is about
4M. Execution speed seems largely unchanged though. A quick run (about
8 minutes) in JProfiler, just serializing and deserializing the same
tree, doesn't reveal any leaks.
Lots better than before -- from a 20:1 to a 4:1 ratio -- a 5:1
improvement. I'd take this bug off the blocker list.
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Davanum Srinivas updated AXIS-1771:
-----------------------------------
Attachment: MemoryTesterClient.java
Try this one!!!!!
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Peter Molettiere updated AXIS-1771:
-----------------------------------
Attachment: MemoryTesterClient.java
New test client.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "tony salerno (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12421122 ]
tony salerno commented on AXIS-1771:
------------------------------------
I tried the 1.4 jar on the test cases attached to this issue and I did see some difference, but not the kind of performance improvement I expeced. Could you let me know what flags etc. I would use with 1.4 to see the performance improvements.
Thanks,
tony
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Issue Type: Bug
> Components: Basic Architecture
> Affects Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assigned To: Venkat Reddy
> Priority: Blocker
> Attachments: axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, MemoryTesterClient.java, MemoryTesterClient.java, MTC_venkat.java, recursion.png, SerializationContext.java.diff, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml
>
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
Re: [jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
Again, thanks for your attention Venkat. It is greatly appreciated.
I'll patch my version of axis and try to reproduce and measure your
improvements.
On Jan 21, 2005, at 7:34 AM, Venkat Reddy (JIRA) wrote:
> deser.parse() consumes only about 15 times the size the soap message
> it is dealing with, whether multi-ref is true or not. However the soap
> message gets about 3 times larger with multi-ref on. So i think what
> we are asking for is optimizing the soap message size when multi-ref
> is on.
>
> 15 is the ratio when we don't run GC before calling dser.parse(). It
> falls to about 10 if we run GC there. Is the ratio of 15 not
> acceptable?
I'd think of it this way:
ANY multiple larger than 2 of the message size which scales linearly
with message size indicates a performance optimization problem. It's
the linear scaling at issue here. I understand that's a basic
architectural issue with axis that you're probably not going to be able
to fix, and so the discussion is now focusing more on what can be done
(relatively) easily than on what will actually fix the underlying
problem.
Fixing the underlying problem, I think, would result in a de/ser
mechanism with a fixed overhead for a given wsdl which could process
messages of arbitrary size. Streaming XML parsers can do this. I
understand you won't be able to make that happen in the current branch
of axis, so I'm happy with any improvements you can offer.
It would be nice though if the axis dev team could look at this issue
in the process of planning the major version of axis and give some
thought to how the architecture might be changed to better support this
type of use for axis.
In the end, if you manage to provide a total memory use ratio of 15 to
1, that's probably about one half to one quarter the memory use I was
seeing, providing that we're talking about 1) memory used compared to
object graph size (not xml message size), and 2) total memory size, and
not some portion of it, ie., if deser.parse() consumes 15 to 1, and
there's some other piece of the deser mechanism that consumes 10 to 1,
we're at 25 to 1. Which is still an improvement to be sure.
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Venkat Reddy (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57891 ]
Venkat Reddy commented on AXIS-1771:
------------------------------------
Some more notes:
deser.parse() consumes only about 15 times the size the soap message it is dealing with, whether multi-ref is true or not. However the soap message gets about 3 times larger with multi-ref on. So i think what we are asking for is optimizing the soap message size when multi-ref is on.
15 is the ratio when we don't run GC before calling dser.parse(). It falls to about 10 if we run GC there. Is the ratio of 15 not acceptable?
Attaching the client code that i'm using.
Venkat
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57743 ]
Peter Molettiere commented on AXIS-1771:
----------------------------------------
Sorry for the bit about 2M object graph resulting in 288M of memory use during serialization. It should read something like "a 1M object graph uses 71M of memory during serialization and deserialization." Half of the 71 to 1 ration is for serialization, and the other half is for deserialization, thus my figure of 30 to 1.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Attachments: memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Peter Molettiere updated AXIS-1771:
-----------------------------------
Attachment: memory-use-test.tgz
"ant run" will do everything you need.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Attachments: memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Resolved: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Davanum Srinivas resolved AXIS-1771:
------------------------------------
Resolution: Fixed
Closing issue.
-- dims
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Peter Molettiere updated AXIS-1771:
-----------------------------------
Attachment: axis-memory-use.gif
Graph of current data of axis memory use compared to ObjectOutputStream memory use.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57759 ]
nishant kumar commented on AXIS-1771:
-------------------------------------
trying to narrow it down.
tried unsetting AxisEngine.PROP_DOMULTIREFS option.
this resulted in smaller message size and deserialization memory usage reduced from 11MB to 4MB. I don't know whether the identity of the object is maintained after deserialization.
is it possible in multirefs that only when an object is referred the second time is an href created.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SerializationContext.java.diff, memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420933 ]
Peter Molettiere commented on AXIS-1771:
----------------------------------------
Yes, I did forget that message --
Reposting my earlier list post here for reference:
http://marc.theaimsgroup.com/?l=axis-dev&m=111109719701313&w=2
That is better... you've gotten it down to about a 4 to 1 ratio, which
means the largest object graph you can de/ser in a 32M heap is about 4M. Execution speed seems largely unchanged though. A quick run (about 8 minutes) in JProfiler, just serializing and deserializing the same tree, doesn't reveal any leaks.
Lots better than before -- from a 20:1 to a 4:1 ratio -- a 5:1
improvement. I'd take this bug off the blocker list.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Archit Shah (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57753 ]
Archit Shah commented on AXIS-1771:
-----------------------------------
just for cross reference, here is my axis-dev posting on the topic: http://marc.theaimsgroup.com/?l=axis-dev&m=110029173620727&w=2
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Jongjin Choi (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_58356 ]
Jongjin Choi commented on AXIS-1771:
------------------------------------
Nishant,
In isPrmitive() method of SerializationContext,
there is a comment whether java primitive wrapper type should be multi-refed or not.
According to that, we might not always turn off multi-ref for java primitive wrappers.
I tested with your another patch for AttributeImpl last week, I don't remember the accurate measurements now, but I feel that it made some amount of memory saving in deserialization.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420925 ]
Peter Molettiere commented on AXIS-1771:
----------------------------------------
I don't know. I just saw your comment, Tony, and noticed that there are six comments from dims on the day of my last post, and that he closed the issue the next day. I wasn't aware the issue had been updated.
About the time I posted relative performance graphs for ObjectOutputStream, we stopped using Axis in favor of that solution. If I'm reading dims comment correctly ("Axis takes 8 times slower consistently as compared to the ObjectOutputStream option."), then I'd say we did the right thing for our application. If ObjectOutputStream uses less memory and is faster to boot, that suits us better, as we don't require strict SOAP interoperability.
Be aware, too, that this issue is mainly problematic when serializing large messages with Axis. If you're in the 90% use case, that is, serializing small messages, you probably won't notice.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "nishant kumar (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_58353 ]
nishant kumar commented on AXIS-1771:
-------------------------------------
did anyone try applying a small patch i had attached some time back. it reduces the number of multirefs elements created by eliminating creating multirefs for primitives.
have a look at SerializationContext.java.diff in the attachments lists above.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Peter Molettiere (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Peter Molettiere updated AXIS-1771:
-----------------------------------
Attachment: memory-use-test.tgz
Updated with RC3, and comparison against ObjectOutputStream.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by Peter Molettiere <pi...@axonstudios.net>.
Thank you all for looking into this issue. Your attention is greatly
appreciated.
On Jan 19, 2005, at 7:16 AM, Venkat Reddy (JIRA) wrote:
> [
> http://issues.apache.org/jira/browse/AXIS-1771?
> page=comments#action_57774 ]
>
> Venkat Reddy commented on AXIS-1771:
> ------------------------------------
> [snip]
> I also see some deep recursion among -
>
> MessageElement.publishToHandler -> SAX2EventRecorder.replay ->
> DeserializationContext.startElement -> BeanDeserializer.startElement
> -> DeserializerImpl.startElement -> publishToHandler, and it starts
> all over again.
>
> Its not an infinite loop, but it goes pretty deep in the call frames
> and lot of object arrays are being allocated, and most of the
> processing time is being spent here.
>
> Did anybody have insights into this area of the code?
Venkat:
My understanding of the call stack you're asking about, which I've
mainly cobbled together from trying to figure out how to write a
Serializer/Deserializer pair, is that for a given object graph, say:
root
- A
- 1
- 2
-B
- 3
- 4
For each node in the tree there is both a MessageElement and a
Deserializer instance. The entire tree shares a single
DerserializationContext. So what you're seeing is something like this;
root(MessageElement.publishToHandler)->SAX->DeserContext-
>root(BeanDeser.startElement)->root(DeserImpl.startElement [a call to
super.startElement() from the BeanDeserializer)->
A(MessageElement.publishToHandler)->SAX->DeserContext-
>A(BeanDeser.startElement)->A(DeserImpl.startElement [a call to
super.startElement() from the BeanDeserializer)->
1(MessageElement.publishToHandler)->SAX->DeserContext-
>1(BeanDeser.startElement)->1(DeserImpl.startElement [a call to
super.startElement() from the BeanDeserializer)->
[insert endElement calls for 1 here]
2(MessageElement.publishToHandler)->SAX->DeserContext-
>2(BeanDeser.startElement)->2(DeserImpl.startElement [a call to
super.startElement() from the BeanDeserializer)->
[insert endElement calls for 2 here]
[insert endElement calls for A here]
B(MessageElement.publishToHandler)->SAX->DeserContext-
>B(BeanDeser.startElement)->B(DeserImpl.startElement [a call to
super.startElement() from the BeanDeserializer)->
3(MessageElement.publishToHandler)->SAX->DeserContext-
>3(BeanDeser.startElement)->3(DeserImpl.startElement [a call to
super.startElement() from the BeanDeserializer)->
[insert endElement calls for 3 here]
4(MessageElement.publishToHandler)->SAX->DeserContext-
>4(BeanDeser.startElement)->4(DeserImpl.startElement [a call to
super.startElement() from the BeanDeserializer)->
[insert endElement calls for 4 here]
[insert endElement calls for B here]
[insert endElement calls for root here]
This means that for my little seven node graph, there are 7
MessageElement instances, 7 Deserializer instances, and a Deser Context
instance. That's 15 objects to deserialize 7 nodes. And I think that
each MessageElement may also have other things like RPCParams and
RPCMessageElements, etc, although my understanding of what's happening
there is pretty thin. Maybe those are just interfaces implemented by
MessageElement.
--
Peter Molettiere
Senior Engineer
Truereq, Inc.
http://www.truereq.com/
[jira] Commented: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Venkat Reddy (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_57774 ]
Venkat Reddy commented on AXIS-1771:
------------------------------------
I think i should dig into working of SAX2EventRecorder. The iinstance variable "events" seems to hold too many array elements during deserialization.
I also see some deep recursion among -
MessageElement.publishToHandler -> SAX2EventRecorder.replay -> DeserializationContext.startElement -> BeanDeserializer.startElement -> DeserializerImpl.startElement -> publishToHandler, and it starts all over again.
Its not an infinite loop, but it goes pretty deep in the call frames and lot of object arrays are being allocated, and most of the processing time is being spent here.
Did anybody have insights into this area of the code?
Venkat
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SerializationContext.java.diff, memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (AXIS-1771) Excessive Memory Use During
Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=comments#action_12420939 ]
Davanum Srinivas commented on AXIS-1771:
----------------------------------------
Tony,
That would be 1.4. *but* there are flags and setting etc. So i'd recommend you try 1.4 and open a new issue against that with your specific test / problem info.
thanks,
dims
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Apache Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Priority: Blocker
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Assigned: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Davanum Srinivas reassigned AXIS-1771:
--------------------------------------
Assign To: Venkat Reddy
Venkat,
i took the liberty of assigning this to you :)
-- dims
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: memory-use-test.tgz
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Davanum Srinivas (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Davanum Srinivas updated AXIS-1771:
-----------------------------------
Attachment: MemoryTesterClient.java
New file with HiFi on (Low memory, but high CPU). Need latest CVS (RC3 has NPE)
thanks,
dims
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: MTC_venkat.java, MemoryTesterClient.java, MemoryTesterClient.java, SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, axis-memory-use.gif, memory-use-test.tgz, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Updated: (AXIS-1771) Excessive Memory Use During Serialization/Deserialization
Posted by "Venkat Reddy (JIRA)" <ax...@ws.apache.org>.
[ http://issues.apache.org/jira/browse/AXIS-1771?page=history ]
Venkat Reddy updated AXIS-1771:
-------------------------------
Attachment: SOAPmsg_multiref_true.xml
SOAPmsg_multiref_false.xml
recursion.png
Attaching SOAP Messages when multi-ref is true and false, and an image with recursive call stack.
> Excessive Memory Use During Serialization/Deserialization
> ---------------------------------------------------------
>
> Key: AXIS-1771
> URL: http://issues.apache.org/jira/browse/AXIS-1771
> Project: Axis
> Type: Bug
> Components: Basic Architecture
> Versions: 1.2RC2
> Environment: JDK 1.4.2, Mac OS X, Linux, Windows
> Reporter: Peter Molettiere
> Assignee: Venkat Reddy
> Attachments: SOAPmsg_multiref_false.xml, SOAPmsg_multiref_true.xml, SerializationContext.java.diff, memory-use-test.tgz, recursion.png
>
> Axis uses pathological amounts of memory during the serialization/deserialization process.
> We see about a 30 to 1 ratio of memory used during (de)serialization to in-memory representation of the objects being (de)serialized. So ser/deser in axis of a 2M graph of objects uses 288M of memory! Further, the memory used seems to scale linearly with the size of the object graph being serialized.
> The memory used does seem to be released once serialization is done, so this isn't a leak.
> Using the attached example code, (based on the code used to demonstrate AXIS-1423) you can see this behavior. The test automatically runs with a max heap size of 1024M, and runs out of memory serializing a 28M object graph.
> As provided, it generates the following output:
> Buildfile: build.xml
> build:
> [javac] Compiling 1 source file to /Users/pietro/Work/Axis Memory Test/build/classes
> run:
> [java] - Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
> [java] Created tree with 5 levels and 3 children at each level
> [java] Axis used 13 MBytes to serialize 230 KBytes, a ratio of 30.0
> [java] GC freed 13 MBytes
> [java] Created tree with 5 levels and 4 children at each level
> [java] Axis used 71 MBytes to serialize 1 MBytes, a ratio of 31.0
> [java] GC freed 71 MBytes
> [java] Created tree with 5 levels and 5 children at each level
> [java] Axis used 288 MBytes to serialize 2 MBytes, a ratio of 51.0
> [java] GC freed 287 MBytes
> [java] Created tree with 5 levels and 6 children at each level
> [java] Axis used 671 MBytes to serialize 11 MBytes, a ratio of 29.0
> [java] GC freed 675 MBytes
> [java] Created tree with 5 levels and 7 children at each level
> [java] Out of Memory serializing 28 MBytes tree.
> [java] Java Result: 1
> BUILD SUCCESSFUL
> Total time: 2 minutes 51 seconds
> Note that the ratios are halved from the reported values, since it includes both serialization and deserialization of the object graph. So axis uses 30.5M to serialize a 1M message, and another 30.5M to deserialize it, resulting in the reported 71M reported above. Also, notice that the ratio stays close to 30 to 1 regardless of object graph size. This is the linear scaling I mention above.
> Note also, that if you tweak the code to generate very small object graphs, you see extremely high ratios, but I would expect this due to simple one-time overhead to operate on very small amounts of data. That's why I start with the graph size that I do.
>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira