You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-users@xerces.apache.org by hk...@teramail.com on 2007/09/20 23:00:37 UTC
[Fwd: memory problem caused by XMLReader]
I am processing 80 XML messages per second and I am encountering
OutOfMemory problem, and I have two questions.
1. I create and delete XercesDOMParser for each message. This may not be
efficient but I do not know how the memoy management is working internally
by reusing a parser. Is the memory keep increasing by reusing a parser ?
How do I release the previous message related memory?
2. Everytime I create a new XercesDOMParser, it seems to need a chink of
memory. XMLReader inside of XercesDOMParser requires about 164KB. After
prcessing about 500K messages, OS generates OutOfMemory exception while my
process is using about 650MB only. Why does this happen? My guess is that
there is memory fragmentation and OS cannot find 164KB of continous memory
space. How can I avoid this problem?
3. For the possible solutions suggested for question 1, does it solve the
problem 2 also?
Many Thanks, HK
Re: [Fwd: memory problem caused by XMLReader]
Posted by Alberto Massari <am...@datadirect.com>.
At 06.49 21/09/2007 +0900, hkim@teramail.com wrote:
>[...]
>There is still a problem without memory leaking, I think.
>Let's assume that the program is increasing memory space continusly.
>Let me clarify this by small example.
>
>loop:
> 1: get XML message
> 2: create a parser
> 3: parse
> 4: do my job
> 5. delete a parser.
> 6. add small data to history list. Let's assume this requires 10B in
>dynamic memory
>
How did you access the DOM tree generated by the parser? If you use
adoptDocument(), it's your responsibility to release it (otherwise,
if you use getDocument(), it will automatically deleted when you
delete the parser).
Alberto
>So we allocated 160KB in step 2 and deallocated it in step 5, but memory
>used in step 6 may use the space returned by step 5. Then the same memory
>space cannot be used for next iteration. By this fragmentation, OS will
>not find any more big space for step 2 soon. Again this is only a guess.
>
>HK
>
>
> >
> > Dave
> >
> >
Re: [Fwd: memory problem caused by XMLReader]
Posted by David Bertoni <db...@apache.org>.
hkim@teramail.com wrote:
> Dear Dave
>
> Thanks for your reply. Please find my comments below.
>
>> hkim@teramail.com wrote:
...
>
> Unfortunately, SAX is not an option because of the compatibility to
> existing code.
> My question was that what happens to the memory allocated for the previous
> message parsing, and how I can delete them if I have to.
>
...
>> Are you sure your application isn't leaking memory somewhere? Normally,
>> memory for a DOM instance is allocated in large blocks, so fragmentation
>> should not be a problem. It might be, however, if your code is leaking.
>
> Maybe, but I would call it a feature ^^.
>
> There is still a problem without memory leaking, I think.
> Let's assume that the program is increasing memory space continusly.
> Let me clarify this by small example.
>
> loop:
> 1: get XML message
> 2: create a parser
> 3: parse
> 4: do my job
> 5. delete a parser.
> 6. add small data to history list. Let's assume this requires 10B in
> dynamic memory
>
>
> So we allocated 160KB in step 2 and deallocated it in step 5, but memory
> used in step 6 may use the space returned by step 5. Then the same memory
> space cannot be used for next iteration. By this fragmentation, OS will
> not find any more big space for step 2 soon. Again this is only a guess.
Yes, I'm aware of how memory can become fragmented, but I doubt that's the
problem, unless your application is leaking memory, or you're keeping
_lots_ of small objects around. But if that's the case, there's nothing
anyone here can help you with, since we're not in control of your
run-time's memory management implementation and you haven't shown us any
source code to see if there are leaks.
Dave
Re: [Fwd: memory problem caused by XMLReader]
Posted by hk...@teramail.com.
Dear Dave
Thanks for your reply. Please find my comments below.
> hkim@teramail.com wrote:
>>
>> I am processing 80 XML messages per second and I am encountering
>> OutOfMemory problem, and I have two questions.
>>
>> 1. I create and delete XercesDOMParser for each message. This may not be
>> efficient but I do not know how the memoy management is working
>> internally
>> by reusing a parser. Is the memory keep increasing by reusing a parser ?
>> How do I release the previous message related memory?
> In general, using the DOM is not as efficient as using SAX-based parsing.
> You can certainly re-use a XercesDOMParser instance, but it might not help
> that much
Unfortunately, SAX is not an option because of the compatibility to
existing code.
My question was that what happens to the memory allocated for the previous
message parsing, and how I can delete them if I have to.
>
>>
>> 2. Everytime I create a new XercesDOMParser, it seems to need a chink of
>> memory. XMLReader inside of XercesDOMParser requires about 164KB. After
>> prcessing about 500K messages, OS generates OutOfMemory exception while
>> my
>> process is using about 650MB only. Why does this happen? My guess is
>> that
>> there is memory fragmentation and OS cannot find 164KB of continous
>> memory
>> space. How can I avoid this problem?
> Are you sure your application isn't leaking memory somewhere? Normally,
> memory for a DOM instance is allocated in large blocks, so fragmentation
> should not be a problem. It might be, however, if your code is leaking.
Maybe, but I would call it a feature ^^.
There is still a problem without memory leaking, I think.
Let's assume that the program is increasing memory space continusly.
Let me clarify this by small example.
loop:
1: get XML message
2: create a parser
3: parse
4: do my job
5. delete a parser.
6. add small data to history list. Let's assume this requires 10B in
dynamic memory
So we allocated 160KB in step 2 and deallocated it in step 5, but memory
used in step 6 may use the space returned by step 5. Then the same memory
space cannot be used for next iteration. By this fragmentation, OS will
not find any more big space for step 2 soon. Again this is only a guess.
HK
>
> Dave
>
>
Re: [Fwd: memory problem caused by XMLReader]
Posted by David Bertoni <db...@apache.org>.
hkim@teramail.com wrote:
>
> I am processing 80 XML messages per second and I am encountering
> OutOfMemory problem, and I have two questions.
>
> 1. I create and delete XercesDOMParser for each message. This may not be
> efficient but I do not know how the memoy management is working internally
> by reusing a parser. Is the memory keep increasing by reusing a parser ?
> How do I release the previous message related memory?
In general, using the DOM is not as efficient as using SAX-based parsing.
You can certainly re-use a XercesDOMParser instance, but it might not help
that much
>
> 2. Everytime I create a new XercesDOMParser, it seems to need a chink of
> memory. XMLReader inside of XercesDOMParser requires about 164KB. After
> prcessing about 500K messages, OS generates OutOfMemory exception while my
> process is using about 650MB only. Why does this happen? My guess is that
> there is memory fragmentation and OS cannot find 164KB of continous memory
> space. How can I avoid this problem?
Are you sure your application isn't leaking memory somewhere? Normally,
memory for a DOM instance is allocated in large blocks, so fragmentation
should not be a problem. It might be, however, if your code is leaking.
Dave