You are viewing a plain text version of this content. The canonical link for it is here.
Posted to xindice-users@xml.apache.org by Robert Spee <qr...@oce.nl> on 2001/12/17 11:09:20 UTC
memory leak
I'm using dbXML-Core-1.04b with the Java XML:DB API and I have the following
problem:
When I do getCollection() and collection.close() not all the memory is
released and the databasse server runs out of memory.
Does anyone knows the reason for this memory leak?
Gr
Robert
Re: memory leak
Posted by Robert Spee <qr...@oce.nl>.
In my test I retrieve the same document repeatedly. If I do not open and
close the collection every loop, no memory leak will arise. In my
application this is not possible because I use different collections. Maybe
I could change the max heap size, but I don't think that's the right
solution. When I use larger xml resources I can run out of memory.
When you chache a collection, it seems to me, that when I reopen that
collection, it is used from the cache and not creating a new one. I don't
see what this problem has to do with caching if chaching does what I think
it should do.
----- Original Message -----
From: "Kimbro Staken" <ks...@dbxmlgroup.com>
To: <xi...@xml.apache.org>
Sent: Friday, December 21, 2001 10:18 AM
Subject: Re: memory leak
> This sounds to me like normal caching. The server will cache documents
> when retrieved and the cache is only garbage collected if the server runs
> out of heap space. It is normal for memory usage to increase as documents
> are retrieved. A more conclusive test would be repeatedly retrieving the
> same document. If memory increases to any significant degree then there
> likely is something wrong.
>
> If your app slowed down as memory increased then I'd look first at
> insuring you have enough RAM to keep the OS from swapping. The default max
> heap size is 168 MB so the cache will not be garbage collected until it
> nears that level. If you don't have enough RAM to keep all your processes
> in memory then it will swap before garbage collecting the cache. If RAM is
> the problem you can decrease the max heap size in the start script, but be
> careful as you run the risk of getting out of memory errors if you go too
> low.
>
> On Friday, December 21, 2001, at 01:38 AM, Robert Spee wrote:
>
> > I discoverd it, when my own complex program became very slow after a
> > while.
> >
> > I ran the test program on a Linux machine, and the memory usage for the
> > database proces grows form 20 MB initially to 80MB after 1000 loops of
the
> > test program (be sure that the resource you try to get exists, otherwise
> > teh
> > memory leak is much smaller). I think the memory usage may not grow,
> > because
> > the database does not grow.
> > On my windows NT4.0 system the memory usage grows from 15 MB to 35 MB in
> > 1000 loops. In these cases I used a very small resource (about 5
> > charachters). When I use a large resource (250 Kb) the memory usage of
my
> > Win NT system grows from 15 to 30 MB in 150 loops.
> >
> > I think there is a problem somewhere. Maybe some resources should be
> > released but aren't. Maybe your Java VM works with it but mines not.
> >
> > By the way: I use Java 2 SDK 1.3.1
> >
> >
> >
> > ----- Original Message -----
> > From: "Kimbro Staken" <ks...@dbxmlgroup.com>
> > To: <xi...@xml.apache.org>
> > Sent: Friday, December 21, 2001 12:38 AM
> > Subject: Re: memory leak
> >
> >
> >> Do you see this problem with a program that doesn't have an infinite
loop
> >> in it? I ran it through 5200 iterations and memory use was flat for
both
> >> client and server. This is on Mac OS X. At what point are you running
out
> >> of memory? What platform?
> >>
> >> On Tuesday, December 18, 2001, at 12:50 AM, Robert Spee wrote:
> >>
> >>> This is the java program I have tried and shows the memory leak.
> >>> The problem is with the DatabaseManeger.getCollection() method.
> >>>
> >>> ----------------------------------
> >>> import org.xmldb.api.base.*;
> >>> import org.xmldb.api.modules.*;
> >>> import org.xmldb.api.DatabaseManager;
> >>>
> >>> // For the dbXML specific CollectionManager service
> >>> import org.dbxml.client.xmldb.services.*;
> >>>
> >>>
> >>> public class test {
> >>> public static void main( String[] args ) throws Exception {
> >>> String vendor = "dbxml";
> >>> String drv = "org.dbxml.client.xmldb.DatabaseImpl";
> >>> String baseCol = "/db/";
> >>>
> >>> String id= "test2";
> >>> String result;
> >>> String collectionName="docbase/xml";
> >>>
> >>> Class c = Class.forName(drv);
> >>> Database db = (Database) c.newInstance();
> >>> DatabaseManager.registerDatabase(db);
> >>>
> >>> for (int i=0;;i++) {
> >>> // if I put the next line (and the col.close() statement) outside
> > the
> >>> for loop there is no memory leak
> >>> Collection col =
> >>>
> >
DatabaseManager.getCollection("xmldb:"+vendor+"://"+baseCol+collectionName)
> >>> ;
> >>>
> >>> System.out.println( "*** loop "+i+" ***");
> >>>
> >>> Resource resource = col.getResource(id);
> >>>
> >>> if (resource != null) {
> >>> result = (String) resource.getContent();
> >>> System.out.println(result);
> >>> } else {
> >>> System.out.println("ID: '"+id+"' does not exist.");
> >>> }
> >>> col.close();
> >>> }//for
> >>> } //main
> >>> } //class
> >>> ---------------------------------
> >>>
> >>> Hope anyone can help.
> >>>
> >>> Gr
> >>>
> >>> Robert
> >>>
> >>> ----- Original Message -----
> >>> From: "Kimbro Staken" <ks...@dbxmlgroup.com>
> >>> To: <xi...@xml.apache.org>
> >>> Sent: Monday, December 17, 2001 9:13 PM
> >>> Subject: Re: memory leak
> >>>
> >>>
> >>>>
> >>>> On Monday, December 17, 2001, at 03:09 AM, Robert Spee wrote:
> >>>>
> >>>>> I'm using dbXML-Core-1.04b with the Java XML:DB API and I have the
> >>>>> following
> >>>>> problem:
> >>>>>
> >>>>> When I do getCollection() and collection.close() not all the memory
> >>>>> is
> >>>>> released and the databasse server runs out of memory.
> >>>>>
> >>>>> Does anyone knows the reason for this memory leak?
> >>>>>
> >>>>
> >>>> I am not aware of any memory leaks in the beta 4 release. Can you
> >>>> describe
> >>>> more about what you're doing and where the error is occurring?
> >>>>
> >>>>> Gr
> >>>>>
> >>>>> Robert
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> Kimbro Staken
> XML Database Software, Consulting and Writing
> http://www.xmldatabases.org/
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
> >
>
>
Re: memory leak
Posted by Kimbro Staken <ks...@dbxmlgroup.com>.
This sounds to me like normal caching. The server will cache documents
when retrieved and the cache is only garbage collected if the server runs
out of heap space. It is normal for memory usage to increase as documents
are retrieved. A more conclusive test would be repeatedly retrieving the
same document. If memory increases to any significant degree then there
likely is something wrong.
If your app slowed down as memory increased then I'd look first at
insuring you have enough RAM to keep the OS from swapping. The default max
heap size is 168 MB so the cache will not be garbage collected until it
nears that level. If you don't have enough RAM to keep all your processes
in memory then it will swap before garbage collecting the cache. If RAM is
the problem you can decrease the max heap size in the start script, but be
careful as you run the risk of getting out of memory errors if you go too
low.
On Friday, December 21, 2001, at 01:38 AM, Robert Spee wrote:
> I discoverd it, when my own complex program became very slow after a
> while.
>
> I ran the test program on a Linux machine, and the memory usage for the
> database proces grows form 20 MB initially to 80MB after 1000 loops of the
> test program (be sure that the resource you try to get exists, otherwise
> teh
> memory leak is much smaller). I think the memory usage may not grow,
> because
> the database does not grow.
> On my windows NT4.0 system the memory usage grows from 15 MB to 35 MB in
> 1000 loops. In these cases I used a very small resource (about 5
> charachters). When I use a large resource (250 Kb) the memory usage of my
> Win NT system grows from 15 to 30 MB in 150 loops.
>
> I think there is a problem somewhere. Maybe some resources should be
> released but aren't. Maybe your Java VM works with it but mines not.
>
> By the way: I use Java 2 SDK 1.3.1
>
>
>
> ----- Original Message -----
> From: "Kimbro Staken" <ks...@dbxmlgroup.com>
> To: <xi...@xml.apache.org>
> Sent: Friday, December 21, 2001 12:38 AM
> Subject: Re: memory leak
>
>
>> Do you see this problem with a program that doesn't have an infinite loop
>> in it? I ran it through 5200 iterations and memory use was flat for both
>> client and server. This is on Mac OS X. At what point are you running out
>> of memory? What platform?
>>
>> On Tuesday, December 18, 2001, at 12:50 AM, Robert Spee wrote:
>>
>>> This is the java program I have tried and shows the memory leak.
>>> The problem is with the DatabaseManeger.getCollection() method.
>>>
>>> ----------------------------------
>>> import org.xmldb.api.base.*;
>>> import org.xmldb.api.modules.*;
>>> import org.xmldb.api.DatabaseManager;
>>>
>>> // For the dbXML specific CollectionManager service
>>> import org.dbxml.client.xmldb.services.*;
>>>
>>>
>>> public class test {
>>> public static void main( String[] args ) throws Exception {
>>> String vendor = "dbxml";
>>> String drv = "org.dbxml.client.xmldb.DatabaseImpl";
>>> String baseCol = "/db/";
>>>
>>> String id= "test2";
>>> String result;
>>> String collectionName="docbase/xml";
>>>
>>> Class c = Class.forName(drv);
>>> Database db = (Database) c.newInstance();
>>> DatabaseManager.registerDatabase(db);
>>>
>>> for (int i=0;;i++) {
>>> // if I put the next line (and the col.close() statement) outside
> the
>>> for loop there is no memory leak
>>> Collection col =
>>>
> DatabaseManager.getCollection("xmldb:"+vendor+"://"+baseCol+collectionName)
>>> ;
>>>
>>> System.out.println( "*** loop "+i+" ***");
>>>
>>> Resource resource = col.getResource(id);
>>>
>>> if (resource != null) {
>>> result = (String) resource.getContent();
>>> System.out.println(result);
>>> } else {
>>> System.out.println("ID: '"+id+"' does not exist.");
>>> }
>>> col.close();
>>> }//for
>>> } //main
>>> } //class
>>> ---------------------------------
>>>
>>> Hope anyone can help.
>>>
>>> Gr
>>>
>>> Robert
>>>
>>> ----- Original Message -----
>>> From: "Kimbro Staken" <ks...@dbxmlgroup.com>
>>> To: <xi...@xml.apache.org>
>>> Sent: Monday, December 17, 2001 9:13 PM
>>> Subject: Re: memory leak
>>>
>>>
>>>>
>>>> On Monday, December 17, 2001, at 03:09 AM, Robert Spee wrote:
>>>>
>>>>> I'm using dbXML-Core-1.04b with the Java XML:DB API and I have the
>>>>> following
>>>>> problem:
>>>>>
>>>>> When I do getCollection() and collection.close() not all the memory
>>>>> is
>>>>> released and the databasse server runs out of memory.
>>>>>
>>>>> Does anyone knows the reason for this memory leak?
>>>>>
>>>>
>>>> I am not aware of any memory leaks in the beta 4 release. Can you
>>>> describe
>>>> more about what you're doing and where the error is occurring?
>>>>
>>>>> Gr
>>>>>
>>>>> Robert
>>>>>
>>>>>
>>>>>
>>>>>
Kimbro Staken
XML Database Software, Consulting and Writing
http://www.xmldatabases.org/
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
>
Re: memory leak
Posted by Robert Spee <qr...@oce.nl>.
I discoverd it, when my own complex program became very slow after a while.
I ran the test program on a Linux machine, and the memory usage for the
database proces grows form 20 MB initially to 80MB after 1000 loops of the
test program (be sure that the resource you try to get exists, otherwise teh
memory leak is much smaller). I think the memory usage may not grow, because
the database does not grow.
On my windows NT4.0 system the memory usage grows from 15 MB to 35 MB in
1000 loops. In these cases I used a very small resource (about 5
charachters). When I use a large resource (250 Kb) the memory usage of my
Win NT system grows from 15 to 30 MB in 150 loops.
I think there is a problem somewhere. Maybe some resources should be
released but aren't. Maybe your Java VM works with it but mines not.
By the way: I use Java 2 SDK 1.3.1
----- Original Message -----
From: "Kimbro Staken" <ks...@dbxmlgroup.com>
To: <xi...@xml.apache.org>
Sent: Friday, December 21, 2001 12:38 AM
Subject: Re: memory leak
> Do you see this problem with a program that doesn't have an infinite loop
> in it? I ran it through 5200 iterations and memory use was flat for both
> client and server. This is on Mac OS X. At what point are you running out
> of memory? What platform?
>
> On Tuesday, December 18, 2001, at 12:50 AM, Robert Spee wrote:
>
> > This is the java program I have tried and shows the memory leak.
> > The problem is with the DatabaseManeger.getCollection() method.
> >
> > ----------------------------------
> > import org.xmldb.api.base.*;
> > import org.xmldb.api.modules.*;
> > import org.xmldb.api.DatabaseManager;
> >
> > // For the dbXML specific CollectionManager service
> > import org.dbxml.client.xmldb.services.*;
> >
> >
> > public class test {
> > public static void main( String[] args ) throws Exception {
> > String vendor = "dbxml";
> > String drv = "org.dbxml.client.xmldb.DatabaseImpl";
> > String baseCol = "/db/";
> >
> > String id= "test2";
> > String result;
> > String collectionName="docbase/xml";
> >
> > Class c = Class.forName(drv);
> > Database db = (Database) c.newInstance();
> > DatabaseManager.registerDatabase(db);
> >
> > for (int i=0;;i++) {
> > // if I put the next line (and the col.close() statement) outside
the
> > for loop there is no memory leak
> > Collection col =
> >
DatabaseManager.getCollection("xmldb:"+vendor+"://"+baseCol+collectionName)
> > ;
> >
> > System.out.println( "*** loop "+i+" ***");
> >
> > Resource resource = col.getResource(id);
> >
> > if (resource != null) {
> > result = (String) resource.getContent();
> > System.out.println(result);
> > } else {
> > System.out.println("ID: '"+id+"' does not exist.");
> > }
> > col.close();
> > }//for
> > } //main
> > } //class
> > ---------------------------------
> >
> > Hope anyone can help.
> >
> > Gr
> >
> > Robert
> >
> > ----- Original Message -----
> > From: "Kimbro Staken" <ks...@dbxmlgroup.com>
> > To: <xi...@xml.apache.org>
> > Sent: Monday, December 17, 2001 9:13 PM
> > Subject: Re: memory leak
> >
> >
> >>
> >> On Monday, December 17, 2001, at 03:09 AM, Robert Spee wrote:
> >>
> >>> I'm using dbXML-Core-1.04b with the Java XML:DB API and I have the
> >>> following
> >>> problem:
> >>>
> >>> When I do getCollection() and collection.close() not all the memory is
> >>> released and the databasse server runs out of memory.
> >>>
> >>> Does anyone knows the reason for this memory leak?
> >>>
> >>
> >> I am not aware of any memory leaks in the beta 4 release. Can you
> >> describe
> >> more about what you're doing and where the error is occurring?
> >>
> >>> Gr
> >>>
> >>> Robert
> >>>
> >>>
> >>>
> >>>
> Kimbro Staken
> XML Database Software, Consulting and Writing
> http://www.xmldatabases.org/
> >>
> >>
> >
> >
> >
>
>
Re: memory leak
Posted by Kimbro Staken <ks...@dbxmlgroup.com>.
Do you see this problem with a program that doesn't have an infinite loop
in it? I ran it through 5200 iterations and memory use was flat for both
client and server. This is on Mac OS X. At what point are you running out
of memory? What platform?
On Tuesday, December 18, 2001, at 12:50 AM, Robert Spee wrote:
> This is the java program I have tried and shows the memory leak.
> The problem is with the DatabaseManeger.getCollection() method.
>
> ----------------------------------
> import org.xmldb.api.base.*;
> import org.xmldb.api.modules.*;
> import org.xmldb.api.DatabaseManager;
>
> // For the dbXML specific CollectionManager service
> import org.dbxml.client.xmldb.services.*;
>
>
> public class test {
> public static void main( String[] args ) throws Exception {
> String vendor = "dbxml";
> String drv = "org.dbxml.client.xmldb.DatabaseImpl";
> String baseCol = "/db/";
>
> String id= "test2";
> String result;
> String collectionName="docbase/xml";
>
> Class c = Class.forName(drv);
> Database db = (Database) c.newInstance();
> DatabaseManager.registerDatabase(db);
>
> for (int i=0;;i++) {
> // if I put the next line (and the col.close() statement) outside the
> for loop there is no memory leak
> Collection col =
> DatabaseManager.getCollection("xmldb:"+vendor+"://"+baseCol+collectionName)
> ;
>
> System.out.println( "*** loop "+i+" ***");
>
> Resource resource = col.getResource(id);
>
> if (resource != null) {
> result = (String) resource.getContent();
> System.out.println(result);
> } else {
> System.out.println("ID: '"+id+"' does not exist.");
> }
> col.close();
> }//for
> } //main
> } //class
> ---------------------------------
>
> Hope anyone can help.
>
> Gr
>
> Robert
>
> ----- Original Message -----
> From: "Kimbro Staken" <ks...@dbxmlgroup.com>
> To: <xi...@xml.apache.org>
> Sent: Monday, December 17, 2001 9:13 PM
> Subject: Re: memory leak
>
>
>>
>> On Monday, December 17, 2001, at 03:09 AM, Robert Spee wrote:
>>
>>> I'm using dbXML-Core-1.04b with the Java XML:DB API and I have the
>>> following
>>> problem:
>>>
>>> When I do getCollection() and collection.close() not all the memory is
>>> released and the databasse server runs out of memory.
>>>
>>> Does anyone knows the reason for this memory leak?
>>>
>>
>> I am not aware of any memory leaks in the beta 4 release. Can you
>> describe
>> more about what you're doing and where the error is occurring?
>>
>>> Gr
>>>
>>> Robert
>>>
>>>
>>>
>>>
Kimbro Staken
XML Database Software, Consulting and Writing
http://www.xmldatabases.org/
>>
>>
>
>
>
Re: memory leak
Posted by Robert Spee <qr...@oce.nl>.
This is the java program I have tried and shows the memory leak.
The problem is with the DatabaseManeger.getCollection() method.
----------------------------------
import org.xmldb.api.base.*;
import org.xmldb.api.modules.*;
import org.xmldb.api.DatabaseManager;
// For the dbXML specific CollectionManager service
import org.dbxml.client.xmldb.services.*;
public class test {
public static void main( String[] args ) throws Exception {
String vendor = "dbxml";
String drv = "org.dbxml.client.xmldb.DatabaseImpl";
String baseCol = "/db/";
String id= "test2";
String result;
String collectionName="docbase/xml";
Class c = Class.forName(drv);
Database db = (Database) c.newInstance();
DatabaseManager.registerDatabase(db);
for (int i=0;;i++) {
// if I put the next line (and the col.close() statement) outside the
for loop there is no memory leak
Collection col =
DatabaseManager.getCollection("xmldb:"+vendor+"://"+baseCol+collectionName);
System.out.println( "*** loop "+i+" ***");
Resource resource = col.getResource(id);
if (resource != null) {
result = (String) resource.getContent();
System.out.println(result);
} else {
System.out.println("ID: '"+id+"' does not exist.");
}
col.close();
}//for
} //main
} //class
---------------------------------
Hope anyone can help.
Gr
Robert
----- Original Message -----
From: "Kimbro Staken" <ks...@dbxmlgroup.com>
To: <xi...@xml.apache.org>
Sent: Monday, December 17, 2001 9:13 PM
Subject: Re: memory leak
>
> On Monday, December 17, 2001, at 03:09 AM, Robert Spee wrote:
>
> > I'm using dbXML-Core-1.04b with the Java XML:DB API and I have the
> > following
> > problem:
> >
> > When I do getCollection() and collection.close() not all the memory is
> > released and the databasse server runs out of memory.
> >
> > Does anyone knows the reason for this memory leak?
> >
>
> I am not aware of any memory leaks in the beta 4 release. Can you describe
> more about what you're doing and where the error is occurring?
>
> > Gr
> >
> > Robert
> >
> >
> >
> >
> Kimbro Staken
> XML Database Software, Consulting and Writing
> http://www.xmldatabases.org/
>
>
Re: memory leak
Posted by Kimbro Staken <ks...@dbxmlgroup.com>.
On Monday, December 17, 2001, at 03:09 AM, Robert Spee wrote:
> I'm using dbXML-Core-1.04b with the Java XML:DB API and I have the
> following
> problem:
>
> When I do getCollection() and collection.close() not all the memory is
> released and the databasse server runs out of memory.
>
> Does anyone knows the reason for this memory leak?
>
I am not aware of any memory leaks in the beta 4 release. Can you describe
more about what you're doing and where the error is occurring?
> Gr
>
> Robert
>
>
>
>
Kimbro Staken
XML Database Software, Consulting and Writing
http://www.xmldatabases.org/