You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Grégoire Dubois <gr...@online.fr> on 2005/08/31 15:41:05 UTC
BLOB : java.lang.OutOfMemoryError
Hi all,
I'm using the org.apache.derby.jdbc.ClientDriver driver to access the
Derby database through network.
Writting small files (smaller than 5Mo) into the database works fine,
but I can't write big files (40Mo for example), without getting the
exception java.lang.OutOfMemoryError.
I think the driver is writting all the file into memory before sending
it to the database. Is that a known issue? Is there a plan to improve
this?
Thaks a lot.
Grégoire.
Re: BLOB : java.lang.OutOfMemoryError
Posted by "Michael J. Segel" <ms...@segel.com>.
On Wednesday 31 August 2005 13:32, Grégoire Dubois wrote:
> Perhaps it could help : the DBMS Mckoi ( http://mckoi.com/database/ )
> has got a JDBC driver that supports blob streaming over network. Code is
> open.
>
> I tried with IBM DB2, and same as Derby, streaming isn't supported with
> their JDBC driver. For a so big DBMS, I found it odd.
>
> I'm aiming to try Oracle, but I have some difficulties to use it. Does
> anyone knows if the Oracle JDBC supports blob streaming ?
>
Uhm going from memory, 8g did. (Wrote a Java API to insert XML insurance
claims in to an ODS...) However, I did have some problems. You have to use
the right JDBC driver and even then there was a memory leak...
> It would be difficult to implement in Derby ? I'm familiar with java
> network developpment. I develop in java for 4 years.
>
> Greg
>
I think it should be straight forward.
But then again, my mind is fried right now.
> > > Dan.
--
Michael Segel
Principal
MSCC
(312) 952-8175
Re: BLOB : java.lang.OutOfMemoryError
Posted by Grégoire Dubois <gr...@online.fr>.
Thank you.
Le vendredi 02 septembre 2005 à 11:08 -0700, Andrew McIntyre a écrit :
> On 9/1/05, Grégoire Dubois <gr...@online.fr> wrote:
> > Ok, could I have the "developper" privileges in Jira ?
> > I just created my account : greg
> >
> > Thank you
>
> Hi Grégoire,
>
> I've added you to the derby-developers list in JIRA.
>
> andrew
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Andrew McIntyre <mc...@gmail.com>.
On 9/1/05, Grégoire Dubois <gr...@online.fr> wrote:
> Ok, could I have the "developper" privileges in Jira ?
> I just created my account : greg
>
> Thank you
Hi Grégoire,
I've added you to the derby-developers list in JIRA.
andrew
Re: BLOB : java.lang.OutOfMemoryError
Posted by Grégoire Dubois <gr...@online.fr>.
Hi,
I logged the bug :
http://issues.apache.org/jira/browse/DERBY-550
Le mercredi 31 août 2005 à 14:27 -0700, Kathey Marsden a écrit :
> Kathey Marsden wrote:
>
> >Craig Russell wrote:
> >
> >
> >
> >>Of course, since we use DRDA, we would have to see if that protocol
> >>supports blob streaming...
> >>
> >>
> >
> >
> >Yes it does. The issue is with our writeStream methods they take a
> >length parameter which they use but don't actually write out. See the
> >issue DERBY-326 for details
> >
> >http://issues.apache.org/jira/browse/DERBY-326
> >
> >I am sending this to derby-dev with a bcc of derby-user so this thread
> >can get redirected to the development list where it belongs. If you
> >want to join this discussion, see you on derby-dev.
> >
> >
> >
> Actually I should qualify that I have only looked at the server side of
> this issue. When returning large objects from the server the memory
> consumption is caused by the need to know the length. Perhaps the
> client side issues are different.
>
> Grégoire, can you log a new bug with your specific reproducible case
> and link it to DERBY-326?
> That way your issue can be addressed directly.
>
> Kathey
>
>
>
>
>
>
>
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Grégoire Dubois <gr...@online.fr>.
Ok, could I have the "developper" privileges in Jira ?
I just created my account : greg
Thank you.
Le mercredi 31 août 2005 à 14:27 -0700, Kathey Marsden a écrit :
> Kathey Marsden wrote:
>
> >Craig Russell wrote:
> >
> >
> >
> >>Of course, since we use DRDA, we would have to see if that protocol
> >>supports blob streaming...
> >>
> >>
> >
> >
> >Yes it does. The issue is with our writeStream methods they take a
> >length parameter which they use but don't actually write out. See the
> >issue DERBY-326 for details
> >
> >http://issues.apache.org/jira/browse/DERBY-326
> >
> >I am sending this to derby-dev with a bcc of derby-user so this thread
> >can get redirected to the development list where it belongs. If you
> >want to join this discussion, see you on derby-dev.
> >
> >
> >
> Actually I should qualify that I have only looked at the server side of
> this issue. When returning large objects from the server the memory
> consumption is caused by the need to know the length. Perhaps the
> client side issues are different.
>
> Grégoire, can you log a new bug with your specific reproducible case
> and link it to DERBY-326?
> That way your issue can be addressed directly.
>
> Kathey
>
>
>
>
>
>
>
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Kathey Marsden <km...@sbcglobal.net>.
Kathey Marsden wrote:
>Craig Russell wrote:
>
>
>
>>Of course, since we use DRDA, we would have to see if that protocol
>>supports blob streaming...
>>
>>
>
>
>Yes it does. The issue is with our writeStream methods they take a
>length parameter which they use but don't actually write out. See the
>issue DERBY-326 for details
>
>http://issues.apache.org/jira/browse/DERBY-326
>
>I am sending this to derby-dev with a bcc of derby-user so this thread
>can get redirected to the development list where it belongs. If you
>want to join this discussion, see you on derby-dev.
>
>
>
Actually I should qualify that I have only looked at the server side of
this issue. When returning large objects from the server the memory
consumption is caused by the need to know the length. Perhaps the
client side issues are different.
Grégoire, can you log a new bug with your specific reproducible case
and link it to DERBY-326?
That way your issue can be addressed directly.
Kathey
Re: BLOB : java.lang.OutOfMemoryError
Posted by Kathey Marsden <km...@sbcglobal.net>.
Craig Russell wrote:
> Of course, since we use DRDA, we would have to see if that protocol
> supports blob streaming...
Yes it does. The issue is with our writeStream methods they take a
length parameter which they use but don't actually write out. See the
issue DERBY-326 for details
http://issues.apache.org/jira/browse/DERBY-326
I am sending this to derby-dev with a bcc of derby-user so this thread
can get redirected to the development list where it belongs. If you
want to join this discussion, see you on derby-dev.
Kathey
Re: BLOB : java.lang.OutOfMemoryError
Posted by Kathey Marsden <km...@sbcglobal.net>.
Craig Russell wrote:
> Of course, since we use DRDA, we would have to see if that protocol
> supports blob streaming...
Yes it does. The issue is with our writeStream methods they take a
length parameter which they use but don't actually write out. See the
issue DERBY-326 for details
http://issues.apache.org/jira/browse/DERBY-326
I am sending this to derby-dev with a bcc of derby-user so this thread
can get redirected to the development list where it belongs. If you
want to join this discussion, see you on derby-dev.
Kathey
Re: BLOB : java.lang.OutOfMemoryError
Posted by Craig Russell <Cr...@Sun.COM>.
Of course, since we use DRDA, we would have to see if that protocol
supports blob streaming...
Craig
On Aug 31, 2005, at 11:47 AM, David W. Van Couvering wrote:
> Again, yes, your help would be very much appreciated! Check out
> the code, get on derby-dev, ask questions, we'll be glad to help.
>
> David
>
> Grégoire Dubois wrote:
>
>
>> Perhaps it could help : the DBMS Mckoi ( http://mckoi.com/database/ )
>> has got a JDBC driver that supports blob streaming over network.
>> Code is
>> open.
>>
>> I tried with IBM DB2, and same as Derby, streaming isn't supported
>> with
>> their JDBC driver. For a so big DBMS, I found it odd.
>>
>> I'm aiming to try Oracle, but I have some difficulties to use it.
>> Does
>> anyone knows if the Oracle JDBC supports blob streaming ?
>>
>> It would be difficult to implement in Derby ? I'm familiar with java
>> network developpment. I develop in java for 4 years.
>>
>> Greg
>>
>> Le mercredi 31 août 2005 à 11:04 -0700, Satheesh Bandaram a écrit :
>>
>>
>>> Right... Network server currently sends whole datatype over to
>>> client.
>>> While this works for most datatypes, since they are limited in size,
>>> it may be a problem for clob/blob types. Server needs to be able to
>>> stream these large datatypes over the wire on demand, so no matter
>>> whether we have a common client or not, server needs to be enhanced.
>>>
>>> Satheesh
>>>
>>> Daniel John Debrunner wrote:
>>>
>>>> Michael J. Segel wrote:
>>>>
>>>>
>>>>
>>>>> On Wednesday 31 August 2005 10:49, Satheesh Bandaram wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Thanks for the offer.. It would really be great to have more
>>>>>> developpers
>>>>>> working on Derby. Join the derby-dev alias to participate in the
>>>>>> development.
>>>>>>
>>>>>> Derby embedded driver already has the capability to stream
>>>>>> blob/clob,
>>>>>> without requiring reading them completely into memory. It
>>>>>> would be great to
>>>>>> enhance Derby network server and Derby client to support this
>>>>>> kind of
>>>>>> behavior.
>>>>>>
>>>>>> Satheesh
>>>>>>
>>>>>>
>>>>>>
>>>>> [SNIP]
>>>>>
>>>>> This may be a dumb question...
>>>>>
>>>>> Maybe its hindsight, but why isn't the Server Framework either
>>>>> a subclass or an extension to the embedded driver? Someone had
>>>>> posted that the Network Driver utilized the embedded driver...
>>>>> (Or is that bathtub gin affecting my memory?)
>>>>>
>>>>> I guess it goes more to the point of why not have more focus on
>>>>> a "universal" driver?
>>>>>
>>>>>
>>>> My view
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/db-derby-dev/
>>>> 200508.mbox/%3c430A43DE.3060803@debrunners.com%3e
>>>>
>>>> Really the two drivers are doing different things, one has to
>>>> convert
>>>> requests into messages using DRDA over tcp/ip, the other just
>>>> calls java
>>>> methods.
>>>>
>>>> Dan.
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>> <david.vancouvering.vcf>
>
Craig Russell
Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
408 276-5638 mailto:Craig.Russell@sun.com
P.S. A good JDO? O, Gasp!
Re: BLOB : java.lang.OutOfMemoryError
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Again, yes, your help would be very much appreciated! Check out the
code, get on derby-dev, ask questions, we'll be glad to help.
David
Gr�goire Dubois wrote:
>Perhaps it could help : the DBMS Mckoi ( http://mckoi.com/database/ )
>has got a JDBC driver that supports blob streaming over network. Code is
>open.
>
>I tried with IBM DB2, and same as Derby, streaming isn't supported with
>their JDBC driver. For a so big DBMS, I found it odd.
>
>I'm aiming to try Oracle, but I have some difficulties to use it. Does
>anyone knows if the Oracle JDBC supports blob streaming ?
>
>It would be difficult to implement in Derby ? I'm familiar with java
>network developpment. I develop in java for 4 years.
>
>Greg
>
>Le mercredi 31 ao�t 2005 � 11:04 -0700, Satheesh Bandaram a �crit :
>
>
>>Right... Network server currently sends whole datatype over to client.
>>While this works for most datatypes, since they are limited in size,
>>it may be a problem for clob/blob types. Server needs to be able to
>>stream these large datatypes over the wire on demand, so no matter
>>whether we have a common client or not, server needs to be enhanced.
>>
>>Satheesh
>>
>>Daniel John Debrunner wrote:
>>
>>
>>>Michael J. Segel wrote:
>>>
>>>
>>>
>>>
>>>>On Wednesday 31 August 2005 10:49, Satheesh Bandaram wrote:
>>>>
>>>>
>>>>
>>>>
>>>>>Thanks for the offer.. It would really be great to have more developpers
>>>>>working on Derby. Join the derby-dev alias to participate in the
>>>>>development.
>>>>>
>>>>>Derby embedded driver already has the capability to stream blob/clob,
>>>>>without requiring reading them completely into memory. It would be great to
>>>>>enhance Derby network server and Derby client to support this kind of
>>>>>behavior.
>>>>>
>>>>>Satheesh
>>>>>
>>>>>
>>>>>
>>>>>
>>>>[SNIP]
>>>>
>>>>This may be a dumb question...
>>>>
>>>>Maybe its hindsight, but why isn't the Server Framework either a subclass or
>>>>an extension to the embedded driver? Someone had posted that the Network
>>>>Driver utilized the embedded driver... (Or is that bathtub gin affecting my
>>>>memory?)
>>>>
>>>>I guess it goes more to the point of why not have more focus on a "universal"
>>>>driver?
>>>>
>>>>
>>>>
>>>My view
>>>
>>>http://mail-archives.apache.org/mod_mbox/db-derby-dev/200508.mbox/%3c430A43DE.3060803@debrunners.com%3e
>>>
>>>Really the two drivers are doing different things, one has to convert
>>>requests into messages using DRDA over tcp/ip, the other just calls java
>>>methods.
>>>
>>>Dan.
>>>
>>>
>>>
>>>
>>>
>>>
>
>
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Grégoire Dubois wrote:
> Perhaps it could help : the DBMS Mckoi ( http://mckoi.com/database/ )
> has got a JDBC driver that supports blob streaming over network. Code is
> open.
Open but under a different licence (GPL). Copying code from another
project needs to be according to the terms of its licence. Since GPL is
viral such code copying could require that Derby code become GPL which
is not acceptable.
If any contributors are not the originator of any code, that fact needs
to be disclosed so that it can be determined if the code is acceptable
to the ASF.
Dan.
Re: BLOB : java.lang.OutOfMemoryError
Posted by Grégoire Dubois <gr...@online.fr>.
Perhaps it could help : the DBMS Mckoi ( http://mckoi.com/database/ )
has got a JDBC driver that supports blob streaming over network. Code is
open.
I tried with IBM DB2, and same as Derby, streaming isn't supported with
their JDBC driver. For a so big DBMS, I found it odd.
I'm aiming to try Oracle, but I have some difficulties to use it. Does
anyone knows if the Oracle JDBC supports blob streaming ?
It would be difficult to implement in Derby ? I'm familiar with java
network developpment. I develop in java for 4 years.
Greg
Le mercredi 31 août 2005 à 11:04 -0700, Satheesh Bandaram a écrit :
> Right... Network server currently sends whole datatype over to client.
> While this works for most datatypes, since they are limited in size,
> it may be a problem for clob/blob types. Server needs to be able to
> stream these large datatypes over the wire on demand, so no matter
> whether we have a common client or not, server needs to be enhanced.
>
> Satheesh
>
> Daniel John Debrunner wrote:
> > Michael J. Segel wrote:
> >
> >
> > > On Wednesday 31 August 2005 10:49, Satheesh Bandaram wrote:
> > >
> > >
> > > > Thanks for the offer.. It would really be great to have more developpers
> > > > working on Derby. Join the derby-dev alias to participate in the
> > > > development.
> > > >
> > > > Derby embedded driver already has the capability to stream blob/clob,
> > > > without requiring reading them completely into memory. It would be great to
> > > > enhance Derby network server and Derby client to support this kind of
> > > > behavior.
> > > >
> > > > Satheesh
> > > >
> > > >
> > > [SNIP]
> > >
> > > This may be a dumb question...
> > >
> > > Maybe its hindsight, but why isn't the Server Framework either a subclass or
> > > an extension to the embedded driver? Someone had posted that the Network
> > > Driver utilized the embedded driver... (Or is that bathtub gin affecting my
> > > memory?)
> > >
> > > I guess it goes more to the point of why not have more focus on a "universal"
> > > driver?
> > >
> >
> > My view
> >
> > http://mail-archives.apache.org/mod_mbox/db-derby-dev/200508.mbox/%3c430A43DE.3060803@debrunners.com%3e
> >
> > Really the two drivers are doing different things, one has to convert
> > requests into messages using DRDA over tcp/ip, the other just calls java
> > methods.
> >
> > Dan.
> >
> >
> >
> >
Re: BLOB : java.lang.OutOfMemoryError
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Michael J. Segel wrote:
> On Wednesday 31 August 2005 10:49, Satheesh Bandaram wrote:
>
>> Thanks for the offer.. It would really be great to have more developpers
>>working on Derby. Join the derby-dev alias to participate in the
>>development.
>>
>> Derby embedded driver already has the capability to stream blob/clob,
>>without requiring reading them completely into memory. It would be great to
>>enhance Derby network server and Derby client to support this kind of
>>behavior.
>>
>> Satheesh
>>
>
> [SNIP]
>
> This may be a dumb question...
>
> Maybe its hindsight, but why isn't the Server Framework either a subclass or
> an extension to the embedded driver? Someone had posted that the Network
> Driver utilized the embedded driver... (Or is that bathtub gin affecting my
> memory?)
>
> I guess it goes more to the point of why not have more focus on a "universal"
> driver?
My view
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200508.mbox/%3c430A43DE.3060803@debrunners.com%3e
Really the two drivers are doing different things, one has to convert
requests into messages using DRDA over tcp/ip, the other just calls java
methods.
Dan.
Re: BLOB : java.lang.OutOfMemoryError
Posted by David Van Couvering <Da...@Sun.COM>.
Ah, you have struck upon a nerve, the knee kicks up. If I were skilled
at finding old threads in our mailing list (which I'm not) I would show
you the discussion we'd be having on derby-dev about code sharing.
I will try to summarize, but I'm sure others will chime in:
* Currently the network client and the embedded driver and the
network server are all totally independent packages of code
(except for some shared constants)
* We have had numerous discussions about enabling more code sharing,
but it's trickier than you might think
* I think we are coalescing on making it a goal to incrementally
share more and more code. Two major areas: sharing the network
code between the embedded client and the network server, and
sharing the driver code between the embedded and network drivers.
Other areas include utility services that span these areas.
* Your participation in this effort would be most appreciated. So
far we haven't made much progress mostly because of lack of
resources (and also a lot of back and forth debate about exactly
how to do it).
Thanks,
David
Michael J. Segel wrote:
>On Wednesday 31 August 2005 10:49, Satheesh Bandaram wrote:
>
>
>> Thanks for the offer.. It would really be great to have more developpers
>>working on Derby. Join the derby-dev alias to participate in the
>>development.
>>
>> Derby embedded driver already has the capability to stream blob/clob,
>>without requiring reading them completely into memory. It would be great to
>>enhance Derby network server and Derby client to support this kind of
>>behavior.
>>
>> Satheesh
>>
>>
>>
>[SNIP]
>
>This may be a dumb question...
>
>Maybe its hindsight, but why isn't the Server Framework either a subclass or
>an extension to the embedded driver? Someone had posted that the Network
>Driver utilized the embedded driver... (Or is that bathtub gin affecting my
>memory?)
>
>I guess it goes more to the point of why not have more focus on a "universal"
>driver?
>
>
>
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Michael J. Segel wrote:
> [SNIP]
>
> This may be a dumb question...
>
> Maybe its hindsight, but why isn't the Server Framework either a subclass or
> an extension to the embedded driver? Someone had posted that the Network
> Driver utilized the embedded driver... (Or is that bathtub gin affecting my
> memory?)
Just to clarify an important point, the network *server* uses the
embedded driver. The network client(driver) does not interact with the
embedded driver. The stack looks something like this
application
jdbc
network client
drda over tcp/ip
network server
embedded driver
derby engine
Dan.
Re: BLOB : java.lang.OutOfMemoryError
Posted by "Michael J. Segel" <ms...@segel.com>.
On Wednesday 31 August 2005 10:49, Satheesh Bandaram wrote:
> Thanks for the offer.. It would really be great to have more developpers
> working on Derby. Join the derby-dev alias to participate in the
> development.
>
> Derby embedded driver already has the capability to stream blob/clob,
> without requiring reading them completely into memory. It would be great to
> enhance Derby network server and Derby client to support this kind of
> behavior.
>
> Satheesh
>
[SNIP]
This may be a dumb question...
Maybe its hindsight, but why isn't the Server Framework either a subclass or
an extension to the embedded driver? Someone had posted that the Network
Driver utilized the embedded driver... (Or is that bathtub gin affecting my
memory?)
I guess it goes more to the point of why not have more focus on a "universal"
driver?
--
Michael Segel
Principal
MSCC
(312) 952-8175
Re: BLOB : java.lang.OutOfMemoryError
Posted by David Van Couvering <Da...@Sun.COM>.
Your services would be most appreciated! Please use derby-dev to get
help and guidance on this.
David
Gr�goire Dubois wrote:
>Thank you for your answer.
>
>Can I propose my services for this developpement? Being able to write
>big BLOBs in the Derby through the JDBC driver is a very very important
>behavior for me.
>
>As Derby supports BLOBs with a capacity of max 2Gb, and knowing that
>almost nobody has got a computer with more than 2Gb RAM, is there an
>other way to write big BLOBs in the Derby ?
>
>Thank you very much.
>Gr�goire.
>
>Le mercredi 31 ao�t 2005 � 07:23 -0700, Kathey Marsden a �crit :
>
>
>>Gr�goire Dubois wrote:
>>
>>
>>
>>>Hi all,
>>>
>>>I'm using the org.apache.derby.jdbc.ClientDriver driver to access the
>>>Derby database through network.
>>>
>>>Writting small files (smaller than 5Mo) into the database works fine,
>>>but I can't write big files (40Mo for example), without getting the
>>>exception java.lang.OutOfMemoryError.
>>>
>>>I think the driver is writting all the file into memory before sending
>>>it to the database. Is that a known issue? Is there a plan to improve
>>>this?
>>>
>>>
>>>
>>>
>>>
>>This is DERBY-326. There is not currently a developer assigned to this
>>issue. Right now you would just have to increase the maximum heap size
>>of the jvm with -Xmx<size>
>>
>>Right now this is logged as an improvement but perhaps it would make
>>sense for you to log a Bug for your specific case and OutOfMemory error
>>and link it to this isssue. Then vote in Jira to have it fixed.
>>
>>
>>Kathey
>>
>>
>>
>>
>>
>
>
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Sunitha Kambhampati <ks...@gmail.com>.
Rajes Akkineni wrote:
> Hi,
>
> Dose that mean if i have 4 tables with BLOBs it will take up 120mb ~?
No, it does not depend on the tables in database. Page cache is per
database.
Sunitha.
<snip>
>
> On Thu, 01 Sep 2005 23:32:40 +0530, Sunitha Kambhampati
> <ks...@gmail.com> wrote:
>
>> Rajes Akkineni wrote:
>>
>>> Hi,
>>>
>>> I have got the similar problem.
>>
>>
>>> I was unable to insert bigger files in to the database.
>>> But my situation is little different than this. I am not inserting
>>> total 40mb file in one blob.
>>> I have split them in to 64kb chunks and inserting in multiple rows.
>>>
>>> I got outofmemory error.
>>
>>
>> Are you using embedded driver or the network client driver ? I
>> think the issue discussed in this earlier thread is referring to the
>> client driver. ( -DERBY-326). But if you are noticing the problem
>> with embedded, it would be great if you could post a reproduction
>> program, so we can take a look.
>>
>> I have tested inserting 1000 rows of 64kb blobs on my T40 laptop and
>> it works ok both with default jvm heap size and also with
>> restricting the jvm max heap size to 40mb. Note by default , for a
>> table containing blob column, page size is 32k and page cache is
>> 1000 pages, so the pagecache would take about 32mb of memory.
>>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Rajes Akkineni <ra...@vimukti.com>.
Hi,
Dose that mean if i have 4 tables with BLOBs it will take up 120mb ~?
I have made a testing program..but as you said it worked fine.
But in my product the situation is much complex. May be due to that is it
now taking more memory.
The environment is...
1. Derby is used through as eclipse plugin
2. It is beeing used from two different plugins to create around 20
(15,5)tables.
3. I have make a static class to do all database operations with
syncronized methods which are beeing accessed from different diff threads.
4. There are 5 tables with BLOB columns out of which ony one table is
beeing used.
5. As JXTA is beeing used i think it takes most of the memory. (I dont
know how to mesure the heap usage)
Here are the my tables and the select query which is giving problem with
BLOB and later with the data(not able to retrive which i have inserted).
TABLE CREATION STATEMENT
--------------------------
"CREATE TABLE dataQueue(messageId BIGINT NOT NULL, objectId VARCHAR(100)
,data BLOB, sentId BIGINT, sequenceNo INTEGER,inoutstate SMALLINT, PRIMARY
KEY(objectId,sequenceNo,inoutstate))"
"CREATE TABLE objectQueue(objectId VARCHAR(100), source
VARCHAR(100),destination VARCHAR(100), priority SMALLINT, type SMALLINT,
size INTEGER, splitCount INTEGER, date DATE, isGroup SMALLINT,inoutstate
SMALLINT)"
"CREATE TABLE groups(groupId VARCHAR(100),identityId VARCHAR(100), sentId
BIGINT)"
SELECT QUERY
-------------
SELECT
dq.messageId,dq.objectId,dq.data,dq.sentId,dq.sequenceNo,oq.source,oq.destination,oq.priority,oq.type,oq.size,oq.splitCount,oq.date,oq.isGroup
from dataQueue as dq,objectQueue as oq,groups as g where
((dq.objectId=oq.objectId) AND ((oq.destination=?) OR (oq.destination
=g.groupid AND g.identityId=?)) AND ((dq.sentId=0) OR (dq.sentId >
g.sentId)) AND (oq.inoutstate=0)) ORDER BY
oq.priority,dq.sentId,dq.sequenceNo"
Is there some thing very stupid thing that i am doing here?
This is my second database dependednt application. So i dont have much
experiance.
Thanks
Rajesh
On Thu, 01 Sep 2005 23:32:40 +0530, Sunitha Kambhampati
<ks...@gmail.com> wrote:
> Rajes Akkineni wrote:
>
>> Hi,
>>
>> I have got the similar problem.
>
>> I was unable to insert bigger files in to the database.
>> But my situation is little different than this. I am not inserting
>> total 40mb file in one blob.
>> I have split them in to 64kb chunks and inserting in multiple rows.
>>
>> I got outofmemory error.
>
> Are you using embedded driver or the network client driver ? I think
> the issue discussed in this earlier thread is referring to the client
> driver. ( -DERBY-326). But if you are noticing the problem with
> embedded, it would be great if you could post a reproduction program,
> so we can take a look.
>
> I have tested inserting 1000 rows of 64kb blobs on my T40 laptop and it
> works ok both with default jvm heap size and also with restricting the
> jvm max heap size to 40mb. Note by default , for a table containing blob
> column, page size is 32k and page cache is 1000 pages, so the pagecache
> would take about 32mb of memory.
>
>>
>> Now i seem to have another issue with the DERBY.
>> when i am Inserting, Select data using prepared statements(they are
>> all created once, and used from different different threads later)
>> some time i am not able to get the data which i have inserted. i
>> checked the insert operation. it is successfull but later when i query
>> from different thread it is not showing any rows in the table.
>
> Are you running with autocommit off, if so might be good to check if a
> commit was given after the insert or not.
>
> Sunitha.
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: BLOB : java.lang.OutOfMemoryError
Posted by Grégoire Dubois <gr...@online.fr>.
It works great with the embedded driver. The problem comes with the
network client driver.
I wrote a log for the bug : ( -DERBY-550)
I'll watch the 326.
Le jeudi 01 septembre 2005 à 11:02 -0700, Sunitha Kambhampati a écrit :
> Rajes Akkineni wrote:
>
> > Hi,
> >
> > I have got the similar problem.
>
> > I was unable to insert bigger files in to the database.
> > But my situation is little different than this. I am not inserting
> > total 40mb file in one blob.
> > I have split them in to 64kb chunks and inserting in multiple rows.
> >
> > I got outofmemory error.
>
> Are you using embedded driver or the network client driver ? I think
> the issue discussed in this earlier thread is referring to the client
> driver. ( -DERBY-326). But if you are noticing the problem with
> embedded, it would be great if you could post a reproduction program,
> so we can take a look.
>
> I have tested inserting 1000 rows of 64kb blobs on my T40 laptop and it
> works ok both with default jvm heap size and also with restricting the
> jvm max heap size to 40mb. Note by default , for a table containing blob
> column, page size is 32k and page cache is 1000 pages, so the pagecache
> would take about 32mb of memory.
>
> >
> > Now i seem to have another issue with the DERBY.
> > when i am Inserting, Select data using prepared statements(they are
> > all created once, and used from different different threads later)
> > some time i am not able to get the data which i have inserted. i
> > checked the insert operation. it is successfull but later when i
> > query from different thread it is not showing any rows in the table.
>
> Are you running with autocommit off, if so might be good to check if a
> commit was given after the insert or not.
>
> Sunitha.
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Sunitha Kambhampati <ks...@gmail.com>.
Rajes Akkineni wrote:
> Hi,
>
> I have got the similar problem.
> I was unable to insert bigger files in to the database.
> But my situation is little different than this. I am not inserting
> total 40mb file in one blob.
> I have split them in to 64kb chunks and inserting in multiple rows.
>
> I got outofmemory error.
Are you using embedded driver or the network client driver ? I think
the issue discussed in this earlier thread is referring to the client
driver. ( -DERBY-326). But if you are noticing the problem with
embedded, it would be great if you could post a reproduction program,
so we can take a look.
I have tested inserting 1000 rows of 64kb blobs on my T40 laptop and it
works ok both with default jvm heap size and also with restricting the
jvm max heap size to 40mb. Note by default , for a table containing blob
column, page size is 32k and page cache is 1000 pages, so the pagecache
would take about 32mb of memory.
>
> Now i seem to have another issue with the DERBY.
> when i am Inserting, Select data using prepared statements(they are
> all created once, and used from different different threads later)
> some time i am not able to get the data which i have inserted. i
> checked the insert operation. it is successfull but later when i
> query from different thread it is not showing any rows in the table.
Are you running with autocommit off, if so might be good to check if a
commit was given after the insert or not.
Sunitha.
Re: BLOB : java.lang.OutOfMemoryError
Posted by Rajes Akkineni <ra...@vimukti.com>.
Hi,
I have got the similar problem.
I was unable to insert bigger files in to the database.
But my situation is little different than this. I am not inserting total
40mb file in one blob.
I have split them in to 64kb chunks and inserting in multiple rows.
I got outofmemory error.
Then i switched to writing storing that blob in a file.
We will also be must interested in solving this problem.
Now i seem to have another issue with the DERBY.
when i am Inserting, Select data using prepared statements(they are all
created once, and used from different different threads later) some time i
am not able to get the data which i have inserted. i checked the insert
operation. it is successfull but later when i query from different thread
it is not showing any rows in the table.
Did any one experianced this problem.
We are a team of programmers with 2 years of programming experiance. We
are not sure if we can fix the problem in the derby. Please suggest us
what to do.
This problem is stopping our project. We dont have any alternative to
derby at present(due to the work done, platform,licence). Should we try to
fix the problem in the derby?
We are trying to write a program to simulate this bug.
prepared statements are beeing used from threads created by JXTA. We are
not aware if there can be anything specific to those threads.
Thanks
Rajesh
On Wed, 31 Aug 2005 21:19:42 +0530, Satheesh Bandaram
<sa...@Sourcery.Org> wrote:
> Thanks for the offer.. It would really be great to have more developpers
> working
> on Derby. Join the derby-dev alias to participate in the development.
>
> Derby embedded driver already has the capability to stream blob/clob,
> without
> requiring reading them completely into memory. It would be great to
> enhance
> Derby network server and Derby client to support this kind of behavior.
>
> Satheesh
>
> Grégoire Dubois wrote:
>
> Thank you for your answer.
>
> Can I propose my services for this developpement? Being able to write
> big BLOBs in the Derby through the JDBC driver is a very very important
> behavior for me.
>
> As Derby supports BLOBs with a capacity of max 2Gb, and knowing that
> almost nobody has got a computer with more than 2Gb RAM, is there an
> other way to write big BLOBs in the Derby ?
>
> Thank you very much.
> Grégoire.
>
> Le mercredi 31 août 2005 à 07:23 -0700, Kathey Marsden a écrit :
>
> Grégoire Dubois wrote:
>
>
> Hi all,
>
> I'm using the org.apache.derby.jdbc.ClientDriver driver to access the
> Derby database through network.
>
> Writting small files (smaller than 5Mo) into the database works fine,
> but I can't write big files (40Mo for example), without getting the
> exception java.lang.OutOfMemoryError.
>
> I think the driver is writting all the file into memory before sending
> it to the database. Is that a known issue? Is there a plan to improve
> this?
>
>
>
> This is DERBY-326. There is not currently a developer assigned to this
> issue. Right now you would just have to increase the maximum heap size
> of the jvm with -Xmx<size>
>
> Right now this is logged as an improvement but perhaps it would make
> sense for you to log a Bug for your specific case and OutOfMemory error
> and link it to this isssue. Then vote in Jira to have it fixed.
>
>
> Kathey
>
>
>
>
>
>
>
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: BLOB : java.lang.OutOfMemoryError
Posted by Grégoire Dubois <gr...@online.fr>.
Thank you for your answer.
Can I propose my services for this developpement? Being able to write
big BLOBs in the Derby through the JDBC driver is a very very important
behavior for me.
As Derby supports BLOBs with a capacity of max 2Gb, and knowing that
almost nobody has got a computer with more than 2Gb RAM, is there an
other way to write big BLOBs in the Derby ?
Thank you very much.
Grégoire.
Le mercredi 31 août 2005 à 07:23 -0700, Kathey Marsden a écrit :
> Grégoire Dubois wrote:
>
> >Hi all,
> >
> >I'm using the org.apache.derby.jdbc.ClientDriver driver to access the
> >Derby database through network.
> >
> >Writting small files (smaller than 5Mo) into the database works fine,
> >but I can't write big files (40Mo for example), without getting the
> >exception java.lang.OutOfMemoryError.
> >
> >I think the driver is writting all the file into memory before sending
> >it to the database. Is that a known issue? Is there a plan to improve
> >this?
> >
> >
> >
> This is DERBY-326. There is not currently a developer assigned to this
> issue. Right now you would just have to increase the maximum heap size
> of the jvm with -Xmx<size>
>
> Right now this is logged as an improvement but perhaps it would make
> sense for you to log a Bug for your specific case and OutOfMemory error
> and link it to this isssue. Then vote in Jira to have it fixed.
>
>
> Kathey
>
>
>
Re: BLOB : java.lang.OutOfMemoryError
Posted by Kathey Marsden <km...@sbcglobal.net>.
Grégoire Dubois wrote:
>Hi all,
>
>I'm using the org.apache.derby.jdbc.ClientDriver driver to access the
>Derby database through network.
>
>Writting small files (smaller than 5Mo) into the database works fine,
>but I can't write big files (40Mo for example), without getting the
>exception java.lang.OutOfMemoryError.
>
>I think the driver is writting all the file into memory before sending
>it to the database. Is that a known issue? Is there a plan to improve
>this?
>
>
>
This is DERBY-326. There is not currently a developer assigned to this
issue. Right now you would just have to increase the maximum heap size
of the jvm with -Xmx<size>
Right now this is logged as an improvement but perhaps it would make
sense for you to log a Bug for your specific case and OutOfMemory error
and link it to this isssue. Then vote in Jira to have it fixed.
Kathey