You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Dominique Pfister <do...@gmail.com> on 2006/12/06 11:48:25 UTC
Derby Client Driver and LOBs
Hi,
I'm a developer in the Apache Jackrabbit project and in the process of
making it clusterable we're facing some serious problem: we have all
nodes in the cluster share the repository data and are using derby as
database back end, running standalone. When retrieving large binary
data through the derby client driver, I get OutOfMemoryErrors. The
resource paper "Derby Network Client" says:
"The client driver fully materializes LOBS when the row is retrieved"
which would of course explain those errors when the data gets too
large. Is this limitation being addressed? Or is there some other way
of retrieving binary data from a standalone derby database that
bypasses it?
Thanks
Dominique
Re: Derby Client Driver and LOBs
Posted by TomohitoNakayama <to...@basil.ocn.ne.jp>.
Oh...
Sorry.
I missed DERBY-1560 ....
Best regards.
Bryan Pendleton wrote:
> Dominique Pfister wrote:
>
>> I looked at the relevant parts of the derby client driver (version
>> 10.1.3.1) and quickly coded a solution for large objects exceeding
>> 1MB, using temporary files that are deleted as soon as the blob is no
>> longer referenced. If there is interest, I could further refine my
>> solution and submit a patch.
>
>
> That would be great! We're always looking for more contributors!
>
> I think that this issue may already be logged in JIRA as:
> http://issues.apache.org/jira/browse/DERBY-1560
>
> If you agree, you could assign yourself to the item and prepare your
> patch proposal as time permits.
>
> If possible, I think it would be preferable to work against a more
> recent version of the code, as 10.1.3.1 is somewhat out of date now.
>
> thanks,
>
> bryan
>
>
>
--
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
tmnk@apache.org
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
Re: Derby Client Driver and LOBs
Posted by Bryan Pendleton <bp...@amberpoint.com>.
Dominique Pfister wrote:
> I looked at the relevant parts of the derby client driver (version
> 10.1.3.1) and quickly coded a solution for large objects exceeding
> 1MB, using temporary files that are deleted as soon as the blob is no
> longer referenced. If there is interest, I could further refine my
> solution and submit a patch.
That would be great! We're always looking for more contributors!
I think that this issue may already be logged in JIRA as:
http://issues.apache.org/jira/browse/DERBY-1560
If you agree, you could assign yourself to the item and prepare your
patch proposal as time permits.
If possible, I think it would be preferable to work against a more
recent version of the code, as 10.1.3.1 is somewhat out of date now.
thanks,
bryan
Re: Derby Client Driver and LOBs
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Øystein Grøvlen wrote:
> Dominique Pfister wrote:
> > I looked at the relevant parts of the derby client driver (version
> > 10.1.3.1) and quickly coded a solution for large objects exceeding
> > 1MB, using temporary files that are deleted as soon as the blob is no
> > longer referenced. If there is interest, I could further refine my
> > solution and submit a patch.
>
> Such a solution has been considered, but I fear it is not a general
> solution. Generally, we cannot assume that we have permission to
> create files on the client side. Instead, we want to be able to
> stream the blob from the server without materializing it on the
> server.
>
I forgot to add that DERBY-1560 has been filed for using temporary files
for LOBs on the client. You are welcome to attach your patch to that
issue.
--
Øystein
Re: Derby Client Driver and LOBs
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Dominique Pfister wrote:
> I looked at the relevant parts of the derby client driver (version
> 10.1.3.1) and quickly coded a solution for large objects exceeding
> 1MB, using temporary files that are deleted as soon as the blob is no
> longer referenced. If there is interest, I could further refine my
> solution and submit a patch.
Such a solution has been considered, but I fear it is not a general
solution. Generally, we cannot assume that we have permission to
create files on the client side. Instead, we want to be able to
stream the blob from the server without materializing it on the
server.
--
Øystein
Re: Derby Client Driver and LOBs
Posted by Dominique Pfister <do...@gmail.com>.
Hello Tomohito,
thank you very much for your reply! Jackrabbit, the JSR-170 reference
implementation, uses an embedded derby database to store its content,
by default. In order to cluster several repository nodes, derby is
started standalone and all cluster nodes, possibly running on
different machines, access the same database. Binary large objects are
stored there as well, in a table that is created as follows:
create table BINVAL (BINVAL_ID varchar(1024) not null, BINVAL_DATA
blob(100M) not null)
We then retrieve binary data by a SQL clause such as the following:
select BINVAL_DATA from BINVAL where BINVAL_ID = ?
I looked at the relevant parts of the derby client driver (version
10.1.3.1) and quickly coded a solution for large objects exceeding
1MB, using temporary files that are deleted as soon as the blob is no
longer referenced. If there is interest, I could further refine my
solution and submit a patch.
Kind regards
Dominique
On 12/6/06, TomohitoNakayama <to...@basil.ocn.ne.jp> wrote:
> Hello.
>
> I'm afraid that your case is not addressed yet ...
>
> Several improvement around memory usage for LOB was achieved / is under
> work.
> But I think memory usage in NetworkClient driver when receiving LOB from
> NetworkServer is not improved yet.
>
> If you don't mind, could you tell us your case in more detail ?
> I want to file it as our task for improvement.
>
> Best regards.
>
>
> Dominique Pfister wrote:
>
> > Hi,
> >
> > I'm a developer in the Apache Jackrabbit project and in the process of
> > making it clusterable we're facing some serious problem: we have all
> > nodes in the cluster share the repository data and are using derby as
> > database back end, running standalone. When retrieving large binary
> > data through the derby client driver, I get OutOfMemoryErrors. The
> > resource paper "Derby Network Client" says:
> >
> > "The client driver fully materializes LOBS when the row is retrieved"
> >
> > which would of course explain those errors when the data gets too
> > large. Is this limitation being addressed? Or is there some other way
> > of retrieving binary data from a standalone derby database that
> > bypasses it?
> >
> > Thanks
> > Dominique
> >
> >
>
> --
> /*
>
> Tomohito Nakayama
> tomonaka@basil.ocn.ne.jp
> tomohito@rose.zero.ad.jp
> tmnk@apache.org
>
> Naka
> http://www5.ocn.ne.jp/~tomohito/TopPage.html
>
> */
>
>
Re: Derby Client Driver and LOBs
Posted by TomohitoNakayama <to...@basil.ocn.ne.jp>.
Hello.
I'm afraid that your case is not addressed yet ...
Several improvement around memory usage for LOB was achieved / is under
work.
But I think memory usage in NetworkClient driver when receiving LOB from
NetworkServer is not improved yet.
If you don't mind, could you tell us your case in more detail ?
I want to file it as our task for improvement.
Best regards.
Dominique Pfister wrote:
> Hi,
>
> I'm a developer in the Apache Jackrabbit project and in the process of
> making it clusterable we're facing some serious problem: we have all
> nodes in the cluster share the repository data and are using derby as
> database back end, running standalone. When retrieving large binary
> data through the derby client driver, I get OutOfMemoryErrors. The
> resource paper "Derby Network Client" says:
>
> "The client driver fully materializes LOBS when the row is retrieved"
>
> which would of course explain those errors when the data gets too
> large. Is this limitation being addressed? Or is there some other way
> of retrieving binary data from a standalone derby database that
> bypasses it?
>
> Thanks
> Dominique
>
>
--
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
tmnk@apache.org
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
Re: Derby Client Driver and LOBs
Posted by Bryan Pendleton <bp...@amberpoint.com>.
Øystein Grøvlen wrote:
> DERBY-208 is filed to address this.
Thanks Øystein ! I linked DERBY-1560 to DERBY-208 because
they seem closely related.
bryan
Re: Derby Client Driver and LOBs
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>
> "The client driver fully materializes LOBS when the row is retrieved"
>
> which would of course explain those errors when the data gets too
> large. Is this limitation being addressed? Or is there some other way
> of retrieving binary data from a standalone derby database that
> bypasses it?
DERBY-208 is filed to address this. Fernanda has started a discussion
on how to implement a locator-based implementation. Hopefully this can
be implemented for the next release.
--
Øystein