You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Kasper Nielsen <ne...@kav.dk> on 2006/11/24 10:43:17 UTC
Using Derby as a binary store
Hi,
I'm evaluating Derby for a project of mine. I really would like to avoid
using SQL and just use Derby as a binary store. Is this possible?
My data consists of key -> value pairs where the keys are uniformly
distributed 160 bit numbers and the values are byte arrays.
For now, I need to support 2 basic queries
* Give me the value mapping for the key K or null if it doesn't exist.
* Give me a map of (key->value) for the key range of key_start to key_end.
I had a look at the org.apache.derby.iap.store packages but wasn't quite
sure where to start.
Cheers
Kasper
Re: Using Derby as a binary store
Posted by David Van Couvering <da...@vancouvering.com>.
Ah, I thought it might be the license :)
Well, there have been some rumblings from time to time about making the
store API public, but so far nothing has come of it.
You say "SQL gets in the way." How do you know that's the true
resource consumer for a key/value use of Derby? It would seem to me
that if all your tables are basically primary key/data, then all your
queries would be based on the primary key, and after the first use of
your prepared statements the resulting compiled queries should be pretty
darn efficient, especially when those bytecodes get compiled into native
code by the JIT compiler.
Derby is actually quite fast for basic primary key operations.
I would recommend putting together a performance test and see if your
concerns are justified (or have you already done this?). Maybe Derby
is good enough for your needs...?
David
Kasper Nielsen wrote:
> David Van Couvering wrote:
>> To be honest, this looks more like a job for BDB than for Derby. I
>> would love to see the Derby store API made public at some point, but
>> currently it's not public and I suspect it may be difficult to work
>> with. Are there reasons BDB/Java won't work for you?
>
> Not besides the license:)
>
> Its for an BSD licensed project, so BDB is pretty much out of the picture.
>
> And since I haven't been able to find any projects resembling BDB my
> hope was that I could use Derby with some minor tweaking.
>
> - Kasper
>
>> David
>>
>> Kasper Nielsen wrote:
>>> Julius.Stroffek wrote:
>>>> Do you need to store them also on a disk?
>>>>
>>>> Cheers
>>>>
>>>> Julo
>>> Yes,
>>>
>>> I'm only interested in the actual storage/indexing. Any caching I'll
>>> take of myself.
>>>
>>> Let's for simplicity's sake say I need the following interface in my
>>> application.
>>>
>>> interface Store {
>>> void put(byte[] key, byte[] value);
>>> byte[] get(byte[] key);
>>> Map<byte[], byte[]> getRange(byte[] key_from, byte[] key_to);
>>> }
>>>
>>> put and get works like the java.util.Map put/get operation.
>>> getRange(83,103) would return all the existing key->value bindings
>>> for numbers between 83 and 103.
>>>
>>> I could implement this myself, but I would like to use the
>>> indexing/transactional features of Derby instead. However, I'm not
>>> willing to pay the performance cost for using SQL since I'm only
>>> storing byte[] arrays (if it is possible).
>>>
>>> Cheers
>>> Kasper
>>>
>>>>
>>>> Kasper Nielsen wrote:
>>>>> I'm just looking for a lightweight solution where I can store bytes
>>>>> (key->value pairs) and create a few indexes on the data.
>>>>>
>>>>>
>>>>> Cheers
>>>>> Kasper
>>>>>
>>>>>
>>>>>> On 11/24/06, Kasper Nielsen <ne...@kav.dk> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm evaluating Derby for a project of mine. I really would like
>>>>>>> to avoid
>>>>>>> using SQL and just use Derby as a binary store. Is this possible?
>>>>>>>
>>>>>>> My data consists of key -> value pairs where the keys are uniformly
>>>>>>> distributed 160 bit numbers and the values are byte arrays.
>>>>>>>
>>>>>>> For now, I need to support 2 basic queries
>>>>>>> * Give me the value mapping for the key K or null if it doesn't
>>>>>>> exist.
>>>>>>> * Give me a map of (key->value) for the key range of key_start
>>>>>>> to key_end.
>>>>>>>
>>>>>>> I had a look at the org.apache.derby.iap.store packages but
>>>>>>> wasn't quite
>>>>>>> sure where to start.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Kasper
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
BDB license (was: Using Derby as a binary store)
Posted by Kasper Nielsen <ne...@kav.dk>.
Dyre.Tjeldvoll@Sun.COM wrote:
> Kasper Nielsen <ne...@kav.dk> writes:
>
>> David Van Couvering wrote:
>>> To be honest, this looks more like a job for BDB than for Derby. I
>>> would love to see the Derby store API made public at some point, but
>>> currently it's not public and I suspect it may be difficult to work
>>> with. Are there reasons BDB/Java won't work for you?
>> Not besides the license:)
>>
>> Its for an BSD licensed project, so BDB is pretty much out of the picture.
>
> According to David's recent blog
>
> http://weblogs.java.net/blog/davidvc/archive/2006/11/oracle_benchmar_1.html
>
> "The BDB license is clear: if your product that uses BDB is
> closed-source, or you want indemnification, then you need to us pony
> up $$$ for the commercial license. If your product is open source, you
> can use the BDB open source license, which is a variant of a BSD
> license."
>
> So if your project is BSD also, shouldn't you be able to use BDB in
> it?
The BDB license is a viral license much like GPL and in fact was
designed to be identical in effect with the GPL. But for some historical
reasons they didn't choose GPL. So if you include BDB in your library.
Anybody using your library would need to make their application open
source.
Now, I'm no expert on licenses. But wouldn't the following scenario be
allowed? In fact I don't see any difference compared to how you would
use MySQL or PostgreSQL from within a Java application.
Launch a process with a standalone BDB database with a small (open
sourced) network layer on top. Then launch your application (in another
process) which would use a another small network layer to communicate
with the BDB process. There would probably be some overhead involved
with using local socket communication. But probably not much compared to
disk latency.
Of course the application would require two VM instances but for some
applications that might not be a problem. Perhaps you could even launch
the BDB database from within your own program. So users wouldn't need to
start both manually.
Cheers
Kasper
Re: Using Derby as a binary store
Posted by Dy...@Sun.COM.
Kasper Nielsen <ne...@kav.dk> writes:
> David Van Couvering wrote:
>> To be honest, this looks more like a job for BDB than for Derby. I
>> would love to see the Derby store API made public at some point, but
>> currently it's not public and I suspect it may be difficult to work
>> with. Are there reasons BDB/Java won't work for you?
>
> Not besides the license:)
>
> Its for an BSD licensed project, so BDB is pretty much out of the picture.
According to David's recent blog
http://weblogs.java.net/blog/davidvc/archive/2006/11/oracle_benchmar_1.html
"The BDB license is clear: if your product that uses BDB is
closed-source, or you want indemnification, then you need to us pony
up $$$ for the commercial license. If your product is open source, you
can use the BDB open source license, which is a variant of a BSD
license."
So if your project is BSD also, shouldn't you be able to use BDB in
it?
--
dt
Re: Using Derby as a binary store
Posted by Daniel Noll <da...@nuix.com>.
Paul J DeCoursey wrote:
> What is this binary data you are storing? Have you looked at
> Jackrabbit? Have you thought about just using java.nio and perhaps
> Lucene fro indexing? Definitely consider the above questions as well,
> look at the bigger picture.
Actually we were initially using plain files to store the data. It
turned out to be more efficient to use SQL to store them because the
glut of smaller files caused issues on NTFS.
But I consider NTFS to be a legacy filesystem. On modern filesystems
like ReiserFS, it would probably be quite efficient.
Daniel
--
Daniel Noll
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://nuix.com/ Fax: +61 2 9212 6902
This message is intended only for the named recipient. If you are not
the intended recipient you are notified that disclosing, copying,
distributing or taking any action in reliance on the contents of this
message or attachment is strictly prohibited.
Re: Using Derby as a binary store
Posted by Paul J DeCoursey <pa...@decoursey.net>.
Mike wrote:
>
>> -----Original Message-----
>> From: Paul J DeCoursey [mailto:paul@decoursey.net]
>> Sent: Tuesday, November 28, 2006 7:45 AM
>> To: Derby Discussion
>> Subject: Re: Using Derby as a binary store
>>
>> derby@segel.com wrote:
>>
>>> [mjs]
>>> The simplest path would be to write his own from scratch.
>>> The questions that haven't been asked:
>>> 1) Are there other pieces of the application that may benefit from being
>>>
>> in
>>
>>> a relational database.
>>> 2) What is the use case of the application? And what are the future
>>> enhancements that are already planned?
>>>
>>>
>>> That should be your driving reason for using a database. Choosing a
>>>
>> database
>>
>>> that doesn't support the data type you wish to use would mean 1) Either
>>>
>> a
>>
>>> rethink of your application to use a different solution so that it would
>>>
>> map
>>
>>> to the database better. Or 2) Choose a more appropriate storage model
>>>
>> What is this binary data you are storing? Have you looked at
>> Jackrabbit? Have you thought about just using java.nio and perhaps
>> Lucene fro indexing? Definitely consider the above questions as well,
>> look at the bigger picture.
>>
>> Paul
>>
>
> Paul,
> It wasn't my question.
>
> I was responding to the earlier thread.
>
> Hand coding or using freely available tools is up to the user and their
> personal preference.
>
> For example, I wrote my own calendar DB and surrounding app. I could have
> used one that was freely available, but since I wanted a custom look and
> feel, plus my own hooks in to my app framework, it was just as easy for me
> to do this on my own.
>
> When designing custom apps, its sometimes easier, efficient, and frankly
> more fun when you develop your own code, rather than try to make a "free"
> package fit your needs.
>
> The question at hand wasn't about which other off topic tool to choose, but
> whether or not to choose derby.
>
> Since Derby doesn't support the Boolean data type and he doesn't see value
> elsewhere in his app for a relational database store, then he should look
> elsewhere.
>
>
>
>
>
>
Mike, I wasn't responding to you... I was refering to your response for
the original poster, sorry if I did not make that clear.
RE: Using Derby as a binary store
Posted by de...@segel.com.
Hey!
Sorry about that.
For some reason my Thinkpad's outlook got a bit garbled on thinking that my
outbound SMTP server required an SSL. (It doesn't. It will only accept
connections from within my network.)
So This got sent as I unclogged it.
Please ignore cause this thread is old....
-Mike
> -----Original Message-----
> From: Mike [mailto:msegel@segel.com]
> Sent: Tuesday, November 28, 2006 8:12 AM
> To: 'Derby Discussion'
> Subject: RE: Using Derby as a binary store
>
>
>
> > -----Original Message-----
> > From: Paul J DeCoursey [mailto:paul@decoursey.net]
> > Sent: Tuesday, November 28, 2006 7:45 AM
> > To: Derby Discussion
> > Subject: Re: Using Derby as a binary store
> >
> > derby@segel.com wrote:
> > > [mjs]
> > > The simplest path would be to write his own from scratch.
> > > The questions that haven't been asked:
> > > 1) Are there other pieces of the application that may benefit from
> being
> > in
> > > a relational database.
> > > 2) What is the use case of the application? And what are the future
> > > enhancements that are already planned?
> > >
> > >
> > > That should be your driving reason for using a database. Choosing a
> > database
> > > that doesn't support the data type you wish to use would mean 1)
> Either
> > a
> > > rethink of your application to use a different solution so that it
> would
> > map
> > > to the database better. Or 2) Choose a more appropriate storage model
> > What is this binary data you are storing? Have you looked at
> > Jackrabbit? Have you thought about just using java.nio and perhaps
> > Lucene fro indexing? Definitely consider the above questions as well,
> > look at the bigger picture.
> >
> > Paul
>
> Paul,
> It wasn't my question.
>
> I was responding to the earlier thread.
>
> Hand coding or using freely available tools is up to the user and their
> personal preference.
>
> For example, I wrote my own calendar DB and surrounding app. I could have
> used one that was freely available, but since I wanted a custom look and
> feel, plus my own hooks in to my app framework, it was just as easy for me
> to do this on my own.
>
> When designing custom apps, its sometimes easier, efficient, and frankly
> more fun when you develop your own code, rather than try to make a "free"
> package fit your needs.
>
> The question at hand wasn't about which other off topic tool to choose,
> but
> whether or not to choose derby.
>
> Since Derby doesn't support the Boolean data type and he doesn't see value
> elsewhere in his app for a relational database store, then he should look
> elsewhere.
>
>
>
RE: Using Derby as a binary store
Posted by Mike <ms...@segel.com>.
> -----Original Message-----
> From: Paul J DeCoursey [mailto:paul@decoursey.net]
> Sent: Tuesday, November 28, 2006 7:45 AM
> To: Derby Discussion
> Subject: Re: Using Derby as a binary store
>
> derby@segel.com wrote:
> > [mjs]
> > The simplest path would be to write his own from scratch.
> > The questions that haven't been asked:
> > 1) Are there other pieces of the application that may benefit from being
> in
> > a relational database.
> > 2) What is the use case of the application? And what are the future
> > enhancements that are already planned?
> >
> >
> > That should be your driving reason for using a database. Choosing a
> database
> > that doesn't support the data type you wish to use would mean 1) Either
> a
> > rethink of your application to use a different solution so that it would
> map
> > to the database better. Or 2) Choose a more appropriate storage model
> What is this binary data you are storing? Have you looked at
> Jackrabbit? Have you thought about just using java.nio and perhaps
> Lucene fro indexing? Definitely consider the above questions as well,
> look at the bigger picture.
>
> Paul
Paul,
It wasn't my question.
I was responding to the earlier thread.
Hand coding or using freely available tools is up to the user and their
personal preference.
For example, I wrote my own calendar DB and surrounding app. I could have
used one that was freely available, but since I wanted a custom look and
feel, plus my own hooks in to my app framework, it was just as easy for me
to do this on my own.
When designing custom apps, its sometimes easier, efficient, and frankly
more fun when you develop your own code, rather than try to make a "free"
package fit your needs.
The question at hand wasn't about which other off topic tool to choose, but
whether or not to choose derby.
Since Derby doesn't support the Boolean data type and he doesn't see value
elsewhere in his app for a relational database store, then he should look
elsewhere.
Re: Using Derby as a binary store
Posted by Paul J DeCoursey <pa...@decoursey.net>.
derby@segel.com wrote:
> [mjs]
> The simplest path would be to write his own from scratch.
> The questions that haven't been asked:
> 1) Are there other pieces of the application that may benefit from being in
> a relational database.
> 2) What is the use case of the application? And what are the future
> enhancements that are already planned?
>
>
> That should be your driving reason for using a database. Choosing a database
> that doesn't support the data type you wish to use would mean 1) Either a
> rethink of your application to use a different solution so that it would map
> to the database better. Or 2) Choose a more appropriate storage model
What is this binary data you are storing? Have you looked at
Jackrabbit? Have you thought about just using java.nio and perhaps
Lucene fro indexing? Definitely consider the above questions as well,
look at the bigger picture.
Paul
RE: Using Derby as a binary store
Posted by de...@segel.com.
> -----Original Message-----
> From: Dyre.Tjeldvoll@Sun.COM [mailto:Dyre.Tjeldvoll@Sun.COM]
> Sent: Tuesday, November 28, 2006 3:58 AM
> To: Derby Discussion
> Subject: Re: Using Derby as a binary store
>
> Kasper Nielsen <ne...@kav.dk> writes:
>
> > David Van Couvering wrote:
> >> To be honest, this looks more like a job for BDB than for Derby. I
> >> would love to see the Derby store API made public at some point, but
> >> currently it's not public and I suspect it may be difficult to work
> >> with. Are there reasons BDB/Java won't work for you?
> >
> > Not besides the license:)
>
> It's a shame isn't it?
>
> > Its for an BSD licensed project, so BDB is pretty much out of the
> picture.
> >
> > And since I haven't been able to find any projects resembling BDB my
> > hope was that I could use Derby with some minor tweaking.
>
> I think you can safely assume that it will be MAJOR tweaking. Or
> perhaps we should say "a considerable effort". Perhaps to the point
> where it is easier to write what you need from scratch.
>
> Aren't there some non-transactional/non-db based persistence
> frameworks out there? Maybe you could implement what you need on top
> of them.
>
[mjs]
Scalability is going to be an issue then.
You are correct. It would be easier to write what he needs from scratch.
> I agree with David that you should really try using the SQL interface
> and benchmark it before you conclude that you cannot afford it. For
> the use cases you describe the SQL should not be that difficult, and
> you should be able to hide it in library of methods that the rest of
> your code can use (there are frameworks that will do that for you,
> e.g. IBATIS).
>
> If you make sure that those queries are prepared only once and then
> reused, and that they use indices sensibly, this should perform well.
>
> If you care about performance you should think twice about using
> frameworks that promise to "do all the hard work" and generate the
> SQL for you. There have been countless examples of such
> frameworks (I won't name any) producing completely brain-dead SQL...
>
[mjs]
Yup.
> >>> indexing/transactional features of Derby instead. However, I'm not
> >>> willing to pay the performance cost for using SQL since I'm only
> >>> storing byte[] arrays (if it is possible).
>
> Try it! Write a simple app, or use the simple client found under
> DERBY-1961 and profile it. The cost of compiling the SQL is only done
> once and "amortized" across the uses the prepared statement.
>
> Maybe you can modify the DERBY-1961 to use the data type need. Then
> you can see if the performance is acceptable...
>
> --
> dt
[mjs]
The simplest path would be to write his own from scratch.
The questions that haven't been asked:
1) Are there other pieces of the application that may benefit from being in
a relational database.
2) What is the use case of the application? And what are the future
enhancements that are already planned?
That should be your driving reason for using a database. Choosing a database
that doesn't support the data type you wish to use would mean 1) Either a
rethink of your application to use a different solution so that it would map
to the database better. Or 2) Choose a more appropriate storage model.
Re: Using Derby as a binary store
Posted by Dy...@Sun.COM.
Kasper Nielsen <ne...@kav.dk> writes:
> David Van Couvering wrote:
>> To be honest, this looks more like a job for BDB than for Derby. I
>> would love to see the Derby store API made public at some point, but
>> currently it's not public and I suspect it may be difficult to work
>> with. Are there reasons BDB/Java won't work for you?
>
> Not besides the license:)
It's a shame isn't it?
> Its for an BSD licensed project, so BDB is pretty much out of the picture.
>
> And since I haven't been able to find any projects resembling BDB my
> hope was that I could use Derby with some minor tweaking.
I think you can safely assume that it will be MAJOR tweaking. Or
perhaps we should say "a considerable effort". Perhaps to the point
where it is easier to write what you need from scratch.
Aren't there some non-transactional/non-db based persistence
frameworks out there? Maybe you could implement what you need on top
of them.
I agree with David that you should really try using the SQL interface
and benchmark it before you conclude that you cannot afford it. For
the use cases you describe the SQL should not be that difficult, and
you should be able to hide it in library of methods that the rest of
your code can use (there are frameworks that will do that for you,
e.g. IBATIS).
If you make sure that those queries are prepared only once and then
reused, and that they use indices sensibly, this should perform well.
If you care about performance you should think twice about using
frameworks that promise to "do all the hard work" and generate the
SQL for you. There have been countless examples of such
frameworks (I won't name any) producing completely brain-dead SQL...
>>> indexing/transactional features of Derby instead. However, I'm not
>>> willing to pay the performance cost for using SQL since I'm only
>>> storing byte[] arrays (if it is possible).
Try it! Write a simple app, or use the simple client found under
DERBY-1961 and profile it. The cost of compiling the SQL is only done
once and "amortized" across the uses the prepared statement.
Maybe you can modify the DERBY-1961 to use the data type need. Then
you can see if the performance is acceptable...
--
dt
Re: Using Derby as a binary store
Posted by Kasper Nielsen <ne...@kav.dk>.
David Van Couvering wrote:
> To be honest, this looks more like a job for BDB than for Derby. I
> would love to see the Derby store API made public at some point, but
> currently it's not public and I suspect it may be difficult to work
> with. Are there reasons BDB/Java won't work for you?
Not besides the license:)
Its for an BSD licensed project, so BDB is pretty much out of the picture.
And since I haven't been able to find any projects resembling BDB my
hope was that I could use Derby with some minor tweaking.
- Kasper
> David
>
> Kasper Nielsen wrote:
>> Julius.Stroffek wrote:
>>> Do you need to store them also on a disk?
>>>
>>> Cheers
>>>
>>> Julo
>> Yes,
>>
>> I'm only interested in the actual storage/indexing. Any caching I'll
>> take of myself.
>>
>> Let's for simplicity's sake say I need the following interface in my
>> application.
>>
>> interface Store {
>> void put(byte[] key, byte[] value);
>> byte[] get(byte[] key);
>> Map<byte[], byte[]> getRange(byte[] key_from, byte[] key_to);
>> }
>>
>> put and get works like the java.util.Map put/get operation.
>> getRange(83,103) would return all the existing key->value bindings for
>> numbers between 83 and 103.
>>
>> I could implement this myself, but I would like to use the
>> indexing/transactional features of Derby instead. However, I'm not
>> willing to pay the performance cost for using SQL since I'm only
>> storing byte[] arrays (if it is possible).
>>
>> Cheers
>> Kasper
>>
>>>
>>> Kasper Nielsen wrote:
>>>> I'm just looking for a lightweight solution where I can store bytes
>>>> (key->value pairs) and create a few indexes on the data.
>>>>
>>>>
>>>> Cheers
>>>> Kasper
>>>>
>>>>
>>>>> On 11/24/06, Kasper Nielsen <ne...@kav.dk> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm evaluating Derby for a project of mine. I really would like to
>>>>>> avoid
>>>>>> using SQL and just use Derby as a binary store. Is this possible?
>>>>>>
>>>>>> My data consists of key -> value pairs where the keys are uniformly
>>>>>> distributed 160 bit numbers and the values are byte arrays.
>>>>>>
>>>>>> For now, I need to support 2 basic queries
>>>>>> * Give me the value mapping for the key K or null if it doesn't
>>>>>> exist.
>>>>>> * Give me a map of (key->value) for the key range of key_start to
>>>>>> key_end.
>>>>>>
>>>>>> I had a look at the org.apache.derby.iap.store packages but wasn't
>>>>>> quite
>>>>>> sure where to start.
>>>>>>
>>>>>> Cheers
>>>>>> Kasper
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: Using Derby as a binary store
Posted by David Van Couvering <da...@vancouvering.com>.
To be honest, this looks more like a job for BDB than for Derby. I
would love to see the Derby store API made public at some point, but
currently it's not public and I suspect it may be difficult to work
with. Are there reasons BDB/Java won't work for you?
David
Kasper Nielsen wrote:
> Julius.Stroffek wrote:
>> Do you need to store them also on a disk?
>>
>> Cheers
>>
>> Julo
> Yes,
>
> I'm only interested in the actual storage/indexing. Any caching I'll
> take of myself.
>
> Let's for simplicity's sake say I need the following interface in my
> application.
>
> interface Store {
> void put(byte[] key, byte[] value);
> byte[] get(byte[] key);
> Map<byte[], byte[]> getRange(byte[] key_from, byte[] key_to);
> }
>
> put and get works like the java.util.Map put/get operation.
> getRange(83,103) would return all the existing key->value bindings for
> numbers between 83 and 103.
>
> I could implement this myself, but I would like to use the
> indexing/transactional features of Derby instead. However, I'm not
> willing to pay the performance cost for using SQL since I'm only storing
> byte[] arrays (if it is possible).
>
> Cheers
> Kasper
>
>>
>> Kasper Nielsen wrote:
>>> I'm just looking for a lightweight solution where I can store bytes
>>> (key->value pairs) and create a few indexes on the data.
>>>
>>>
>>> Cheers
>>> Kasper
>>>
>>>
>>>> On 11/24/06, Kasper Nielsen <ne...@kav.dk> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm evaluating Derby for a project of mine. I really would like to
>>>>> avoid
>>>>> using SQL and just use Derby as a binary store. Is this possible?
>>>>>
>>>>> My data consists of key -> value pairs where the keys are uniformly
>>>>> distributed 160 bit numbers and the values are byte arrays.
>>>>>
>>>>> For now, I need to support 2 basic queries
>>>>> * Give me the value mapping for the key K or null if it doesn't exist.
>>>>> * Give me a map of (key->value) for the key range of key_start to
>>>>> key_end.
>>>>>
>>>>> I had a look at the org.apache.derby.iap.store packages but wasn't
>>>>> quite
>>>>> sure where to start.
>>>>>
>>>>> Cheers
>>>>> Kasper
>>>>>
>>>>
>>>
>>
>
>
Re: Using Derby as a binary store
Posted by Kasper Nielsen <ne...@kav.dk>.
Julius.Stroffek wrote:
> Do you need to store them also on a disk?
>
> Cheers
>
> Julo
Yes,
I'm only interested in the actual storage/indexing. Any caching I'll
take of myself.
Let's for simplicity's sake say I need the following interface in my
application.
interface Store {
void put(byte[] key, byte[] value);
byte[] get(byte[] key);
Map<byte[], byte[]> getRange(byte[] key_from, byte[] key_to);
}
put and get works like the java.util.Map put/get operation.
getRange(83,103) would return all the existing key->value bindings for
numbers between 83 and 103.
I could implement this myself, but I would like to use the
indexing/transactional features of Derby instead. However, I'm not
willing to pay the performance cost for using SQL since I'm only storing
byte[] arrays (if it is possible).
Cheers
Kasper
>
> Kasper Nielsen wrote:
>> I'm just looking for a lightweight solution where I can store bytes
>> (key->value pairs) and create a few indexes on the data.
>>
>>
>> Cheers
>> Kasper
>>
>>
>>> On 11/24/06, Kasper Nielsen <ne...@kav.dk> wrote:
>>>> Hi,
>>>>
>>>> I'm evaluating Derby for a project of mine. I really would like to
>>>> avoid
>>>> using SQL and just use Derby as a binary store. Is this possible?
>>>>
>>>> My data consists of key -> value pairs where the keys are uniformly
>>>> distributed 160 bit numbers and the values are byte arrays.
>>>>
>>>> For now, I need to support 2 basic queries
>>>> * Give me the value mapping for the key K or null if it doesn't exist.
>>>> * Give me a map of (key->value) for the key range of key_start to
>>>> key_end.
>>>>
>>>> I had a look at the org.apache.derby.iap.store packages but wasn't
>>>> quite
>>>> sure where to start.
>>>>
>>>> Cheers
>>>> Kasper
>>>>
>>>
>>
>
Re: Using Derby as a binary store
Posted by "Julius.Stroffek" <Ju...@Sun.COM>.
Do you need to store them also on a disk?
Cheers
Julo
Kasper Nielsen wrote:
> I'm just looking for a lightweight solution where I can store bytes
> (key->value pairs) and create a few indexes on the data.
>
>
> Cheers
> Kasper
>
>
>> On 11/24/06, Kasper Nielsen <ne...@kav.dk> wrote:
>>> Hi,
>>>
>>> I'm evaluating Derby for a project of mine. I really would like to
>>> avoid
>>> using SQL and just use Derby as a binary store. Is this possible?
>>>
>>> My data consists of key -> value pairs where the keys are uniformly
>>> distributed 160 bit numbers and the values are byte arrays.
>>>
>>> For now, I need to support 2 basic queries
>>> * Give me the value mapping for the key K or null if it doesn't exist.
>>> * Give me a map of (key->value) for the key range of key_start to
>>> key_end.
>>>
>>> I had a look at the org.apache.derby.iap.store packages but wasn't
>>> quite
>>> sure where to start.
>>>
>>> Cheers
>>> Kasper
>>>
>>
>
Re: Using Derby as a binary store
Posted by Kasper Nielsen <ne...@kav.dk>.
Peter Ondruska wrote:
> you should have a look at Hibernate or Toplink or similar tool if you
> want to _minimize_ SQL. however you will most probably come to a point
> where you cannot avoid SQL.
Hey,
just to clarify my needs. I'm not trying to avoid using SQL/JDBC because
I have a problem with using it. I'm trying to avoid using it because
there is an unacceptable overhead involved with using it (for my purpose).
I'm just looking for a lightweight solution where I can store bytes
(key->value pairs) and create a few indexes on the data.
Cheers
Kasper
> On 11/24/06, Kasper Nielsen <ne...@kav.dk> wrote:
>> Hi,
>>
>> I'm evaluating Derby for a project of mine. I really would like to avoid
>> using SQL and just use Derby as a binary store. Is this possible?
>>
>> My data consists of key -> value pairs where the keys are uniformly
>> distributed 160 bit numbers and the values are byte arrays.
>>
>> For now, I need to support 2 basic queries
>> * Give me the value mapping for the key K or null if it doesn't exist.
>> * Give me a map of (key->value) for the key range of key_start to
>> key_end.
>>
>> I had a look at the org.apache.derby.iap.store packages but wasn't quite
>> sure where to start.
>>
>> Cheers
>> Kasper
>>
>
Re: Using Derby as a binary store
Posted by Peter Ondruska <pe...@gmail.com>.
you should have a look at Hibernate or Toplink or similar tool if you
want to _minimize_ SQL. however you will most probably come to a point
where you cannot avoid SQL.
On 11/24/06, Kasper Nielsen <ne...@kav.dk> wrote:
> Hi,
>
> I'm evaluating Derby for a project of mine. I really would like to avoid
> using SQL and just use Derby as a binary store. Is this possible?
>
> My data consists of key -> value pairs where the keys are uniformly
> distributed 160 bit numbers and the values are byte arrays.
>
> For now, I need to support 2 basic queries
> * Give me the value mapping for the key K or null if it doesn't exist.
> * Give me a map of (key->value) for the key range of key_start to key_end.
>
> I had a look at the org.apache.derby.iap.store packages but wasn't quite
> sure where to start.
>
> Cheers
> Kasper
>
Re: Using Derby as a binary store
Posted by Kasper Nielsen <ne...@kav.dk>.
Kristian Waagan wrote:
> Kasper Nielsen wrote:
>> Kasper Nielsen wrote:
>>> Hi,
>>>
>>> I'm evaluating Derby for a project of mine. I really would like to
>>> avoid using SQL and just use Derby as a binary store. Is this possible?
>>>
>>> My data consists of key -> value pairs where the keys are uniformly
>>> distributed 160 bit numbers and the values are byte arrays.
>>>
>>> For now, I need to support 2 basic queries
>>> * Give me the value mapping for the key K or null if it doesn't exist.
>>> * Give me a map of (key->value) for the key range of key_start to
>>> key_end.
>>>
>>> I had a look at the org.apache.derby.iap.store packages but wasn't
>>> quite sure where to start.
>>>
>>> Cheers
>>> Kasper
>>>
>>
>> Thanks for all the suggestions. I think I'll go ahead and create some
>> test cases and find out whatever overhead (space/performance) is
>> involved.
>>
>> For those interested,
>> http://en.wikipedia.org/wiki/Distributed_hash_table pretty much
>> describes the usage. Because you a providing a general platform you
>> want to limit the storage functionality to simple put/get (key->value)
>> operations. Which is why I don't really need any advanced features. To
>> build more more advanced functionally (indexing, redundancy, ...) any
>> user must build on top of these put/get primitives.
>
> Hi Kasper,
>
> Just as a side note, I once implemented a simple DHT using the Chord
> algorithm and Berkeley DB Java Edition. But this was some time ago and
> before Oracle acquired BDB.
I actually just found a whitepaper
"Oracle Berkeley DB Java Edition vs Apache Derby: A Performance Comparison"
http://www.oracle.com/technology/products/berkeley-db/pdf/je-derby-performance.pdf
Just copying there conclusion here:
"The Berkeley DB Java Edition Persistence API is a high performance,
complete solution for Java object persistence. Berkeley DB Java Edition
performance exceeds Derby performance in every test, by a factor of 3 to
10, clearly demonstrating the superior performance of Berkeley DB Java
Edition"
Cheers
Kasper
Re: Using Derby as a binary store
Posted by Kristian Waagan <Kr...@Sun.COM>.
Kasper Nielsen wrote:
> Kasper Nielsen wrote:
>> Hi,
>>
>> I'm evaluating Derby for a project of mine. I really would like to
>> avoid using SQL and just use Derby as a binary store. Is this possible?
>>
>> My data consists of key -> value pairs where the keys are uniformly
>> distributed 160 bit numbers and the values are byte arrays.
>>
>> For now, I need to support 2 basic queries
>> * Give me the value mapping for the key K or null if it doesn't exist.
>> * Give me a map of (key->value) for the key range of key_start to
>> key_end.
>>
>> I had a look at the org.apache.derby.iap.store packages but wasn't
>> quite sure where to start.
>>
>> Cheers
>> Kasper
>>
>
> Thanks for all the suggestions. I think I'll go ahead and create some
> test cases and find out whatever overhead (space/performance) is involved.
>
> For those interested,
> http://en.wikipedia.org/wiki/Distributed_hash_table pretty much
> describes the usage. Because you a providing a general platform you want
> to limit the storage functionality to simple put/get (key->value)
> operations. Which is why I don't really need any advanced features. To
> build more more advanced functionally (indexing, redundancy, ...) any
> user must build on top of these put/get primitives.
Hi Kasper,
Just as a side note, I once implemented a simple DHT using the Chord
algorithm and Berkeley DB Java Edition. But this was some time ago and
before Oracle acquired BDB.
It was a fun exercise :) Please let us hear if Derby meets the
performance requirements for your DHT, or if there is too much overhead.
regards,
--
Kristian
>
> Some applications might primarily be providing storage to users. So not
> much work for any of the nodes except routing and storing. Which is why
> I'm interested in a low overhead storage mechanism without to much cruft.
>
> - Kasper
Re: Using Derby as a binary store
Posted by Kasper Nielsen <ne...@kav.dk>.
Kasper Nielsen wrote:
> Hi,
>
> I'm evaluating Derby for a project of mine. I really would like to avoid
> using SQL and just use Derby as a binary store. Is this possible?
>
> My data consists of key -> value pairs where the keys are uniformly
> distributed 160 bit numbers and the values are byte arrays.
>
> For now, I need to support 2 basic queries
> * Give me the value mapping for the key K or null if it doesn't exist.
> * Give me a map of (key->value) for the key range of key_start to key_end.
>
> I had a look at the org.apache.derby.iap.store packages but wasn't quite
> sure where to start.
>
> Cheers
> Kasper
>
Thanks for all the suggestions. I think I'll go ahead and create some
test cases and find out whatever overhead (space/performance) is involved.
For those interested,
http://en.wikipedia.org/wiki/Distributed_hash_table pretty much
describes the usage. Because you a providing a general platform you want
to limit the storage functionality to simple put/get (key->value)
operations. Which is why I don't really need any advanced features. To
build more more advanced functionally (indexing, redundancy, ...) any
user must build on top of these put/get primitives.
Some applications might primarily be providing storage to users. So not
much work for any of the nodes except routing and storing. Which is why
I'm interested in a low overhead storage mechanism without to much cruft.
- Kasper