You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by Andrzej Bialecki <ab...@getopt.org> on 2010/11/05 16:31:01 UTC
HSQLDB woes...
Hi,
The HSQL-based SqlStore exhibits awful performance when used with Nutch.
I believe this is related to the way LOBs are handled in HSQL - even for
a tiny crawl of 50 pages the size of the .lob file is in the order of
100MB. Actually, after reaching this point the performance of any
updates drops dramatically so it becomes nearly unusable.
Of course, HSQL was never meant to be used as a serious backend...
still, perhaps there are alternatives that could give us a better
behavior for small / embedded use - and for small operations in the
order of a few thousand records I think we should be able to come up
with something better...
I tried to integrate the H2 database (www.h2database.com), but gave up
after I discovered that it doesn't support Blob.setBinaryStream(..) -
there are workarounds for this in H2, but it would complicate the code
too much...
Any suggestions / comments? Maybe it's time for a BerkeleyDB DataStore?
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: HSQLDB woes...
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-11-08 23:35, Doğacan Güney wrote:
>>> Does anyone know if there is a licensing problem with it?
>>
>> Yes. It's either GPL or commercial.
>>
>
> Hmm... Is it ok if we do not include libraries in our code base but
> pull it via ivy?
I think it's ok, Lucene BDB directory uses this approach so apparently
someone decided in that case that it was ok...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: HSQLDB woes...
Posted by Doğacan Güney <do...@gmail.com>.
On Mon, Nov 8, 2010 at 13:49, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 2010-11-08 21:29, Doğacan Güney wrote:
>> On Mon, Nov 8, 2010 at 10:26, Enis Söztutar <en...@gmail.com> wrote:
>>> From my experience the SQL backend is a mJOR headache. Writing a SQL backend
>>> is actually much much harder than
>>> the HBase or Cassandra backend since we need very custom code for each SQL
>>> server. Plus, there is some code for
>>> dealing with HSQL embedded mode.
>>>
>>> I completely agree to switch to another zero-conf backend for tests and for
>>> nutch. However, I am not sure about BerkeleyDB.
>>> If we can implement a data store easily that would be great.
>>>
>>
>> I looked a bit at BDB a while back. I think it should be easy to
>> implement... I'll look into
>> it this week and report back.
>>
>> Does anyone know if there is a licensing problem with it?
>
> Yes. It's either GPL or commercial.
>
Hmm... Is it ok if we do not include libraries in our code base but
pull it via ivy?
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
--
Doğacan Güney
Re: HSQLDB woes...
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-11-08 21:29, Doğacan Güney wrote:
> On Mon, Nov 8, 2010 at 10:26, Enis Söztutar <en...@gmail.com> wrote:
>> From my experience the SQL backend is a mJOR headache. Writing a SQL backend
>> is actually much much harder than
>> the HBase or Cassandra backend since we need very custom code for each SQL
>> server. Plus, there is some code for
>> dealing with HSQL embedded mode.
>>
>> I completely agree to switch to another zero-conf backend for tests and for
>> nutch. However, I am not sure about BerkeleyDB.
>> If we can implement a data store easily that would be great.
>>
>
> I looked a bit at BDB a while back. I think it should be easy to
> implement... I'll look into
> it this week and report back.
>
> Does anyone know if there is a licensing problem with it?
Yes. It's either GPL or commercial.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: HSQLDB woes...
Posted by Doğacan Güney <do...@gmail.com>.
On Mon, Nov 8, 2010 at 10:26, Enis Söztutar <en...@gmail.com> wrote:
> From my experience the SQL backend is a mJOR headache. Writing a SQL backend
> is actually much much harder than
> the HBase or Cassandra backend since we need very custom code for each SQL
> server. Plus, there is some code for
> dealing with HSQL embedded mode.
>
> I completely agree to switch to another zero-conf backend for tests and for
> nutch. However, I am not sure about BerkeleyDB.
> If we can implement a data store easily that would be great.
>
I looked a bit at BDB a while back. I think it should be easy to
implement... I'll look into
it this week and report back.
Does anyone know if there is a licensing problem with it?
> Enis
>
> On Fri, Nov 5, 2010 at 8:31 AM, Andrzej Bialecki <ab...@getopt.org> wrote:
>
>> Hi,
>>
>> The HSQL-based SqlStore exhibits awful performance when used with Nutch.
>> I believe this is related to the way LOBs are handled in HSQL - even for
>> a tiny crawl of 50 pages the size of the .lob file is in the order of
>> 100MB. Actually, after reaching this point the performance of any
>> updates drops dramatically so it becomes nearly unusable.
>>
>> Of course, HSQL was never meant to be used as a serious backend...
>> still, perhaps there are alternatives that could give us a better
>> behavior for small / embedded use - and for small operations in the
>> order of a few thousand records I think we should be able to come up
>> with something better...
>>
>> I tried to integrate the H2 database (www.h2database.com), but gave up
>> after I discovered that it doesn't support Blob.setBinaryStream(..) -
>> there are workarounds for this in H2, but it would complicate the code
>> too much...
>>
>> Any suggestions / comments? Maybe it's time for a BerkeleyDB DataStore?
>>
>> --
>> Best regards,
>> Andrzej Bialecki <><
>> ___. ___ ___ ___ _ _ __________________________________
>> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
>> ___|||__|| \| || | Embedded Unix, System Integration
>> http://www.sigram.com Contact: info at sigram dot com
>>
>>
>
--
Doğacan Güney
Re: HSQLDB woes...
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-11-08 19:26, Enis Söztutar wrote:
>>>From my experience the SQL backend is a mJOR headache. Writing a SQL backend
> is actually much much harder than
> the HBase or Cassandra backend since we need very custom code for each SQL
> server. Plus, there is some code for
> dealing with HSQL embedded mode.
>
> I completely agree to switch to another zero-conf backend for tests and for
> nutch. However, I am not sure about BerkeleyDB.
> If we can implement a data store easily that would be great.
I started working on three implementations, but neither one works yet in
a satisfactory way... ;)
* Solr-based (GORA-9). Tests all pass, but when I tried to use this
store with Nutch it would lose all data at some point... probably I
misunderstood the way schemas are created / truncated / deleted, or this
could be related to GORA-12. More investigation needed. Another
complication is that Solr needs physical config files that specify
schema in order to create new cores (schema containers, so to speak - I
modeled one schema per Solr core), and there is no REST API in Solr to
create these files on the fly.
* Based on Ehcache. Again, all tests passed, and the store seemed
working ok in tests, but... it turned out that Ehcache is _very_
sensitive to a proper shutdown procedure. If the cache is not properly
shut down it leads to a complete data loss (!) Ouch. Pity I didn't
discover this before I started the implementation... On the off chance
that I missed something obvious I'm perfectly happy to share this work
in progress.
* Based on JDBM2.googlecode.com. This is partially done, simple put/get
works ok, and I think that would be our best bet for a simple
self-contained store ... but I'm probably missing something about the
key-space traversal because I can't make the queries work. Again, I'll
create an issue and I will attach the code.
So I guess we are probably close to having some of them working the way
they should (if possible at all). Unfortunately, for the next couple
week I won't be able to spend much time on this, so I welcome any help,
or code review (once I upload it).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: HSQLDB woes...
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Guys,
We recently used Berkeley XMLDB and it was actually pretty performant. I'm not sure about its license though; we used it internally at JPL.
Cheers,
Chris
On 11/8/10 11:26 AM, "Enis Söztutar" <en...@gmail.com> wrote:
>From my experience the SQL backend is a mJOR headache. Writing a SQL backend
is actually much much harder than
the HBase or Cassandra backend since we need very custom code for each SQL
server. Plus, there is some code for
dealing with HSQL embedded mode.
I completely agree to switch to another zero-conf backend for tests and for
nutch. However, I am not sure about BerkeleyDB.
If we can implement a data store easily that would be great.
Enis
On Fri, Nov 5, 2010 at 8:31 AM, Andrzej Bialecki <ab...@getopt.org> wrote:
> Hi,
>
> The HSQL-based SqlStore exhibits awful performance when used with Nutch.
> I believe this is related to the way LOBs are handled in HSQL - even for
> a tiny crawl of 50 pages the size of the .lob file is in the order of
> 100MB. Actually, after reaching this point the performance of any
> updates drops dramatically so it becomes nearly unusable.
>
> Of course, HSQL was never meant to be used as a serious backend...
> still, perhaps there are alternatives that could give us a better
> behavior for small / embedded use - and for small operations in the
> order of a few thousand records I think we should be able to come up
> with something better...
>
> I tried to integrate the H2 database (www.h2database.com), but gave up
> after I discovered that it doesn't support Blob.setBinaryStream(..) -
> there are workarounds for this in H2, but it would complicate the code
> too much...
>
> Any suggestions / comments? Maybe it's time for a BerkeleyDB DataStore?
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: HSQLDB woes...
Posted by Enis Söztutar <en...@gmail.com>.
>From my experience the SQL backend is a mJOR headache. Writing a SQL backend
is actually much much harder than
the HBase or Cassandra backend since we need very custom code for each SQL
server. Plus, there is some code for
dealing with HSQL embedded mode.
I completely agree to switch to another zero-conf backend for tests and for
nutch. However, I am not sure about BerkeleyDB.
If we can implement a data store easily that would be great.
Enis
On Fri, Nov 5, 2010 at 8:31 AM, Andrzej Bialecki <ab...@getopt.org> wrote:
> Hi,
>
> The HSQL-based SqlStore exhibits awful performance when used with Nutch.
> I believe this is related to the way LOBs are handled in HSQL - even for
> a tiny crawl of 50 pages the size of the .lob file is in the order of
> 100MB. Actually, after reaching this point the performance of any
> updates drops dramatically so it becomes nearly unusable.
>
> Of course, HSQL was never meant to be used as a serious backend...
> still, perhaps there are alternatives that could give us a better
> behavior for small / embedded use - and for small operations in the
> order of a few thousand records I think we should be able to come up
> with something better...
>
> I tried to integrate the H2 database (www.h2database.com), but gave up
> after I discovered that it doesn't support Blob.setBinaryStream(..) -
> there are workarounds for this in H2, but it would complicate the code
> too much...
>
> Any suggestions / comments? Maybe it's time for a BerkeleyDB DataStore?
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>