You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Peter Keane <pk...@mail.utexas.edu> on 2009/05/26 16:50:19 UTC
recommendation for document store to use alongside Solr?
Hi All-
I've just recently began playing with Apache Solr, and it seems to be
a perfect fit for our project (http://code.google.com/p/dase/). I've
been quite surprised at both how easy Solr was to get up and running
and how flexible it seems to be. I've been tempted to use it for not
just search, but document storage as well. Seems, though, this is not
the best road to go down.
I'd like to know if there are recommendations for a document store (or
distributed hash table) that would work well alongside Solr.
Basically, I'd like to be able to deploy in Tomcat and interact w/ the
store over http (like Solr). Recommendations I've seen include
Project Voldemort, BDB, BananaDB, CouchDB, etc. I'd be quite
interested to hear comments about pluses/minuses of any of those or
other options OR comments about Solr suitability as a document store.
thanks-
Peter Keane
daseproject.org
Re: recommendation for document store to use alongside Solr?
Posted by Glen Newton <gl...@gmail.com>.
Depends on how large/many are your documents.
Using Solr/Lucene: you can store the documents in the same index as
the search index, or you can have "key" in the search index which is a
look-up in a second index, which has two fields, the key (indexed) and
the document(stored, maybe compressed too). The choice would depend on
the impact of the stored documents on the size of the index. It is
also very application dependent: the ratio of hit result to document
access is important: if the document is accessed 1% of the time it is
in a search result this will have different impact than if the
document is accessed 99% of the time it is in the search result.
I think you could do this with 2 cores in Solr, if I understand Solr correctly.
I have also had good experience with BDB for (non-networked) document storage.
Glen Newton
http://zzzoot.blogspot.com/
2009/5/26 Peter Keane <pk...@mail.utexas.edu>:
> Hi All-
>
> I've just recently began playing with Apache Solr, and it seems to be
> a perfect fit for our project (http://code.google.com/p/dase/). I've
> been quite surprised at both how easy Solr was to get up and running
> and how flexible it seems to be. I've been tempted to use it for not
> just search, but document storage as well. Seems, though, this is not
> the best road to go down.
>
> I'd like to know if there are recommendations for a document store (or
> distributed hash table) that would work well alongside Solr.
> Basically, I'd like to be able to deploy in Tomcat and interact w/ the
> store over http (like Solr). Recommendations I've seen include
> Project Voldemort, BDB, BananaDB, CouchDB, etc. I'd be quite
> interested to hear comments about pluses/minuses of any of those or
> other options OR comments about Solr suitability as a document store.
>
> thanks-
> Peter Keane
> daseproject.org
>
--
-
Re: recommendation for document store to use alongside Solr?
Posted by Peter Keane <pk...@mail.utexas.edu>.
Otis-
Thanks much. This is v. useful!
--peter
On Tue, May 26, 2009 at 12:16 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> Peter,
> If you need to sell and selling fewer changes is more likely to succeed, then you could always propose:
> - Offload SQL-based full-text searcher to Solr -- it's going to be faster, more flexible, require less hardware, etc.
> - This reduces load on the DB servers, which you can now use as simple data storage
>
>
> Of course, using a *R*DBM just for that is a bit of misuse, but it doesn't introduce yet another new component into the system.
> And if the number of hits to the DB is not too high, the DB could handle it now that it's not being hammered with search requests.
> Of course, you could also put a caching layer (e.g. memcached) between Solr and RDBMS and lower the hit rate even further (assuming requests for documents are not super random and could thus benefit from a document cache).
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Peter Keane <pk...@mail.utexas.edu>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, May 26, 2009 1:02:15 PM
>> Subject: Re: recommendation for document store to use alongside Solr?
>>
>> Thanks Otis (for this and your recs on Twitter!).
>>
>> I'm just figuring this stuff out, nice to get a better perspective.
>> Our application (a large digital repository at UT Austin) is, up until
>> now, is all PHP+PostgreSQL. Creating another dependency (like Solr)
>> means a bit of a sell on my part (convincing admins to run a dev &
>> production Tomcat server, etc.) so I had been hesitant.
>>
>> Turns out load on the RDBMS (mainly in support of SQL-based search)
>> has gotten too high to be maintainable. My experiments with Solr over
>> the last few days have been fairly astonishing to me (ease of use,
>> performance, flexibility). Solr will allow us to do all sorts of
>> sophisticated indexing and querying that would've been impossible with
>> the DB. I no longer expect it to be a tough sell to our admins! :-).
>>
>> I guess it has spoiled me to want a similarly easy-to-use and
>> performant document store. The main (preferred) requirement is an
>> http interface. All documents are Atom Entries and Atom Feeds. Until
>> now I've been using a DB table (for Atom Entries) and a file-based
>> cache (for Atom feeds). Continuing with that is still an option.
>>
>> So far I like Tokyo (Cabinet+Tyrant) and CouchDB. CouchDB may be an
>> easier sell.
>>
>> thanks!
>> Peter
>>
>>
>>
>> On Tue, May 26, 2009 at 11:44 AM, Otis Gospodnetic
>> wrote:
>> >
>> > Hi Peter,
>> >
>> > Yeah, familiar recommendations - http://twitter.com/otisg/status/1907773452
>> >
>> > I've used BDB for this type of stuff in the past and was pleased with BDBs
>> performance, ability to scale, and the simplicity of the API. I would not use
>> Solr itself for this.
>> >
>> > This one is from January:
>> >
>> http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/
>> > It doesn't include some of the newer things I found - see
>> http://www.simpy.com/user/otis/tag/%22hash+table%22
>> >
>> > Otis
>> > --
>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>> >
>> >
>> >
>> > ----- Original Message ----
>> >> From: Peter Keane
>> >> To: solr-user@lucene.apache.org
>> >> Sent: Tuesday, May 26, 2009 10:50:19 AM
>> >> Subject: recommendation for document store to use alongside Solr?
>> >>
>> >> Hi All-
>> >>
>> >> I've just recently began playing with Apache Solr, and it seems to be
>> >> a perfect fit for our project (http://code.google.com/p/dase/). I've
>> >> been quite surprised at both how easy Solr was to get up and running
>> >> and how flexible it seems to be. I've been tempted to use it for not
>> >> just search, but document storage as well. Seems, though, this is not
>> >> the best road to go down.
>> >>
>> >> I'd like to know if there are recommendations for a document store (or
>> >> distributed hash table) that would work well alongside Solr.
>> >> Basically, I'd like to be able to deploy in Tomcat and interact w/ the
>> >> store over http (like Solr). Recommendations I've seen include
>> >> Project Voldemort, BDB, BananaDB, CouchDB, etc. I'd be quite
>> >> interested to hear comments about pluses/minuses of any of those or
>> >> other options OR comments about Solr suitability as a document store.
>> >>
>> >> thanks-
>> >> Peter Keane
>> >> daseproject.org
>> >
>> >
>
>
Re: recommendation for document store to use alongside Solr?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Peter,
If you need to sell and selling fewer changes is more likely to succeed, then you could always propose:
- Offload SQL-based full-text searcher to Solr -- it's going to be faster, more flexible, require less hardware, etc.
- This reduces load on the DB servers, which you can now use as simple data storage
Of course, using a *R*DBM just for that is a bit of misuse, but it doesn't introduce yet another new component into the system.
And if the number of hits to the DB is not too high, the DB could handle it now that it's not being hammered with search requests.
Of course, you could also put a caching layer (e.g. memcached) between Solr and RDBMS and lower the hit rate even further (assuming requests for documents are not super random and could thus benefit from a document cache).
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Peter Keane <pk...@mail.utexas.edu>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, May 26, 2009 1:02:15 PM
> Subject: Re: recommendation for document store to use alongside Solr?
>
> Thanks Otis (for this and your recs on Twitter!).
>
> I'm just figuring this stuff out, nice to get a better perspective.
> Our application (a large digital repository at UT Austin) is, up until
> now, is all PHP+PostgreSQL. Creating another dependency (like Solr)
> means a bit of a sell on my part (convincing admins to run a dev &
> production Tomcat server, etc.) so I had been hesitant.
>
> Turns out load on the RDBMS (mainly in support of SQL-based search)
> has gotten too high to be maintainable. My experiments with Solr over
> the last few days have been fairly astonishing to me (ease of use,
> performance, flexibility). Solr will allow us to do all sorts of
> sophisticated indexing and querying that would've been impossible with
> the DB. I no longer expect it to be a tough sell to our admins! :-).
>
> I guess it has spoiled me to want a similarly easy-to-use and
> performant document store. The main (preferred) requirement is an
> http interface. All documents are Atom Entries and Atom Feeds. Until
> now I've been using a DB table (for Atom Entries) and a file-based
> cache (for Atom feeds). Continuing with that is still an option.
>
> So far I like Tokyo (Cabinet+Tyrant) and CouchDB. CouchDB may be an
> easier sell.
>
> thanks!
> Peter
>
>
>
> On Tue, May 26, 2009 at 11:44 AM, Otis Gospodnetic
> wrote:
> >
> > Hi Peter,
> >
> > Yeah, familiar recommendations - http://twitter.com/otisg/status/1907773452
> >
> > I've used BDB for this type of stuff in the past and was pleased with BDBs
> performance, ability to scale, and the simplicity of the API. I would not use
> Solr itself for this.
> >
> > This one is from January:
> >
> http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/
> > It doesn't include some of the newer things I found - see
> http://www.simpy.com/user/otis/tag/%22hash+table%22
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: Peter Keane
> >> To: solr-user@lucene.apache.org
> >> Sent: Tuesday, May 26, 2009 10:50:19 AM
> >> Subject: recommendation for document store to use alongside Solr?
> >>
> >> Hi All-
> >>
> >> I've just recently began playing with Apache Solr, and it seems to be
> >> a perfect fit for our project (http://code.google.com/p/dase/). I've
> >> been quite surprised at both how easy Solr was to get up and running
> >> and how flexible it seems to be. I've been tempted to use it for not
> >> just search, but document storage as well. Seems, though, this is not
> >> the best road to go down.
> >>
> >> I'd like to know if there are recommendations for a document store (or
> >> distributed hash table) that would work well alongside Solr.
> >> Basically, I'd like to be able to deploy in Tomcat and interact w/ the
> >> store over http (like Solr). Recommendations I've seen include
> >> Project Voldemort, BDB, BananaDB, CouchDB, etc. I'd be quite
> >> interested to hear comments about pluses/minuses of any of those or
> >> other options OR comments about Solr suitability as a document store.
> >>
> >> thanks-
> >> Peter Keane
> >> daseproject.org
> >
> >
Re: recommendation for document store to use alongside Solr?
Posted by Peter Keane <pk...@mail.utexas.edu>.
Thanks Otis (for this and your recs on Twitter!).
I'm just figuring this stuff out, nice to get a better perspective.
Our application (a large digital repository at UT Austin) is, up until
now, is all PHP+PostgreSQL. Creating another dependency (like Solr)
means a bit of a sell on my part (convincing admins to run a dev &
production Tomcat server, etc.) so I had been hesitant.
Turns out load on the RDBMS (mainly in support of SQL-based search)
has gotten too high to be maintainable. My experiments with Solr over
the last few days have been fairly astonishing to me (ease of use,
performance, flexibility). Solr will allow us to do all sorts of
sophisticated indexing and querying that would've been impossible with
the DB. I no longer expect it to be a tough sell to our admins! :-).
I guess it has spoiled me to want a similarly easy-to-use and
performant document store. The main (preferred) requirement is an
http interface. All documents are Atom Entries and Atom Feeds. Until
now I've been using a DB table (for Atom Entries) and a file-based
cache (for Atom feeds). Continuing with that is still an option.
So far I like Tokyo (Cabinet+Tyrant) and CouchDB. CouchDB may be an
easier sell.
thanks!
Peter
On Tue, May 26, 2009 at 11:44 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> Hi Peter,
>
> Yeah, familiar recommendations - http://twitter.com/otisg/status/1907773452
>
> I've used BDB for this type of stuff in the past and was pleased with BDBs performance, ability to scale, and the simplicity of the API. I would not use Solr itself for this.
>
> This one is from January:
> http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/
> It doesn't include some of the newer things I found - see http://www.simpy.com/user/otis/tag/%22hash+table%22
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Peter Keane <pk...@mail.utexas.edu>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, May 26, 2009 10:50:19 AM
>> Subject: recommendation for document store to use alongside Solr?
>>
>> Hi All-
>>
>> I've just recently began playing with Apache Solr, and it seems to be
>> a perfect fit for our project (http://code.google.com/p/dase/). I've
>> been quite surprised at both how easy Solr was to get up and running
>> and how flexible it seems to be. I've been tempted to use it for not
>> just search, but document storage as well. Seems, though, this is not
>> the best road to go down.
>>
>> I'd like to know if there are recommendations for a document store (or
>> distributed hash table) that would work well alongside Solr.
>> Basically, I'd like to be able to deploy in Tomcat and interact w/ the
>> store over http (like Solr). Recommendations I've seen include
>> Project Voldemort, BDB, BananaDB, CouchDB, etc. I'd be quite
>> interested to hear comments about pluses/minuses of any of those or
>> other options OR comments about Solr suitability as a document store.
>>
>> thanks-
>> Peter Keane
>> daseproject.org
>
>
Re: recommendation for document store to use alongside Solr?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Peter,
Yeah, familiar recommendations - http://twitter.com/otisg/status/1907773452
I've used BDB for this type of stuff in the past and was pleased with BDBs performance, ability to scale, and the simplicity of the API. I would not use Solr itself for this.
This one is from January:
http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/
It doesn't include some of the newer things I found - see http://www.simpy.com/user/otis/tag/%22hash+table%22
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Peter Keane <pk...@mail.utexas.edu>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, May 26, 2009 10:50:19 AM
> Subject: recommendation for document store to use alongside Solr?
>
> Hi All-
>
> I've just recently began playing with Apache Solr, and it seems to be
> a perfect fit for our project (http://code.google.com/p/dase/). I've
> been quite surprised at both how easy Solr was to get up and running
> and how flexible it seems to be. I've been tempted to use it for not
> just search, but document storage as well. Seems, though, this is not
> the best road to go down.
>
> I'd like to know if there are recommendations for a document store (or
> distributed hash table) that would work well alongside Solr.
> Basically, I'd like to be able to deploy in Tomcat and interact w/ the
> store over http (like Solr). Recommendations I've seen include
> Project Voldemort, BDB, BananaDB, CouchDB, etc. I'd be quite
> interested to hear comments about pluses/minuses of any of those or
> other options OR comments about Solr suitability as a document store.
>
> thanks-
> Peter Keane
> daseproject.org
Re: recommendation for document store to use alongside Solr?
Posted by Steve Conover <sc...@gmail.com>.
It's not distributed, but I like Tokyo Cabinet. Tokyo Tyrant adds
http on top, I've never used Tyrant though.
-Steve
On Tue, May 26, 2009 at 7:50 AM, Peter Keane <pk...@mail.utexas.edu> wrote:
> Hi All-
>
> I've just recently began playing with Apache Solr, and it seems to be
> a perfect fit for our project (http://code.google.com/p/dase/). I've
> been quite surprised at both how easy Solr was to get up and running
> and how flexible it seems to be. I've been tempted to use it for not
> just search, but document storage as well. Seems, though, this is not
> the best road to go down.
>
> I'd like to know if there are recommendations for a document store (or
> distributed hash table) that would work well alongside Solr.
> Basically, I'd like to be able to deploy in Tomcat and interact w/ the
> store over http (like Solr). Recommendations I've seen include
> Project Voldemort, BDB, BananaDB, CouchDB, etc. I'd be quite
> interested to hear comments about pluses/minuses of any of those or
> other options OR comments about Solr suitability as a document store.
>
> thanks-
> Peter Keane
> daseproject.org
>