You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Dmitry Goldenberg <dm...@weblayers.com> on 2006/04/06 21:30:27 UTC
RE: Distributed Lucene.. - clustering as a requirement
I firmly believe that clustering support should be a part of Lucene. We've tried implementing it ourselves and so far have been unsuccessful. We tried storing Lucene indices in a database that is the back-end repository for our app in a clustered environment and could not overcome the indexing exceptions in our custom Directory implementation.
I think it'd be perfect if some of the Lucene gurus were to implement an RDBMS-backed Directory and post it (in addition to the sleepycat.db package that's currently in contrib). The nitty-gritties of dealing with Lucene indexing structures at a single byte level are just way too much trouble to deal with for application integrator like myself.
- Dmitry
________________________________
From: Samuru Jackson [mailto:samurujackson@googlemail.com]
Sent: Mon 3/6/2006 10:05 AM
To: java-user@lucene.apache.org
Subject: Re: Distributed Lucene..
Do you plan to release some kind of a commerical product including an API?
I ask because I'm evaluating different technologies for a prototype
which is part of my diploma thesis.
The problem is that I have to deal with real huge data amounts and one
machine is simply not enough to handle those amounts of data.
Lucene seems to be a good choice but it won't scale up for real big
data amounts. So I thought about expanding the indexes over several
machines in junks so that it fits into the memory of those machines.
One machine should collect the results and calculate some kind of
score out of the delivered hits from the machines.
As I'm not familiar with the concrete mechanisms of Lucene this is
just a naive thought, but I think that such a clustering mechanism
could become a killer app.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Clusterization of searching
Posted by an...@orbita1.ru.
What be way for clusterizations of searching?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Distributed Lucene.. - clustering as a requirement
Posted by Dmitry Goldenberg <dm...@weblayers.com>.
I guess Compass is probably the way to go - http://www.opensymphony.com/compass/
________________________________
From: Prasenjit Mukherjee [mailto:prasenjitm@aol.com]
Sent: Tue 4/11/2006 2:45 AM
To: java-user@lucene.apache.org
Subject: Re: Distributed Lucene.. - clustering as a requirement
Agreed, an inverted index cannot be efficiently maintained in a
B-tree(hence RDBMS). But I think we can(or should) have the option of
a B-tree based storage for unindexed fields, whereas for indexed fields
we can use the existing lucene's architecture.
prasen
cutting@apache.org wrote:
> Dmitry Goldenberg wrote:
>
>> For an enterprise-level application, Lucene appears too file-system and
>
> too byte-sequence-centric a technology. Just my opinion. The
> Directory API is just too low-level.
>
> There are good reasons why Lucene is not built on top of a RDBMS. An
> inverted index is not efficiently maintained in a B-Tree, and B-Trees
> are the foundation of RDBMSes.
>
> http://www.haifa.ibm.com/Workshops/ir2005/papers/DougCutting-Haifa05.pdf
>
>> I'd be OK with an RDBMS-based Directory implementation I could take
>> and use. But generally, I think the Lucene authors might like to
>> take a step back and consider splitting off the repository and making
>> it more extensible and high-level. Perhaps something like JSR-170
>> (Java repository API) may be a good route to go....
>
>
> If you have concrete ideas for an improvements to Lucene's Directory
> interface, please propose them to the java-dev mailing list, ideally
> as a patch.
>
> Cheers,
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Distributed Lucene.. - clustering as a requirement
Posted by Prasenjit Mukherjee <pr...@aol.com>.
Agreed, an inverted index cannot be efficiently maintained in a
B-tree(hence RDBMS). But I think we can(or should) have the option of
a B-tree based storage for unindexed fields, whereas for indexed fields
we can use the existing lucene's architecture.
prasen
cutting@apache.org wrote:
> Dmitry Goldenberg wrote:
>
>> For an enterprise-level application, Lucene appears too file-system and
>
> too byte-sequence-centric a technology. Just my opinion. The
> Directory API is just too low-level.
>
> There are good reasons why Lucene is not built on top of a RDBMS. An
> inverted index is not efficiently maintained in a B-Tree, and B-Trees
> are the foundation of RDBMSes.
>
> http://www.haifa.ibm.com/Workshops/ir2005/papers/DougCutting-Haifa05.pdf
>
>> I'd be OK with an RDBMS-based Directory implementation I could take
>> and use. But generally, I think the Lucene authors might like to
>> take a step back and consider splitting off the repository and making
>> it more extensible and high-level. Perhaps something like JSR-170
>> (Java repository API) may be a good route to go....
>
>
> If you have concrete ideas for an improvements to Lucene's Directory
> interface, please propose them to the java-dev mailing list, ideally
> as a patch.
>
> Cheers,
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Distributed Lucene.. - clustering as a requirement
Posted by Doug Cutting <cu...@apache.org>.
Dmitry Goldenberg wrote:
> For an enterprise-level application, Lucene appears too file-system and
too byte-sequence-centric a technology. Just my opinion. The Directory
API is just too low-level.
There are good reasons why Lucene is not built on top of a RDBMS. An
inverted index is not efficiently maintained in a B-Tree, and B-Trees
are the foundation of RDBMSes.
http://www.haifa.ibm.com/Workshops/ir2005/papers/DougCutting-Haifa05.pdf
> I'd be OK with an RDBMS-based Directory implementation I could take and use. But generally, I think the Lucene authors might like to take a step back and consider splitting off the repository and making it more extensible and high-level. Perhaps something like JSR-170 (Java repository API) may be a good route to go....
If you have concrete ideas for an improvements to Lucene's Directory
interface, please propose them to the java-dev mailing list, ideally as
a patch.
Cheers,
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Distributed Lucene.. - clustering as a requirement
Posted by Dmitry Goldenberg <dm...@weblayers.com>.
I think it's a good idea. For an enterprise-level application, Lucene appears too file-system and too byte-sequence-centric a technology. Just my opinion. The Directory API is just too low-level.
I'd be OK with an RDBMS-based Directory implementation I could take and use. But generally, I think the Lucene authors might like to take a step back and consider splitting off the repository and making it more extensible and high-level. Perhaps something like JSR-170 (Java repository API) may be a good route to go....
________________________________
From: Chris Lamprecht [mailto:clamprecht@gmail.com]
Sent: Thu 4/6/2006 3:55 PM
To: java-user@lucene.apache.org
Subject: Re: Distributed Lucene.. - clustering as a requirement
What about using lucene just for searching (i.e., no stored fields
except maybe one "ID" primary key field), and using an RDBMS for
storing the actual "documents"? This way you're using lucene for what
lucene is best at, and using the database for what it's good at. At
least up to a point -- RDBMSs have their limits too. OR maybe if you
have a huge dataset, you might want to check out Nutch.
On 4/6/06, Dmitry Goldenberg <dm...@weblayers.com> wrote:
> I firmly believe that clustering support should be a part of Lucene. We've tried implementing it ourselves and so far have been unsuccessful. We tried storing Lucene indices in a database that is the back-end repository for our app in a clustered environment and could not overcome the indexing exceptions in our custom Directory implementation.
>
> I think it'd be perfect if some of the Lucene gurus were to implement an RDBMS-backed Directory and post it (in addition to the sleepycat.db package that's currently in contrib). The nitty-gritties of dealing with Lucene indexing structures at a single byte level are just way too much trouble to deal with for application integrator like myself.
>
> - Dmitry
>
> ________________________________
>
> From: Samuru Jackson [mailto:samurujackson@googlemail.com]
> Sent: Mon 3/6/2006 10:05 AM
> To: java-user@lucene.apache.org
> Subject: Re: Distributed Lucene..
>
>
>
> Do you plan to release some kind of a commerical product including an API?
>
> I ask because I'm evaluating different technologies for a prototype
> which is part of my diploma thesis.
>
> The problem is that I have to deal with real huge data amounts and one
> machine is simply not enough to handle those amounts of data.
>
> Lucene seems to be a good choice but it won't scale up for real big
> data amounts. So I thought about expanding the indexes over several
> machines in junks so that it fits into the memory of those machines.
>
> One machine should collect the results and calculate some kind of
> score out of the delivered hits from the machines.
>
> As I'm not familiar with the concrete mechanisms of Lucene this is
> just a naive thought, but I think that such a clustering mechanism
> could become a killer app.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Distributed Lucene.. - clustering as a requirement
Posted by Chris Lamprecht <cl...@gmail.com>.
What about using lucene just for searching (i.e., no stored fields
except maybe one "ID" primary key field), and using an RDBMS for
storing the actual "documents"? This way you're using lucene for what
lucene is best at, and using the database for what it's good at. At
least up to a point -- RDBMSs have their limits too. OR maybe if you
have a huge dataset, you might want to check out Nutch.
On 4/6/06, Dmitry Goldenberg <dm...@weblayers.com> wrote:
> I firmly believe that clustering support should be a part of Lucene. We've tried implementing it ourselves and so far have been unsuccessful. We tried storing Lucene indices in a database that is the back-end repository for our app in a clustered environment and could not overcome the indexing exceptions in our custom Directory implementation.
>
> I think it'd be perfect if some of the Lucene gurus were to implement an RDBMS-backed Directory and post it (in addition to the sleepycat.db package that's currently in contrib). The nitty-gritties of dealing with Lucene indexing structures at a single byte level are just way too much trouble to deal with for application integrator like myself.
>
> - Dmitry
>
> ________________________________
>
> From: Samuru Jackson [mailto:samurujackson@googlemail.com]
> Sent: Mon 3/6/2006 10:05 AM
> To: java-user@lucene.apache.org
> Subject: Re: Distributed Lucene..
>
>
>
> Do you plan to release some kind of a commerical product including an API?
>
> I ask because I'm evaluating different technologies for a prototype
> which is part of my diploma thesis.
>
> The problem is that I have to deal with real huge data amounts and one
> machine is simply not enough to handle those amounts of data.
>
> Lucene seems to be a good choice but it won't scale up for real big
> data amounts. So I thought about expanding the indexes over several
> machines in junks so that it fits into the memory of those machines.
>
> One machine should collect the results and calculate some kind of
> score out of the delivered hits from the machines.
>
> As I'm not familiar with the concrete mechanisms of Lucene this is
> just a naive thought, but I think that such a clustering mechanism
> could become a killer app.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org