You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Vinicius Carvalho <vi...@gmail.com> on 2008/03/19 15:17:13 UTC
Lucene on a cluster environment
Hello there! I have just started with lucene. Bought the Lucene in action
book [right now I'm at chap 4, plus the 10th chapter, great explanation by
Terence from jGuru, really nice stuff], also I'm reading most that I can at
the wiki :)
Still a bit lost with some stuff, mostly with clusters :)
Our app is running in a cluster environment, now we are going to add search
support for it, and lucene was the choice for it. Since I have many SLSB
spread across the cluster, I would like to know some blueprints to use
lucene in these situation.
Our first idea, was to store the IndexWriter as an MBean inside Jboss and
use it as an HA-Singleton, which would guarantee its uniqueness inside the
cluster. I just would like to get some feedback from you guys on it :)
Best regards
--
"In a world without fences and walls, who needs Gates and Windows?"
RE: Lucene on a cluster environment
Posted by Ro...@ancept.com.
No noticeable performance hit, searches are not a bottleneck in our
system. We don't have disk redundancy.
Dragon Fly <dr...@hotmail.com>
03/19/2008 11:47 AM
Please respond to
java-user@lucene.apache.org
To
<ja...@lucene.apache.org>
cc
Subject
RE: Lucene on a cluster environment
Hi Robert,
Did you run into any performance issues (because multiple searchers
accessed a single index on a shared directory)? Also, did you employ some
redundancy scheme to ensure that the shared directory is always
"available"? Thank you.
> To: java-user@lucene.apache.org
> Subject: Re: Lucene on a cluster environment
> From: Robert.Hastings@ancept.com
> Date: Wed, 19 Mar 2008 10:16:29 -0500
>
> We went through this a couple of years ago. I couldn't find the thread
in
> the archive but the jist of it is as follows:
>
> 1. We have a singleton thread that does all of the writing. new
> Documents and deletions are queued to the writer via a database table.
> 2. Since searchers are "point in time" you can user the same searcher
in
> each cluster member ( this limits the amount of open files), but you
need
> to refresh the searcher when the index changes. We use a SearcherHolder
> and a single SearcherManager class to manage this. Events are sent by
the
> writer and received by the searcher holder when the index changes.
> 3. Finally, older versions (1.4 and earlier) of Lucene had problems
with
> having the index on a shared directory. I think most of these issues
have
> been resolved.
>
> Good Luck.
>
>
>
> "Vinicius Carvalho" <vi...@gmail.com>
> 03/19/2008 09:17 AM
> Please respond to
> java-user@lucene.apache.org
>
>
> To
> java-user@lucene.apache.org
> cc
>
> Subject
> Lucene on a cluster environment
>
>
>
>
>
>
> Hello there! I have just started with lucene. Bought the Lucene in
action
> book [right now I'm at chap 4, plus the 10th chapter, great explanation
by
> Terence from jGuru, really nice stuff], also I'm reading most that I can
> at
> the wiki :)
> Still a bit lost with some stuff, mostly with clusters :)
> Our app is running in a cluster environment, now we are going to add
> search
> support for it, and lucene was the choice for it. Since I have many SLSB
> spread across the cluster, I would like to know some blueprints to use
> lucene in these situation.
>
> Our first idea, was to store the IndexWriter as an MBean inside Jboss
and
> use it as an HA-Singleton, which would guarantee its uniqueness inside
the
> cluster. I just would like to get some feedback from you guys on it :)
>
> Best regards
>
> --
> "In a world without fences and walls, who needs Gates and Windows?"
>
_________________________________________________________________
Don't get caught with egg on your face. Play chicktionary!
http://club.live.com/chicktionary.aspx?icid=chick_wlhmtextlink1_feb
RE: Lucene on a cluster environment
Posted by Dragon Fly <dr...@hotmail.com>.
Hi Robert,
Did you run into any performance issues (because multiple searchers accessed a single index on a shared directory)? Also, did you employ some redundancy scheme to ensure that the shared directory is always "available"? Thank you.
> To: java-user@lucene.apache.org
> Subject: Re: Lucene on a cluster environment
> From: Robert.Hastings@ancept.com
> Date: Wed, 19 Mar 2008 10:16:29 -0500
>
> We went through this a couple of years ago. I couldn't find the thread in
> the archive but the jist of it is as follows:
>
> 1. We have a singleton thread that does all of the writing. new
> Documents and deletions are queued to the writer via a database table.
> 2. Since searchers are "point in time" you can user the same searcher in
> each cluster member ( this limits the amount of open files), but you need
> to refresh the searcher when the index changes. We use a SearcherHolder
> and a single SearcherManager class to manage this. Events are sent by the
> writer and received by the searcher holder when the index changes.
> 3. Finally, older versions (1.4 and earlier) of Lucene had problems with
> having the index on a shared directory. I think most of these issues have
> been resolved.
>
> Good Luck.
>
>
>
> "Vinicius Carvalho" <vi...@gmail.com>
> 03/19/2008 09:17 AM
> Please respond to
> java-user@lucene.apache.org
>
>
> To
> java-user@lucene.apache.org
> cc
>
> Subject
> Lucene on a cluster environment
>
>
>
>
>
>
> Hello there! I have just started with lucene. Bought the Lucene in action
> book [right now I'm at chap 4, plus the 10th chapter, great explanation by
> Terence from jGuru, really nice stuff], also I'm reading most that I can
> at
> the wiki :)
> Still a bit lost with some stuff, mostly with clusters :)
> Our app is running in a cluster environment, now we are going to add
> search
> support for it, and lucene was the choice for it. Since I have many SLSB
> spread across the cluster, I would like to know some blueprints to use
> lucene in these situation.
>
> Our first idea, was to store the IndexWriter as an MBean inside Jboss and
> use it as an HA-Singleton, which would guarantee its uniqueness inside the
> cluster. I just would like to get some feedback from you guys on it :)
>
> Best regards
>
> --
> "In a world without fences and walls, who needs Gates and Windows?"
>
_________________________________________________________________
Don't get caught with egg on your face. Play chicktionary!
http://club.live.com/chicktionary.aspx?icid=chick_wlhmtextlink1_feb
Fwd: Lucene on a cluster environment
Posted by Vinicius Carvalho <vi...@gmail.com>.
Thanks a lot for sharing this :)
I'll try to follow your guidelines
Regards
---------- Forwarded message ----------
From: <Ro...@ancept.com>
Date: Wed, Mar 19, 2008 at 12:16 PM
Subject: Re: Lucene on a cluster environment
To: java-user@lucene.apache.org
We went through this a couple of years ago. I couldn't find the thread in
the archive but the jist of it is as follows:
1. We have a singleton thread that does all of the writing. new
Documents and deletions are queued to the writer via a database table.
2. Since searchers are "point in time" you can user the same searcher in
each cluster member ( this limits the amount of open files), but you need
to refresh the searcher when the index changes. We use a SearcherHolder
and a single SearcherManager class to manage this. Events are sent by the
writer and received by the searcher holder when the index changes.
3. Finally, older versions (1.4 and earlier) of Lucene had problems with
having the index on a shared directory. I think most of these issues have
been resolved.
Good Luck.
"Vinicius Carvalho" <vi...@gmail.com>
03/19/2008 09:17 AM
Please respond to
java-user@lucene.apache.org
To
java-user@lucene.apache.org
cc
Subject
Lucene on a cluster environment
Hello there! I have just started with lucene. Bought the Lucene in action
book [right now I'm at chap 4, plus the 10th chapter, great explanation by
Terence from jGuru, really nice stuff], also I'm reading most that I can
at
the wiki :)
Still a bit lost with some stuff, mostly with clusters :)
Our app is running in a cluster environment, now we are going to add
search
support for it, and lucene was the choice for it. Since I have many SLSB
spread across the cluster, I would like to know some blueprints to use
lucene in these situation.
Our first idea, was to store the IndexWriter as an MBean inside Jboss and
use it as an HA-Singleton, which would guarantee its uniqueness inside the
cluster. I just would like to get some feedback from you guys on it :)
Best regards
--
"In a world without fences and walls, who needs Gates and Windows?"
--
"In a world without fences and walls, who needs Gates and Windows?"
Re: Lucene on a cluster environment
Posted by Ro...@ancept.com.
We went through this a couple of years ago. I couldn't find the thread in
the archive but the jist of it is as follows:
1. We have a singleton thread that does all of the writing. new
Documents and deletions are queued to the writer via a database table.
2. Since searchers are "point in time" you can user the same searcher in
each cluster member ( this limits the amount of open files), but you need
to refresh the searcher when the index changes. We use a SearcherHolder
and a single SearcherManager class to manage this. Events are sent by the
writer and received by the searcher holder when the index changes.
3. Finally, older versions (1.4 and earlier) of Lucene had problems with
having the index on a shared directory. I think most of these issues have
been resolved.
Good Luck.
"Vinicius Carvalho" <vi...@gmail.com>
03/19/2008 09:17 AM
Please respond to
java-user@lucene.apache.org
To
java-user@lucene.apache.org
cc
Subject
Lucene on a cluster environment
Hello there! I have just started with lucene. Bought the Lucene in action
book [right now I'm at chap 4, plus the 10th chapter, great explanation by
Terence from jGuru, really nice stuff], also I'm reading most that I can
at
the wiki :)
Still a bit lost with some stuff, mostly with clusters :)
Our app is running in a cluster environment, now we are going to add
search
support for it, and lucene was the choice for it. Since I have many SLSB
spread across the cluster, I would like to know some blueprints to use
lucene in these situation.
Our first idea, was to store the IndexWriter as an MBean inside Jboss and
use it as an HA-Singleton, which would guarantee its uniqueness inside the
cluster. I just would like to get some feedback from you guys on it :)
Best regards
--
"In a world without fences and walls, who needs Gates and Windows?"