You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Gopal Sharma <go...@algonomy.com> on 2023/10/19 09:48:45 UTC

Lucene Index Writer in a distributed system

Hello Team,

I am new to Lucene and want to use Lucene in a distributed system to write
in a Amazon EFS index.

As per my understanding, the index writer for a particular index needs to
be opened by 1 server only. Is there a way we can achieved this in
distributed system to write parallelly in Lucence

Any document/article where i can read about this problem will be of great
help, Many thanks in advance!!

Gopal

Re: Lucene Index Writer in a distributed system

Posted by Cody Amen <co...@icloud.com.INVALID>.
Zookeeper, right? Look how Zookeeper is used in Solr, but Zookeeper does exactly what you want, I believe.

Sent from my iPhone

> On Oct 19, 2023, at 3:49 AM, Gopal Sharma <go...@algonomy.com> wrote:
> 
> Hello Team,
> 
> I am new to Lucene and want to use Lucene in a distributed system to write
> in a Amazon EFS index.
> 
> As per my understanding, the index writer for a particular index needs to
> be opened by 1 server only. Is there a way we can achieved this in
> distributed system to write parallelly in Lucence
> 
> Any document/article where i can read about this problem will be of great
> help, Many thanks in advance!!
> 
> Gopal

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Lucene Index Writer in a distributed system

Posted by Michael McCandless <lu...@mikemccandless.com>.
Hi Gopal,

Indeed, for a single Lucene index, only one writer may be open at a time.
Lucene tries to catch you if you mess this up, using file-based locking.

If you really need concurrent indexing, you could have N IndexWriters each
writing into a private Directory, and then periodically use addIndexes to
"reduce" these indices down into a single index which you then use for
searching.

Mike McCandless

http://blog.mikemccandless.com


On Thu, Oct 19, 2023 at 5:49 AM Gopal Sharma <go...@algonomy.com>
wrote:

> Hello Team,
>
> I am new to Lucene and want to use Lucene in a distributed system to write
> in a Amazon EFS index.
>
> As per my understanding, the index writer for a particular index needs to
> be opened by 1 server only. Is there a way we can achieved this in
> distributed system to write parallelly in Lucence
>
> Any document/article where i can read about this problem will be of great
> help, Many thanks in advance!!
>
> Gopal
>