Queries on Lucene Replication Approach

[sandy A <sa...@gmail.com>]

Hi Lucene Community Team,

I have a couple of queries related to Lucene replication and would
appreciate your guidance:

*Query 1:*
Is it safe to use tools like *rsync* (on Linux) or *robocopy* (on Windows)
for copying Lucene segment files from one server to another?
I want to understand if there are any potential risks (e.g., partial
copies, consistency issues) or best practices recommended for such
approaches.

*Query 2:*
I noticed that *NRTLuceneReplication* is still marked as "in development."
Could you please clarify why it is not yet considered production-ready? Are
there known limitations or stability concerns that we should be aware of
before experimenting with it?

Any insights or suggestions would be really helpful.


With Regards,
SanthoshKumar A

Re: Queries on Lucene Replication Approach

[Viliam Ďurina <vi...@gmail.com>]

In this context I would also note that while copying is progress, an
attempt to open the directory might report a corrupted index due to a
commit not having all of its files available or due to partially copied
files. For example, if your query servers are calling
`DirectoryReader.openIfChanged()` in regular intervals, sooner or later
they will crash. So you should coordinate reader refresh with the
synchronization.
Viliam

On Mon, Aug 25, 2025 at 6:05 PM Steven Schlansker <
stevenschlansker@gmail.com> wrote:

> Hi, we use Lucene NRT replication in production.
>
> For consistent snapshots, we use SnapshotDeletionPolicy to open a
> snapshot, and then copy the snapshot'ed
> files with your tool of choice like rsync. Without a snapshot, I don't
> think such tools work reliably - you can copy
> commit metadata without the data it refers to, for example.
>
> We've run Lucene NRT replication in production for about 4 years now.
> Overall very happy with it. We intentionally run a simple configuration
> (one indexer, N searchers)
> to avoid difficult problems around leader election and network
> partitioning.
>
> Definitely took a bit of learning, but overall we would choose this path
> again.
> It is more "hands on" than just spinning up an Elasticsearch cluster - but
> on the other hand
> our previous Elasticsearch cluster took a team just just for "care and
> feeding" as the cluster had
> split-brain issues etc. Compare to Lucene NRT replication, where a single
> dev and no ops people
> can easily manage the entire stack, since all state is on a single master
> node.
>
> > On Aug 25, 2025, at 8:36 AM, Adrien Grand <jp...@gmail.com> wrote:
> >
> > Hello,
> >
> > If you can guarantee no concurrent updates while you are doing the copy,
> > then 3rd-party tools that replicate files such as fsync are fine. But
> > otherwise or if you have the choice, I would recommend to use Lucene's
> > built-in support for replication, which can also work if there are
> > concurrent updates to the index.
> >
> > What is NRTLuceneReplication? I cannot find references to it. Lucene's
> > replicator module does have NRT (near-realtime) support through:
> >
> https://lucene.apache.org/core/10_2_0/replicator/org/apache/lucene/replicator/nrt/package-summary.html
> > .
> >
> > On Mon, Aug 25, 2025 at 5:18 PM sandy A <sa...@gmail.com>
> wrote:
> >
> >> Hi Lucene Community Team,
> >>
> >> I have a couple of queries related to Lucene replication and would
> >> appreciate your guidance:
> >>
> >> *Query 1:*
> >> Is it safe to use tools like *rsync* (on Linux) or *robocopy* (on
> Windows)
> >> for copying Lucene segment files from one server to another?
> >> I want to understand if there are any potential risks (e.g., partial
> >> copies, consistency issues) or best practices recommended for such
> >> approaches.
> >>
> >> *Query 2:*
> >> I noticed that *NRTLuceneReplication* is still marked as "in
> development."
> >> Could you please clarify why it is not yet considered production-ready?
> Are
> >> there known limitations or stability concerns that we should be aware of
> >> before experimenting with it?
> >>
> >> Any insights or suggestions would be really helpful.
> >>
> >>
> >> With Regards,
> >> SanthoshKumar A
> >>
> >
> >
> > --
> > Adrien
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Queries on Lucene Replication Approach

[Steven Schlansker <st...@gmail.com>]

Hi, we use Lucene NRT replication in production.

For consistent snapshots, we use SnapshotDeletionPolicy to open a snapshot, and then copy the snapshot'ed
files with your tool of choice like rsync. Without a snapshot, I don't think such tools work reliably - you can copy
commit metadata without the data it refers to, for example.

We've run Lucene NRT replication in production for about 4 years now.
Overall very happy with it. We intentionally run a simple configuration (one indexer, N searchers)
to avoid difficult problems around leader election and network partitioning.

Definitely took a bit of learning, but overall we would choose this path again.
It is more "hands on" than just spinning up an Elasticsearch cluster - but on the other hand
our previous Elasticsearch cluster took a team just just for "care and feeding" as the cluster had
split-brain issues etc. Compare to Lucene NRT replication, where a single dev and no ops people 
can easily manage the entire stack, since all state is on a single master node.

> On Aug 25, 2025, at 8:36 AM, Adrien Grand <jp...@gmail.com> wrote:
> 
> Hello,
> 
> If you can guarantee no concurrent updates while you are doing the copy,
> then 3rd-party tools that replicate files such as fsync are fine. But
> otherwise or if you have the choice, I would recommend to use Lucene's
> built-in support for replication, which can also work if there are
> concurrent updates to the index.
> 
> What is NRTLuceneReplication? I cannot find references to it. Lucene's
> replicator module does have NRT (near-realtime) support through:
> https://lucene.apache.org/core/10_2_0/replicator/org/apache/lucene/replicator/nrt/package-summary.html
> .
> 
> On Mon, Aug 25, 2025 at 5:18 PM sandy A <sa...@gmail.com> wrote:
> 
>> Hi Lucene Community Team,
>> 
>> I have a couple of queries related to Lucene replication and would
>> appreciate your guidance:
>> 
>> *Query 1:*
>> Is it safe to use tools like *rsync* (on Linux) or *robocopy* (on Windows)
>> for copying Lucene segment files from one server to another?
>> I want to understand if there are any potential risks (e.g., partial
>> copies, consistency issues) or best practices recommended for such
>> approaches.
>> 
>> *Query 2:*
>> I noticed that *NRTLuceneReplication* is still marked as "in development."
>> Could you please clarify why it is not yet considered production-ready? Are
>> there known limitations or stability concerns that we should be aware of
>> before experimenting with it?
>> 
>> Any insights or suggestions would be really helpful.
>> 
>> 
>> With Regards,
>> SanthoshKumar A
>> 
> 
> 
> -- 
> Adrien


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Queries on Lucene Replication Approach

[Adrien Grand <jp...@gmail.com>]

Hello,

If you can guarantee no concurrent updates while you are doing the copy,
then 3rd-party tools that replicate files such as fsync are fine. But
otherwise or if you have the choice, I would recommend to use Lucene's
built-in support for replication, which can also work if there are
concurrent updates to the index.

What is NRTLuceneReplication? I cannot find references to it. Lucene's
replicator module does have NRT (near-realtime) support through:
https://lucene.apache.org/core/10_2_0/replicator/org/apache/lucene/replicator/nrt/package-summary.html
.

On Mon, Aug 25, 2025 at 5:18 PM sandy A <sa...@gmail.com> wrote:

> Hi Lucene Community Team,
>
> I have a couple of queries related to Lucene replication and would
> appreciate your guidance:
>
> *Query 1:*
> Is it safe to use tools like *rsync* (on Linux) or *robocopy* (on Windows)
> for copying Lucene segment files from one server to another?
> I want to understand if there are any potential risks (e.g., partial
> copies, consistency issues) or best practices recommended for such
> approaches.
>
> *Query 2:*
> I noticed that *NRTLuceneReplication* is still marked as "in development."
> Could you please clarify why it is not yet considered production-ready? Are
> there known limitations or stability concerns that we should be aware of
> before experimenting with it?
>
> Any insights or suggestions would be really helpful.
>
>
> With Regards,
> SanthoshKumar A
>


-- 
Adrien