You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Denis Bazhenov <do...@gmail.com> on 2011/10/17 04:32:30 UTC
IndexReader#reopen() on externally changed index
We have situation when lucene index is replicated over network. And on that machine reader reopen doesn't make new documents visible to a search.
As far as I know IndexReader.reopen() call does work only if changes are applied using the linked IndexWriter. My question is: how can I implement efficient index reopen (only new segments should be read) when index is changed externally?
---
Denis Bazhenov <do...@gmail.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexReader#reopen() on externally changed index
Posted by Michael McCandless <lu...@mikemccandless.com>.
That's a good idea, if your index is "large enough", and/or you make
heavy use of FieldCache (eg, sorting by field), regardless of whether
you use NRT or "normal" commit + reopen to reopen your reader.
Mike McCandless
http://blog.mikemccandless.com
On Sun, Oct 30, 2011 at 7:36 PM, Denis Bazhenov <do...@gmail.com> wrote:
> Well, if so I guess I should use IndexWarmer to warm up IndexReader before publishing reference to search clients. At least it will pre read all the segments in RAM before issuing search.
>
> On Oct 17, 2011, at 9:47 PM, Michael McCandless wrote:
>
>> You'll have to call .commit() from the IndexWriter to make the changes
>> externally visible.
>>
>> The call IndexReader.reopen to get a reader seeing the committed
>> changes; the reopen will be efficient (only open "new" segments vs the
>> old reader).
>>
>> It's still best to use near-real-time reader when possible (ie, open
>> the IndexReader from the IndexWriter), but it sounds like in your case
>> this is not possible since writer and reader on different
>> JVMs/machines across a network.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sun, Oct 16, 2011 at 10:32 PM, Denis Bazhenov <do...@gmail.com> wrote:
>>> We have situation when lucene index is replicated over network. And on that machine reader reopen doesn't make new documents visible to a search.
>>>
>>> As far as I know IndexReader.reopen() call does work only if changes are applied using the linked IndexWriter. My question is: how can I implement efficient index reopen (only new segments should be read) when index is changed externally?
>>> ---
>>> Denis Bazhenov <do...@gmail.com>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---
> Denis Bazhenov <do...@gmail.com>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexReader#reopen() on externally changed index
Posted by Denis Bazhenov <do...@gmail.com>.
Well, if so I guess I should use IndexWarmer to warm up IndexReader before publishing reference to search clients. At least it will pre read all the segments in RAM before issuing search.
On Oct 17, 2011, at 9:47 PM, Michael McCandless wrote:
> You'll have to call .commit() from the IndexWriter to make the changes
> externally visible.
>
> The call IndexReader.reopen to get a reader seeing the committed
> changes; the reopen will be efficient (only open "new" segments vs the
> old reader).
>
> It's still best to use near-real-time reader when possible (ie, open
> the IndexReader from the IndexWriter), but it sounds like in your case
> this is not possible since writer and reader on different
> JVMs/machines across a network.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Oct 16, 2011 at 10:32 PM, Denis Bazhenov <do...@gmail.com> wrote:
>> We have situation when lucene index is replicated over network. And on that machine reader reopen doesn't make new documents visible to a search.
>>
>> As far as I know IndexReader.reopen() call does work only if changes are applied using the linked IndexWriter. My question is: how can I implement efficient index reopen (only new segments should be read) when index is changed externally?
>> ---
>> Denis Bazhenov <do...@gmail.com>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---
Denis Bazhenov <do...@gmail.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexReader#reopen() on externally changed index
Posted by Michael McCandless <lu...@mikemccandless.com>.
You'll have to call .commit() from the IndexWriter to make the changes
externally visible.
The call IndexReader.reopen to get a reader seeing the committed
changes; the reopen will be efficient (only open "new" segments vs the
old reader).
It's still best to use near-real-time reader when possible (ie, open
the IndexReader from the IndexWriter), but it sounds like in your case
this is not possible since writer and reader on different
JVMs/machines across a network.
Mike McCandless
http://blog.mikemccandless.com
On Sun, Oct 16, 2011 at 10:32 PM, Denis Bazhenov <do...@gmail.com> wrote:
> We have situation when lucene index is replicated over network. And on that machine reader reopen doesn't make new documents visible to a search.
>
> As far as I know IndexReader.reopen() call does work only if changes are applied using the linked IndexWriter. My question is: how can I implement efficient index reopen (only new segments should be read) when index is changed externally?
> ---
> Denis Bazhenov <do...@gmail.com>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexReader#reopen() on externally changed index
Posted by "Devon H. O'Dell" <de...@gmail.com>.
In my experience, reopen will find all changes on an index, whether it was
modified by the same process or not. If you're replicating over a network,
you might need some barrier / lock around the reopen call to make sure the
replicated index is complete. Obviously with something as fickle as a
network, you may need some extra protection around this. It has worked best
for me to replicate using IndexWriter wrapping some transport protocol.
--dho
On Oct 16, 2011 10:33 PM, "Denis Bazhenov" <do...@gmail.com> wrote:
> We have situation when lucene index is replicated over network. And on that
> machine reader reopen doesn't make new documents visible to a search.
>
> As far as I know IndexReader.reopen() call does work only if changes are
> applied using the linked IndexWriter. My question is: how can I implement
> efficient index reopen (only new segments should be read) when index is
> changed externally?
> ---
> Denis Bazhenov <do...@gmail.com>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>