You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Olivier Jaquemet <ol...@jalios.com> on 2005/10/02 19:11:07 UTC
Renewing IndexSearcher on index change.
Hi all,
As I read it on LIA, and as it has already been said on the mailing list
multiple times, you only need one IndexSearcher for all your thread, and
when your index change, you just need to create a new one to reflect
changes.
Otis said in this post you could replaced old searcher and let it being
collected by the GC.
http://mail-archives.apache.org/mod_mbox/lucene-java-user/200502.mbox/%3c20050218222949.15826.qmail@web31111.mail.mud.yahoo.com%3e
Note that my question is maybe more java related than anything else,
but, anyway...
In the source code of IndexSearcher (and all searchers), I could not
find any finalize method that closes the searcher, how can you be sure
you old searcher is going to be closed? am I missing something? Is there
still a step I need to do in order to close the searcher properly ? Do I
need to create a facade to the searcher, keep my own reference of uses
and call a close in a finalize method?
Thanks in advance for your answers :)
Olivier
PS: Another small question while I'm there, if a long indexation of a
lot Documents occurs, can the old searcher still be used for some time
while the indexation takes place, without problems? Cause I encoutered
an ArrayOutOfBoundException under some scorer class, unfortunately I did
not kept the trace.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by Olivier Jaquemet <ol...@jalios.com>.
any thoughts on this subjects ?
Olivier Jaquemet wrote:
> Hi all,
>
> As I read it on LIA, and as it has already been said on the mailing
> list multiple times, you only need one IndexSearcher for all your
> thread, and when your index change, you just need to create a new one
> to reflect changes.
>
> Otis said in this post you could replaced old searcher and let it
> being collected by the GC.
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200502.mbox/%3c20050218222949.15826.qmail@web31111.mail.mud.yahoo.com%3e
>
>
> Note that my question is maybe more java related than anything else,
> but, anyway...
> In the source code of IndexSearcher (and all searchers), I could not
> find any finalize method that closes the searcher, how can you be sure
> you old searcher is going to be closed? am I missing something? Is
> there still a step I need to do in order to close the searcher
> properly ? Do I need to create a facade to the searcher, keep my own
> reference of uses and call a close in a finalize method?
>
> Thanks in advance for your answers :)
>
> Olivier
>
> PS: Another small question while I'm there, if a long indexation of a
> lot Documents occurs, can the old searcher still be used for some time
> while the indexation takes place, without problems? Cause I encoutered
> an ArrayOutOfBoundException under some scorer class, unfortunately I
> did not kept the trace.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
Olivier Jaquemet <ol...@jalios.com>
Ingénieur R&D Jalios S.A.
Tel: 01.39.23.92.83
http://www.jalios.com/
http://support.jalios.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by "J.J. Larrea" <jj...@panix.com>.
Oops! Yes that's correct. Thanks for catching it... - J.J.
At 10:33 AM -0700 10/4/05, Chris Hostetter wrote:
>: // feebly try to prevent concurrent reentry problems
>: IndexWriter w = writer;
>: w = null;
>: try {
>: w.close();
>
>Just to clarify for the folks at home, I'm pretty sure you ment...
>
> IndexWriter w = writer;
> writer = null;
> try {
> w.close();
>
>...correct?
>
>-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by Chris Hostetter <ho...@fucit.org>.
: // feebly try to prevent concurrent reentry problems
: IndexWriter w = writer;
: w = null;
: try {
: w.close();
Just to clarify for the folks at home, I'm pretty sure you ment...
IndexWriter w = writer;
writer = null;
try {
w.close();
...correct?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by Olivier Jaquemet <ol...@jalios.com>.
J.J. Larrea wrote:
>At 6:39 PM +0200 10/4/05, Olivier Jaquemet wrote:
>
>
>>In every case I think I will use this to prevent any problem but why nobody uses finalize methods? is it somehow bad to try to close things correctly that way?
>>
>>
>
>Because they are not run under "brutal termination" conditions. For that you need to add a shutdown hook to the Java runtime. A shutdown hook will run if the runtime process receives a signal eg. SIGHUP which on Unix occurs if the controlling terminal "hangs up", etc. It won't get run on a SIGKILL, but nothing will. It also runs if you get an OutOfMemoryException or similar problem in the runtime, though there is no guarantee the hook code e.g. close() won't also fail in that case.
>
>
[...]
>I'm not sure whether this relates to the IndexSearcher problem you're trying to solve, but it is certainly useful when indexing.
>
>
It does clarify a lot of things regarding proper use (close) of
IndexWriter and IndexSearcher. thanks.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by "J.J. Larrea" <jj...@panix.com>.
At 6:39 PM +0200 10/4/05, Olivier Jaquemet wrote:
>In every case I think I will use this to prevent any problem but why nobody uses finalize methods? is it somehow bad to try to close things correctly that way?
Because they are not run under "brutal termination" conditions. For that you need to add a shutdown hook to the Java runtime. A shutdown hook will run if the runtime process receives a signal eg. SIGHUP which on Unix occurs if the controlling terminal "hangs up", etc. It won't get run on a SIGKILL, but nothing will. It also runs if you get an OutOfMemoryException or similar problem in the runtime, though there is no guarantee the hook code e.g. close() won't also fail in that case.
For example, my indexing code looks something like this:
final IndexWriter writer = ...
Thread shutdown = new Thread() {
public void run() {
if(writer != null) {
// feebly try to prevent concurrent reentry problems
IndexWriter w = writer;
w = null;
try {
w.close();
} catch(Exception ex) {
logger.error("Closing", ex);
}
}
}
};
Runtime.getRuntime().addShutdownHook(shutdown);
// do indexing here
doit(writer, ...);
// no abnormal termination
// just exit and the shutdown hook will run
In a multi-threaded context, one might need to add synchronization to prevent concurrent entry from causing problems.
I'm not sure whether this relates to the IndexSearcher problem you're trying to solve, but it is certainly useful when indexing.
- J.J.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by Olivier Jaquemet <ol...@jalios.com>.
Volodymyr Bychkoviak wrote:
> I'm using following code during program startup
[...]
> but this method should be used if you are running only one instance of
> such program (because one program can unlock index locked by another
> program for indexing for example)
>
Okay this perfect for our application as only one instance can be
launched at the same time. thank you very much.
In every case I think I will use this to prevent any problem but why
nobody uses finalize methods? is it somehow bad to try to close things
correctly that way?
Olivier
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by Volodymyr Bychkoviak <vb...@i-hypergrid.com>.
Olivier Jaquemet wrote:
> Thank you very much Volodymyr!
> Exactly the perfect answer I needed.
> One last question: how do you manage brutal interruption of you
> program? Cause as far as I am concerned, if I don't use a finalize
> method in which I close opened searchers, if for some reason the jvm
> terminates, on next run of my program, lucene throws TimeOut exception
> when trying to open lock file, as it was not close properly.
>
>
I'm using following code during program startup
try {
if ( !IndexReader.indexExists(indexDirectory)) {
// working index doesn't exist so try to create a dummy index.
IndexWriter iw = new IndexWriter(indexDirectory, new
StandardAnalyzer(), true);
iw.close();
} else {
IndexReader.unlock(FSDirectory.getDirectory(indexDirectory, false));
}
} catch (IOException e) {
// exception
}
but this method should be used if you are running only one instance of
such program (because one program can unlock index locked by another
program for indexing for example)
--
regards,
Volodymyr Bychkoviak
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by Olivier Jaquemet <ol...@jalios.com>.
Thank you very much Volodymyr!
Exactly the perfect answer I needed.
One last question: how do you manage brutal interruption of you program?
Cause as far as I am concerned, if I don't use a finalize method in
which I close opened searchers, if for some reason the jvm terminates,
on next run of my program, lucene throws TimeOut exception when trying
to open lock file, as it was not close properly.
Oh, by the way, because I'm sure every lucene developper once had the
same kind of need, maybe it should figure in the distribution of Lucene...?
Volodymyr Bychkoviak wrote:
> Hi Olivier!
>
> In my code i'm using following IndexSearcher extention:
>
> public class IndexSearcherWrapper extends IndexSearcher {
>
> private int referenceCount;
>
> private final IndexReader indexReader;
>
> public IndexSearcherWrapper(IndexReader indexReader) {
> super(indexReader);
> this.indexReader = indexReader;
> this.referenceCount = 1;
> }
>
> public IndexSearcher getReference() {
> referenceCount++;
> return this;
> }
>
> public void close() throws IOException {
> referenceCount--;
> if (referenceCount <= 0) {
> super.close();
> indexReader.close();
> }
> }
>
> };
>
> in the code obtaining IndexSearcher I do call getReference() method
> and use indexSearcher as usual (search & close);
> also referenceCount initially is set to 1 to prevent underlying
> indexReader being closed after first call of close() method.
>
> when opening new IndexSearcher (after index change) I do call close()
> method one more time ensuring this IndexSearcher will be closed as
> soon as last thread closes this IndexSearcher.
>
> and about second question:
> while changing index all open IndexReaders(and IndexSearchers) don't
> "see" changes until reopening.
>
>
> Olivier Jaquemet wrote:
>
>> Hi all,
>>
>> As I read it on LIA, and as it has already been said on the mailing
>> list multiple times, you only need one IndexSearcher for all your
>> thread, and when your index change, you just need to create a new one
>> to reflect changes.
>>
>> Otis said in this post you could replaced old searcher and let it
>> being collected by the GC.
>> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200502.mbox/%3c20050218222949.15826.qmail@web31111.mail.mud.yahoo.com%3e
>>
>>
>> Note that my question is maybe more java related than anything else,
>> but, anyway...
>> In the source code of IndexSearcher (and all searchers), I could not
>> find any finalize method that closes the searcher, how can you be
>> sure you old searcher is going to be closed? am I missing something?
>> Is there still a step I need to do in order to close the searcher
>> properly ? Do I need to create a facade to the searcher, keep my own
>> reference of uses and call a close in a finalize method?
>>
>> Thanks in advance for your answers :)
>>
>> Olivier
>>
>> PS: Another small question while I'm there, if a long indexation of a
>> lot Documents occurs, can the old searcher still be used for some
>> time while the indexation takes place, without problems? Cause I
>> encoutered an ArrayOutOfBoundException under some scorer class,
>> unfortunately I did not kept the trace.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
--
Olivier Jaquemet <ol...@jalios.com>
Ingénieur R&D Jalios S.A.
Tel: 01.39.23.92.83
http://www.jalios.com/
http://support.jalios.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Renewing IndexSearcher on index change.
Posted by Volodymyr Bychkoviak <vb...@i-hypergrid.com>.
Hi Olivier!
In my code i'm using following IndexSearcher extention:
public class IndexSearcherWrapper extends IndexSearcher {
private int referenceCount;
private final IndexReader indexReader;
public IndexSearcherWrapper(IndexReader indexReader) {
super(indexReader);
this.indexReader = indexReader;
this.referenceCount = 1;
}
public IndexSearcher getReference() {
referenceCount++;
return this;
}
public void close() throws IOException {
referenceCount--;
if (referenceCount <= 0) {
super.close();
indexReader.close();
}
}
};
in the code obtaining IndexSearcher I do call getReference() method and
use indexSearcher as usual (search & close);
also referenceCount initially is set to 1 to prevent underlying
indexReader being closed after first call of close() method.
when opening new IndexSearcher (after index change) I do call close()
method one more time ensuring this IndexSearcher will be closed as soon
as last thread closes this IndexSearcher.
and about second question:
while changing index all open IndexReaders(and IndexSearchers) don't
"see" changes until reopening.
Olivier Jaquemet wrote:
> Hi all,
>
> As I read it on LIA, and as it has already been said on the mailing
> list multiple times, you only need one IndexSearcher for all your
> thread, and when your index change, you just need to create a new one
> to reflect changes.
>
> Otis said in this post you could replaced old searcher and let it
> being collected by the GC.
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200502.mbox/%3c20050218222949.15826.qmail@web31111.mail.mud.yahoo.com%3e
>
>
> Note that my question is maybe more java related than anything else,
> but, anyway...
> In the source code of IndexSearcher (and all searchers), I could not
> find any finalize method that closes the searcher, how can you be sure
> you old searcher is going to be closed? am I missing something? Is
> there still a step I need to do in order to close the searcher
> properly ? Do I need to create a facade to the searcher, keep my own
> reference of uses and call a close in a finalize method?
>
> Thanks in advance for your answers :)
>
> Olivier
>
> PS: Another small question while I'm there, if a long indexation of a
> lot Documents occurs, can the old searcher still be used for some time
> while the indexation takes place, without problems? Cause I encoutered
> an ArrayOutOfBoundException under some scorer class, unfortunately I
> did not kept the trace.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
regards,
Volodymyr Bychkoviak
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org