You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by andynuss <an...@yahoo.com> on 2010/06/21 17:02:06 UTC
search hits not returned until I stop and restart application
Hi,
I have an IndexWriter singleton in my program, and an IndexSearcher
singleton based on a readonly IndexReader singleton. When I use the
IndexWriter to index a large document to lucene, and then, while the program
is still running, use my previously created IndexSearcher to find hits in
that book, they are not found. But if I stop and restart the application,
then they are found.
Andy
--
View this message in context: http://lucene.472066.n3.nabble.com/search-hits-not-returned-until-I-stop-and-restart-application-tp911711p911711.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: search hits not returned until I stop and restart application
Posted by Steven A Rowe <sa...@syr.edu>.
Andy,
I think batching commits either by time or number of documents is common.
Do you know about NRT (Near Realtime Search)?: <http://wiki.apache.org/lucene-java/NearRealtimeSearch>. Using IndexWriter.getReader(), you can avoid commits altogether, as well as reducing update->search latency. See IndexWriter.getReader() javadocs for more details: <http://lucene.apache.org/java/3_0_2/api/all/org/apache/lucene/index/IndexWriter.html#getReader%28%29>.
Depending on requirements, these two strategies can be combined.
Steve
> -----Original Message-----
> From: andynuss [mailto:andrew_nuss@yahoo.com]
> Sent: Monday, June 21, 2010 2:44 PM
> To: java-user@lucene.apache.org
> Subject: RE: search hits not returned until I stop and restart application
>
>
> "Maybe you aren't using the IndexReader instance returned by reopen(), but
> instead are continuing to use the instance on which you called reopen()?
> It's tough to figure this kind of thing out without looking at the code."
>
> That was it, I was not using the newly (re)opened index. By the way, one
> last question. It doesn't matter for this because I'm indexing one huge
> document at a time, and then committing. But later, I will also be
> indexing very small documents frequently. In that case, it would seem
> that if I index a very small document, I don't want to be thrashing with a
> commit after each one, and then a reopen of the reader and reconstruction
> of my searcher. Do others manage this type of thing with a thread that
> fires at intervals to commit if dirty?
RE: search hits not returned until I stop and restart application
Posted by andynuss <an...@yahoo.com>.
"Maybe you aren't using the IndexReader instance returned by reopen(), but
instead are continuing to use the instance on which you called reopen()?
It's tough to figure this kind of thing out without looking at the code."
That was it, I was not using the newly (re)opened index. By the way, one
last question. It doesn't matter for this because I'm indexing one huge
document at a time, and then committing. But later, I will also be indexing
very small documents frequently. In that case, it would seem that if I
index a very small document, I don't want to be thrashing with a commit
after each one, and then a reopen of the reader and reconstruction of my
searcher. Do others manage this type of thing with a thread that fires at
intervals to commit if dirty?
--
View this message in context: http://lucene.472066.n3.nabble.com/search-hits-not-returned-until-I-stop-and-restart-application-tp911711p912345.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: search hits not returned until I stop and restart application
Posted by Steven A Rowe <sa...@syr.edu>.
Andy, it sounds like you're doing the right thing.
Maybe you aren't using the IndexReader instance returned by reopen(), but instead are continuing to use the instance on which you called reopen()? It's tough to figure this kind of thing out without looking at the code.
For example, what do you mean by "singleton"? (You mentioned this in reference to both IndexWriter and IndexReader.) Is it possible that some part of your code is maintaining a reference to the original IndexReader instance and using it, rather than using the newly opened instance?
Steve
> -----Original Message-----
> From: andynuss [mailto:andrew_nuss@yahoo.com]
> Sent: Monday, June 21, 2010 1:29 PM
> To: java-user@lucene.apache.org
> Subject: RE: search hits not returned until I stop and restart application
>
>
> "So you gotta call commit() or close(). Once you've done that, you can
> reduce the (expensive) cost of opening a new IndexReader by calling
> reopen(): "
>
> Steve,
>
> I tried this, and I must have done something wrong.
>
> After my document set was ingested, I called a function which (1) called
> the IndexWriter singleton commit() function, (2) then called the
> IndexReader singleton reopen() function (no arguments). (My IndexReader
> is read only.) Still didn't find hits in that book. Then I tried (3)
> creating a new IndexSearcher on top of this IndexReader and that also
> didn't help.
>
> Wonder what I could be doing wrong.
>
> Andy
> --
> View this message in context: http://lucene.472066.n3.nabble.com/search-
> hits-not-returned-until-I-stop-and-restart-application-
> tp911711p912096.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
RE: search hits not returned until I stop and restart application
Posted by andynuss <an...@yahoo.com>.
"So you gotta call commit() or close(). Once you've done that, you can
reduce the (expensive) cost of opening a new IndexReader by calling
reopen(): "
Steve,
I tried this, and I must have done something wrong.
After my document set was ingested, I called a function which (1) called the
IndexWriter singleton commit() function, (2) then called the IndexReader
singleton reopen() function (no arguments). (My IndexReader is read only.)
Still didn't find hits in that book. Then I tried (3) creating a new
IndexSearcher on top of this IndexReader and that also didn't help.
Wonder what I could be doing wrong.
Andy
--
View this message in context: http://lucene.472066.n3.nabble.com/search-hits-not-returned-until-I-stop-and-restart-application-tp911711p912096.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: search hits not returned until I stop and restart application
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Andy,
From the API docs for IndexWriter <http://lucene.apache.org/java/3_0_2/api/all/org/apache/lucene/index/IndexWriter.html>:
[D]ocuments are added with addDocument and removed
with deleteDocuments(Term) or deleteDocuments(Query).
A document can be updated with updateDocument (which
just deletes and then adds the entire document).
When finished adding, deleting and updating documents,
close should be called.
These changes .... are not visible to IndexReader
until either commit() or close() is called.
So you gotta call commit() or close(). Once you've done that, you can reduce the (expensive) cost of opening a new IndexReader by calling reopen():
<http://lucene.apache.org/java/3_0_2/api/all/org/apache/lucene/index/IndexReader.html#reopen%28%29>
Steve
> -----Original Message-----
> From: andynuss [mailto:andrew_nuss@yahoo.com]
> Sent: Monday, June 21, 2010 11:02 AM
> To: java-user@lucene.apache.org
> Subject: search hits not returned until I stop and restart application
>
>
> Hi,
>
> I have an IndexWriter singleton in my program, and an IndexSearcher
> singleton based on a readonly IndexReader singleton. When I use the
> IndexWriter to index a large document to lucene, and then, while the
> program is still running, use my previously created IndexSearcher to find
> hits in that book, they are not found. But if I stop and restart the
> application, then they are found.
>
> Andy
> --
> View this message in context: http://lucene.472066.n3.nabble.com/search-
> hits-not-returned-until-I-stop-and-restart-application-
> tp911711p911711.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org