You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Eric Brown <yo...@gmail.com> on 2006/11/29 11:40:38 UTC
Cached Hits / closing IndexSearcher after add/delete w/IndexModifier
I'm using lucene as a backend for my webservices that provide add,
remove and search operations. When I add or remove documents via
IndexModifier, I believe I'm supposed to close the IndexSearcher I
use for query requests. However, I cache Hits and I believe the
javadocs indicate closing an IndexSearcher will invalidate the cached
Hits such that trying to retrieve a doc may throw an exception. So my
solution is to wrap the IndexSearcher in another class with a
finalizer that closes the IndexSearcher when all the Hits dereference
it. My question is whether my cached hits have a solid reference to
IndexSearcher such that I can rely on it being closed only after I
expire the Hits from the cache?
Also, should I keep one IndexModifier open for the life of my service/
application or should I open and close it when I get new requests to
add or remove documents? (I don't really have control over batching
unfortunately -- though I've certainly pointed it out.)
Thanks,
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier
Posted by Yonik Seeley <yo...@apache.org>.
On 11/29/06, Yonik Seeley <yo...@apache.org> wrote:
> On 11/29/06, Eric Brown <yo...@gmail.com> wrote:
> > > IndexModifier currently has very low performance with mixed adds
> > > and deletes.
> > > You can keep the same one over the lifetime of the app though.
> >
> > Assuming I call flush() after every operation (they won't be that
> > frequent), if I don't call close() when my application shuts down,
> > will I run into locking issues when I restart my application?
>
> IndexWriter has a finalizer, bug finalizers aren't guaranteed to be run AFAIK.
> There is a JVM shutdown hook you could use to clean up though.
> You could also use native locks (in the current lucene devel version).
System.runFinalizersOnExit(true) is also an option.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier
Posted by Yonik Seeley <yo...@apache.org>.
On 11/29/06, Eric Brown <yo...@gmail.com> wrote:
> > IndexModifier currently has very low performance with mixed adds
> > and deletes.
> > You can keep the same one over the lifetime of the app though.
>
> Assuming I call flush() after every operation (they won't be that
> frequent), if I don't call close() when my application shuts down,
> will I run into locking issues when I restart my application?
IndexWriter has a finalizer, bug finalizers aren't guaranteed to be run AFAIK.
There is a JVM shutdown hook you could use to clean up though.
You could also use native locks (in the current lucene devel version).
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier
Posted by Yonik Seeley <yo...@apache.org>.
On 11/29/06, Mark Miller <ma...@gmail.com> wrote:
> If you haven't seen the following option you might want to check it out. It uses reference counting to keep track of writers/searchers/etc(I think solr does as well).
Right, Solr uses reference counting... relying on GC will get out
out-of-filehandle exceptions.
http://svn.apache.org/viewvc/incubator/solr/trunk/src/java/org/apache/solr/util/RefCounted.java?view=markup
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier
Posted by Mark Miller <ma...@gmail.com>.
If you haven't seen the following option you might want to check it out. It uses reference counting to keep track of writers/searchers/etc(I think solr does as well).
Has worked great for everything I have thrown at it other than need some tweaks to its multisearcher support. Also is a good
base for adding the ability to 'warm' searchers.
The oversimplified gist is: When a writer is closed, new searchers are opened for the index.
BEGIN QUOTE:
You can download the source here:
http://www.blizzy.de/lucene/lucene-indexaccess-0.1.0.zip
Using LuceneIndexAccessor is incredibly simple:
Directory directory = ...
Analyzer analyzer = ...
// somewhere near program start
IIndexAccessProvider accessProvider =
new IndexAccessProvider(directory, analyzer);
ILuceneIndexAccessor accessor = new LuceneIndexAccessor(accessProvider);
accessor.open();
IndexWriter writer = null;
try {
writer = accessor.getWriter();
// use writer...
} catch (IOException e) {
// ...
} finally {
accessor.release(write);
}
// somewhere near program exit
accessor.close();
Eric Brown wrote:
> Hi Yonik,
>
> On Nov 29, 2006, at 11:16 AM, Yonik Seeley wrote:
>> On 11/29/06, Eric Brown <yo...@gmail.com> wrote:
> [snip]
>>> Also, should I keep one IndexModifier open for the life of my service/
>>> application or should I open and close it when I get new requests to
>>> add or remove documents? (I don't really have control over batching
>>> unfortunately -- though I've certainly pointed it out.)
>>
>> IndexModifier currently has very low performance with mixed adds and
>> deletes.
>> You can keep the same one over the lifetime of the app though.
>
> Assuming I call flush() after every operation (they won't be that
> frequent), if I don't call close() when my application shuts down,
> will I run into locking issues when I restart my application?
>
>> If you haven't seen it, another alternative that might fit your needs
>> is Solr.
>
> I just took a look. It looks very applicable and I'll look at it
> seriously for subsequent roll-outs of our site. For the short-term I'm
> too tight for time to change direction.
>
> Thanks!
> Eric
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier
Posted by Eric Brown <yo...@gmail.com>.
Hi Yonik,
On Nov 29, 2006, at 11:16 AM, Yonik Seeley wrote:
> On 11/29/06, Eric Brown <yo...@gmail.com> wrote:
[snip]
>> Also, should I keep one IndexModifier open for the life of my
>> service/
>> application or should I open and close it when I get new requests to
>> add or remove documents? (I don't really have control over batching
>> unfortunately -- though I've certainly pointed it out.)
>
> IndexModifier currently has very low performance with mixed adds
> and deletes.
> You can keep the same one over the lifetime of the app though.
Assuming I call flush() after every operation (they won't be that
frequent), if I don't call close() when my application shuts down,
will I run into locking issues when I restart my application?
> If you haven't seen it, another alternative that might fit your
> needs is Solr.
I just took a look. It looks very applicable and I'll look at it
seriously for subsequent roll-outs of our site. For the short-term
I'm too tight for time to change direction.
Thanks!
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier
Posted by Yonik Seeley <yo...@apache.org>.
On 11/29/06, Eric Brown <yo...@gmail.com> wrote:
> I'm using lucene as a backend for my webservices that provide add,
> remove and search operations. When I add or remove documents via
> IndexModifier, I believe I'm supposed to close the IndexSearcher I
> use for query requests. However, I cache Hits and I believe the
> javadocs indicate closing an IndexSearcher will invalidate the cached
> Hits such that trying to retrieve a doc may throw an exception. So my
> solution is to wrap the IndexSearcher in another class with a
> finalizer that closes the IndexSearcher when all the Hits dereference
> it. My question is whether my cached hits have a solid reference to
> IndexSearcher such that I can rely on it being closed only after I
> expire the Hits from the cache?
Yes, they do.
> Also, should I keep one IndexModifier open for the life of my service/
> application or should I open and close it when I get new requests to
> add or remove documents? (I don't really have control over batching
> unfortunately -- though I've certainly pointed it out.)
IndexModifier currently has very low performance with mixed adds and deletes.
You can keep the same one over the lifetime of the app though.
If you haven't seen it, another alternative that might fit your needs is Solr.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org