You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Cyril Barlow <in...@fantasyfooty.org> on 2005/10/05 15:15:22 UTC
IndexSearcher in servlet containers
Has anyone got experience of using the IndexSearcher in a servlet? I'm having caching problems when there's a lot of different concurrent users with the current setup. Currently the setup is to create a IndexSearcher per servlet doPost call and close it after. But with concurrent requests - especially ones which are performing fairly complex ranked boolean queries that take a few seconds are causing caching problems - sometimes a completely different set of results will be returned or nothing at all. Has anyone come across errors like these before or know best practices for searching in concurrent servlets?
Thanks
Re: IndexSearcher in servlet containers
Posted by Cyril Barlow <in...@fantasyfooty.org>.
>
> There really is no need to close an IndexSearcher until you need to
> instantiate another one, and even then you can let the old instance
> go without closing and all will still be well. If you construct
> IndexSearcher with a String directory name, there is no need to close
> anything other than (optionally) the IndexSearcher.
>
> Erik
>
Cheers Erik. Looking at IndexSearcher over RMI but that seems to be throwing
up serialization problems.
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Oct 5, 2005, at 9:32 PM, Cyril Barlow wrote:
>> Creating an IndexSearcher for every request goes against how to use
>> Lucene best. A _single_ IndexSearcher for all searches is optimum.
>>
>> You really ought to look into using a single instance.
>>
>> Erik
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> Actually scrap that idea - it's back to caching problems again,
> back to find
> a work around for the single instance indexsearcher- there probably
> is a
> close method somewhere I've not commented out in the servlet..
> When you create a IndexSearcher with a string directory rather than an
> IndexReader - do you still have an IndexReader underneath - and if
> so do you
> have to close that as well as an IndexSearcher?
There really is no need to close an IndexSearcher until you need to
instantiate another one, and even then you can let the old instance
go without closing and all will still be well. If you construct
IndexSearcher with a String directory name, there is no need to close
anything other than (optionally) the IndexSearcher.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Cyril Barlow <in...@fantasyfooty.org>.
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: <ja...@lucene.apache.org>
Sent: Thursday, October 06, 2005 2:10 AM
Subject: Re: IndexSearcher in servlet containers
>
> On Oct 5, 2005, at 9:03 PM, Cyril Barlow wrote:
>
> >
> >
> >>> I'm getting :
> >>>
> >>> java.io.IOException: The handle is invalid at
> >>> java.io.RandomAccessFile.seek(Native Method) at
> >>>
> >>
> >> Did you perhaps close the IndexSearcher somewhere along the way?
> >>
> >> Erik
> >>
> >
> >
> > No, but I'm now creating new IndexSearchers every servlet doPost
> > method and
> > not closing them. It works without the caching probems for some
> > reason and
> > it's much quicker?? I'm hoping the garbage collector sorts the
> > cleaning up
> > though..
>
> Creating an IndexSearcher for every request goes against how to use
> Lucene best. A _single_ IndexSearcher for all searches is optimum.
>
> You really ought to look into using a single instance.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Actually scrap that idea - it's back to caching problems again, back to find
a work around for the single instance indexsearcher- there probably is a
close method somewhere I've not commented out in the servlet..
When you create a IndexSearcher with a string directory rather than an
IndexReader - do you still have an IndexReader underneath - and if so do you
have to close that as well as an IndexSearcher?
___________________________________________________________
Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Oct 5, 2005, at 9:03 PM, Cyril Barlow wrote:
>
>
>>> I'm getting :
>>>
>>> java.io.IOException: The handle is invalid at
>>> java.io.RandomAccessFile.seek(Native Method) at
>>>
>>
>> Did you perhaps close the IndexSearcher somewhere along the way?
>>
>> Erik
>>
>
>
> No, but I'm now creating new IndexSearchers every servlet doPost
> method and
> not closing them. It works without the caching probems for some
> reason and
> it's much quicker?? I'm hoping the garbage collector sorts the
> cleaning up
> though..
Creating an IndexSearcher for every request goes against how to use
Lucene best. A _single_ IndexSearcher for all searches is optimum.
You really ought to look into using a single instance.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Cyril Barlow <in...@fantasyfooty.org>.
> > I'm getting :
> >
> > java.io.IOException: The handle is invalid at
> > java.io.RandomAccessFile.seek(Native Method) at
>
> Did you perhaps close the IndexSearcher somewhere along the way?
>
> Erik
No, but I'm now creating new IndexSearchers every servlet doPost method and
not closing them. It works without the caching probems for some reason and
it's much quicker?? I'm hoping the garbage collector sorts the cleaning up
though..
___________________________________________________________
Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Oct 5, 2005, at 8:15 PM, Cyril Barlow wrote:
> Using one IndexSearcher across the whole application doesn't seem
> to work.
It works for lucenebook.com :)
> I'm getting :
>
> java.io.IOException: The handle is invalid at
> java.io.RandomAccessFile.seek(Native Method) at
Did you perhaps close the IndexSearcher somewhere along the way?
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Cyril Barlow <in...@fantasyfooty.org>.
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: <ja...@lucene.apache.org>
Sent: Wednesday, October 05, 2005 8:20 PM
Subject: Re: IndexSearcher in servlet containers
>
> On Oct 5, 2005, at 2:09 PM, Cyril Barlow wrote:
> >> I'm really confused on the dilemma here.
> >>
> >> You can create a startup hook using one of the Servlet specification
> >> listeners, create an IndexSearcher there, stuff it into application
> >> scope (context.setAttribute()).
> >>
> >> There is no digging into Jetty's guts needed, just plain and simple
> >> servlet API will suffice.
> >>
> >> Of course you'll also want a facility to re-instantiate the
> >> IndexSearcher if the index is changing, but that is just building
> >> upon the pieces I just described.
> >>
> >
> > OK, I add a IndexSearcher object using the context.setAttribute(IS)
> > but how
> > do I refer back to it from the servlet code?
>
> Uh... IndexSearcher searcher = (IndexSearcher) context.getAttribute
> ("<some key>")
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Using one IndexSearcher across the whole application doesn't seem to work.
I'm getting :
java.io.IOException: The handle is invalid at
java.io.RandomAccessFile.seek(Native Method) at
org.apache.lucene.store.FSInputStream.readInternal(FSDirectory.java:415) at
org.apache.lucene.store.InputStream.readBytes(InputStream.java:61) at
org.apache.lucene.index.CompoundFileReader$CSInputStream.readInternal(Compou
ndFileReader.java:220) at
org.apache.lucene.store.InputStream.refill(InputStream.java:158) at
org.apache.lucene.store.InputStream.readByte(InputStream.java:43) at
org.apache.lucene.store.InputStream.readVInt(InputStream.java:83) at
org.apache.lucene.index.SegmentTermEnum.readTerm(SegmentTermEnum.java:142)
at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:115) at
org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:143)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:132) at
org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:253) at
org.apache.lucene.index.MultiReader.docFreq(MultiReader.java:192) at
org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:69) at
org.apache.lucene.search.Similarity.idf(Similarity.java:255) at
org.apache.lucene.search.TermQuery$TermWeight.sumOfSquaredWeights(TermQuery.
java:47) at
org.apache.lucene.search.BooleanQuery$BooleanWeight.sumOfSquaredWeights(Bool
eanQuery.java:110) at org.apache.lucene.search.Query.weight(Query.java:86)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:117) at
org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64) at
org.apache.lucene.search.Hits.(Hits.java:51) at
org.apache.lucene.search.Searcher.search(Searcher.java:41) at ..........
This is after a few searches. Its ok before that. Any ideas?
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Oct 5, 2005, at 2:09 PM, Cyril Barlow wrote:
>> I'm really confused on the dilemma here.
>>
>> You can create a startup hook using one of the Servlet specification
>> listeners, create an IndexSearcher there, stuff it into application
>> scope (context.setAttribute()).
>>
>> There is no digging into Jetty's guts needed, just plain and simple
>> servlet API will suffice.
>>
>> Of course you'll also want a facility to re-instantiate the
>> IndexSearcher if the index is changing, but that is just building
>> upon the pieces I just described.
>>
>
> OK, I add a IndexSearcher object using the context.setAttribute(IS)
> but how
> do I refer back to it from the servlet code?
Uh... IndexSearcher searcher = (IndexSearcher) context.getAttribute
("<some key>")
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Cyril Barlow <in...@fantasyfooty.org>.
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: <ja...@lucene.apache.org>
Sent: Wednesday, October 05, 2005 6:34 PM
Subject: Re: IndexSearcher in servlet containers
>
> On Oct 5, 2005, at 9:54 AM, Cyril Barlow wrote:
> > Thanks all for your feedback. I'm going to look at building a
> > dedicated
> > server that keeps one IndexSearcher open and each servlet would use
> > that.
> > Either that or look deeper into the Jetty framework to see if I can
> > simply
> > invoke a global IndexSearcher when the server starts and use that.
> > Not quite
> > sure how. Luc, DelayCloseIndexSearcher looks like something that
> > might be of
> > help - I'll have a look at that as well.
>
> I'm really confused on the dilemma here.
>
> You can create a startup hook using one of the Servlet specification
> listeners, create an IndexSearcher there, stuff it into application
> scope (context.setAttribute()).
>
> There is no digging into Jetty's guts needed, just plain and simple
> servlet API will suffice.
>
> Of course you'll also want a facility to re-instantiate the
> IndexSearcher if the index is changing, but that is just building
> upon the pieces I just described.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>You can create a startup hook using one of the Servlet specification
> listeners, create an IndexSearcher there, stuff it into application
> scope (context.setAttribute()).
OK, I add a IndexSearcher object using the context.setAttribute(IS) but how
do I refer back to it from the servlet code?
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Oct 5, 2005, at 9:54 AM, Cyril Barlow wrote:
> Thanks all for your feedback. I'm going to look at building a
> dedicated
> server that keeps one IndexSearcher open and each servlet would use
> that.
> Either that or look deeper into the Jetty framework to see if I can
> simply
> invoke a global IndexSearcher when the server starts and use that.
> Not quite
> sure how. Luc, DelayCloseIndexSearcher looks like something that
> might be of
> help - I'll have a look at that as well.
I'm really confused on the dilemma here.
You can create a startup hook using one of the Servlet specification
listeners, create an IndexSearcher there, stuff it into application
scope (context.setAttribute()).
There is no digging into Jetty's guts needed, just plain and simple
servlet API will suffice.
Of course you'll also want a facility to re-instantiate the
IndexSearcher if the index is changing, but that is just building
upon the pieces I just described.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Cyril Barlow <in...@fantasyfooty.org>.
Thanks all for your feedback. I'm going to look at building a dedicated
server that keeps one IndexSearcher open and each servlet would use that.
Either that or look deeper into the Jetty framework to see if I can simply
invoke a global IndexSearcher when the server starts and use that. Not quite
sure how. Luc, DelayCloseIndexSearcher looks like something that might be of
help - I'll have a look at that as well.
Cheers
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher in servlet containers
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Cache IndexSearcher and only use *one* instance for all requests.
Application scope works well for this in a servlet environment.
Erik
On Oct 5, 2005, at 9:15 AM, Cyril Barlow wrote:
> Has anyone got experience of using the IndexSearcher in a servlet?
> I'm having caching problems when there's a lot of different
> concurrent users with the current setup. Currently the setup is to
> create a IndexSearcher per servlet doPost call and close it after.
> But with concurrent requests - especially ones which are performing
> fairly complex ranked boolean queries that take a few seconds are
> causing caching problems - sometimes a completely different set of
> results will be returned or nothing at all. Has anyone come across
> errors like these before or know best practices for searching in
> concurrent servlets?
>
> Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org