You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Anson Lau <al...@fulfil-net.com> on 2004/06/29 03:05:42 UTC
best ways of using IndexSearcher
Hi Guys,
What's the recommended way of using IndexSearcher? Should IndexSearcher be a
singleton or pooled? Would pooling provide a more scalable solution by
allowing you to decide how many IndexSearcher to use based on say how many
CPU u have on ur server?
Thanks,
Anson
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: best ways of using IndexSearcher
Posted by Aviran <am...@infosciences.com>.
Yes, IndexSearcher is thread safe.
Aviran
http://www.aviransplace.com
-----Original Message-----
From: Abhay Saswade [mailto:ays@i.frys.com]
Sent: Tuesday, November 16, 2004 15:16 PM
To: Lucene Users List
Subject: Re: best ways of using IndexSearcher
Hello,
Can I use single instance of IndexSearcher in multiple threads with sorting?
Thanks, Abhay
----- Original Message -----
From: "Otis Gospodnetic" <ot...@yahoo.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Monday, June 28, 2004 8:51 PM
Subject: Re: best ways of using IndexSearcher
> Anson,
>
> Use a single instance of IndexSearcher and, if you want to always
> 'see' even the latest index changes (deletes and adds since you opened
> the
> IndexSearcher) make sure to re-create the IndexSearcher when you detect
> that the index version has changed (see
>
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReade
r.html#getCurrentVersion(org.apache.lucene.store.Directory))
>
> When you get the new IndexSearcher, leave the old instance alone - let
> the GC take care of it, and don't call close() on it, in case
> something in your application is still using that instance.
>
> This stuff is not really CPU intensive. Disk I/O tends to be the
> bottleneck. If you are working with multiple indices, spread them
> over multiple disks (not just partitions, real disks), if you can.
>
> Otis
>
>
> --- Anson Lau <al...@fulfil-net.com> wrote:
> > Hi Guys,
> >
> > What's the recommended way of using IndexSearcher? Should
> > IndexSearcher be a singleton or pooled? Would pooling provide a
> > more scalable solution by
> > allowing you to decide how many IndexSearcher to use based on say how
> > many
> > CPU u have on ur server?
> >
> > Thanks,
> >
> > Anson
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best ways of using IndexSearcher
Posted by Abhay Saswade <ay...@i.frys.com>.
Hello,
Can I use single instance of IndexSearcher in multiple threads with sorting?
Thanks,
Abhay
----- Original Message -----
From: "Otis Gospodnetic" <ot...@yahoo.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Monday, June 28, 2004 8:51 PM
Subject: Re: best ways of using IndexSearcher
> Anson,
>
> Use a single instance of IndexSearcher and, if you want to always 'see'
> even the latest index changes (deletes and adds since you opened the
> IndexSearcher) make sure to re-create the IndexSearcher when you detect
> that the index version has changed (see
>
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReader.html#getCurrentVersion(org.apache.lucene.store.Directory))
>
> When you get the new IndexSearcher, leave the old instance alone - let
> the GC take care of it, and don't call close() on it, in case something
> in your application is still using that instance.
>
> This stuff is not really CPU intensive. Disk I/O tends to be the
> bottleneck. If you are working with multiple indices, spread them over
> multiple disks (not just partitions, real disks), if you can.
>
> Otis
>
>
> --- Anson Lau <al...@fulfil-net.com> wrote:
> > Hi Guys,
> >
> > What's the recommended way of using IndexSearcher? Should
> > IndexSearcher be a
> > singleton or pooled? Would pooling provide a more scalable solution
> > by
> > allowing you to decide how many IndexSearcher to use based on say how
> > many
> > CPU u have on ur server?
> >
> > Thanks,
> >
> > Anson
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: best ways of using IndexSearcher
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Anson,
I was indeed referring to search, but I could be wrong. I have not
tested it, so my answer was more instinctive than scientific. :(
Regardless, a single IndexSearcher should do. Are you finding that
your queries are taking too long? If so, I suggest you profile things,
and find the bottleneck. One person has recently provided some
execution stack traces, which revealed 1 bottleneck, and I believe Doug
has subsequently applied a patch that improved search performance.
Otis
--- Anson Lau <al...@fulfil-net.com> wrote:
> Otis,
>
> Thanks for the advice. When you say "This stuff is not really CPU
> intensive" are you refering to the search itself or something else?
> In my
> experience the search tends to be ultimately bounded by CPU.
>
> Anson
>
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Sent: Tuesday, June 29, 2004 2:51 PM
> To: Lucene Users List
> Subject: Re: best ways of using IndexSearcher
>
> Anson,
>
> Use a single instance of IndexSearcher and, if you want to always
> 'see'
> even the latest index changes (deletes and adds since you opened the
> IndexSearcher) make sure to re-create the IndexSearcher when you
> detect
> that the index version has changed (see
>
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReade
> r.html#getCurrentVersion(org.apache.lucene.store.Directory))
>
> When you get the new IndexSearcher, leave the old instance alone -
> let
> the GC take care of it, and don't call close() on it, in case
> something
> in your application is still using that instance.
>
> This stuff is not really CPU intensive. Disk I/O tends to be the
> bottleneck. If you are working with multiple indices, spread them
> over
> multiple disks (not just partitions, real disks), if you can.
>
> Otis
>
>
> --- Anson Lau <al...@fulfil-net.com> wrote:
> > Hi Guys,
> >
> > What's the recommended way of using IndexSearcher? Should
> > IndexSearcher be a
> > singleton or pooled? Would pooling provide a more scalable
> solution
> > by
> > allowing you to decide how many IndexSearcher to use based on say
> how
> > many
> > CPU u have on ur server?
> >
> > Thanks,
> >
> > Anson
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: best ways of using IndexSearcher
Posted by Anson Lau <al...@fulfil-net.com>.
Otis,
Thanks for the advice. When you say "This stuff is not really CPU
intensive" are you refering to the search itself or something else? In my
experience the search tends to be ultimately bounded by CPU.
Anson
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Tuesday, June 29, 2004 2:51 PM
To: Lucene Users List
Subject: Re: best ways of using IndexSearcher
Anson,
Use a single instance of IndexSearcher and, if you want to always 'see'
even the latest index changes (deletes and adds since you opened the
IndexSearcher) make sure to re-create the IndexSearcher when you detect
that the index version has changed (see
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReade
r.html#getCurrentVersion(org.apache.lucene.store.Directory))
When you get the new IndexSearcher, leave the old instance alone - let
the GC take care of it, and don't call close() on it, in case something
in your application is still using that instance.
This stuff is not really CPU intensive. Disk I/O tends to be the
bottleneck. If you are working with multiple indices, spread them over
multiple disks (not just partitions, real disks), if you can.
Otis
--- Anson Lau <al...@fulfil-net.com> wrote:
> Hi Guys,
>
> What's the recommended way of using IndexSearcher? Should
> IndexSearcher be a
> singleton or pooled? Would pooling provide a more scalable solution
> by
> allowing you to decide how many IndexSearcher to use based on say how
> many
> CPU u have on ur server?
>
> Thanks,
>
> Anson
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: When will Lucene 1.4
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
I'll add a little bit to what Otis said....
On Jun 28, 2004, at 10:08 PM, Karthik N S wrote:
> 2) Will it contain lot's of featured Examples (lucene1.3-final does
> not
> have many of features based examples)
It has been my dream to really enhance the examples and initial
experience users new to Lucene get when opening up a binary Lucene
distribution. I had hoped to do really beef up the example application
(i.e. scrap it and create something much nicer, including a few useful
applications) as our work on Lucene in Action.
Time has gotten away from us and in order to pragmatically get the book
done sooner rather than later (and it's already much later than we'd
like, but at least nicely timed with Lucene 1.4), the example overhaul
has not been done.
However, all of the source code to our book will be freely available at
Manning's website (http://www.manning.com/hatcher2) once the book is
completely done. When will that be? Soon! (that's the best I can
estimate right now) The examples we've developed include:
- A robust suite of JUnit tests that demonstrate many features of
Lucene. Some tests I've created have been added back to Lucene's own
test suite where coverage did not already exist.
- A document handler framework, similar but richer than the
DocumentHandler interface that is part of the 'ant' area of the sandbox
repository. This unifies indexing of Word, PDF, HTML, and XML
documents with an extensible framework to add other handlers.
- A demo Swing application that looks like a cell phone interface which
uses the WordNet database (except indexed by each words T9 equivalent
representation) to rapidly look up words that match the numbers keyed
in.
- A Google-like image search engine web interface (based on Tapestry)
that I cobbled together this weekend to demonstrate at JavaOne in a few
hours.
- Odds and ends that demonstrate computing term vector angles to do
some rudimentary latent semantic analysis, using custom geographic
distance (based on the users location) sorting to order search results,
an example of using JUnitPerf (contributed by Mike Clark himself) to
benchmark Lucene performance and, for example, demonstrate issues that
can happen when indexing dates using DateField when you really only
care about month, day, and year, and probably a few other goodies I'm
forgetting at the moment.
I will do my best to get this code packaged into a releasable state as
soon as possible (by the end of July is the best I can promise) and
made available prior to the book itself being available. Again, the
code for the book will be freely available, of course with the hope
that it'd spark interest in purchasing a copy of the book itself.
Where it fits, I'm fine with any of that code being added to Lucene or
the sandbox too.
It is still my intention, once my work is complete with the book, to
devote my free time to fleshing out a much nicer example suite to ship
with Lucene.
> [ The major Concern for our Organization, since we plan to move our
> Code
> from lucene1.3 final to 1.4 final ]
I too suggest that you go ahead and give the latest 1.4 RC release a
try and base on that. You will more than likely find it is as stable
as 1.3 and if you run into any issues you can report them and get them
fixed in 1.4 final. The alternative is for you wait and base on 1.4
final and that one undiscovered bug that you would have found earlier
is now in the final release! :)
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: When will Lucene 1.4
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Answers inlined...
--- Karthik N S <ka...@controlnet.co.in> wrote:
>
> Hey
> Dev -Guys
>
> First Apologies for the Question [ Intentionally did not wanted to
> disrupt
> the proceesings]
>
>
> 1) When will LUCENE1.4 - final be released
We haven't discussed this yet. Probably not until the Fall.
> 2) Will it contain lot's of featured Examples (lucene1.3-final does
> not have many of features based examples)
Probably no more examples than there are now, unless somebody
contributes them. However, there will be a Lucene book in the Fall.
> 3) Will the Furture releases be synchronized with releases of J2DK
> (since every new version of J2SDK is released 6 months)
Lucene's releases are not tied to Java releases in any way.
> [ The major Concern for our Organization, since we plan to move our
> Code
> from lucene1.3 final to 1.4 final ]
I would suggest you just use 1.4RC3 for now. The API is not likely to
change between RC3 and the final release, and the code is pretty stable
and bug-free (that is - same bugs are in 1.3, nothing new that I can
recall at the moment).
Otis
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
When will Lucene 1.4
Posted by Karthik N S <ka...@controlnet.co.in>.
Hey
Dev -Guys
First Apologies for the Question [ Intentionally did not wanted to disrupt
the proceesings]
1) When will LUCENE1.4 - final be released
2) Will it contain lot's of featured Examples (lucene1.3-final does not
have many of features based examples)
3) Will the Furture releases be synchronized with releases of J2DK (since
every new version of J2SDK is released 6 months)
[ The major Concern for our Organization, since we plan to move our Code
from lucene1.3 final to 1.4 final ]
Thx in advance.....
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best ways of using IndexSearcher
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Anson,
Use a single instance of IndexSearcher and, if you want to always 'see'
even the latest index changes (deletes and adds since you opened the
IndexSearcher) make sure to re-create the IndexSearcher when you detect
that the index version has changed (see
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReader.html#getCurrentVersion(org.apache.lucene.store.Directory))
When you get the new IndexSearcher, leave the old instance alone - let
the GC take care of it, and don't call close() on it, in case something
in your application is still using that instance.
This stuff is not really CPU intensive. Disk I/O tends to be the
bottleneck. If you are working with multiple indices, spread them over
multiple disks (not just partitions, real disks), if you can.
Otis
--- Anson Lau <al...@fulfil-net.com> wrote:
> Hi Guys,
>
> What's the recommended way of using IndexSearcher? Should
> IndexSearcher be a
> singleton or pooled? Would pooling provide a more scalable solution
> by
> allowing you to decide how many IndexSearcher to use based on say how
> many
> CPU u have on ur server?
>
> Thanks,
>
> Anson
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org