You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Desidero <de...@gmail.com> on 2013/09/18 21:28:24 UTC
Query performance in Lucene 4.x
Hello,
Over the last few weeks I've been working on upgrading an application from
Lucene 3.x to Lucene 4.x in hopes of improving performance. Unfortunately,
after going through the full migration process and playing with all sorts
of tweaks I found online and in the documentation, Lucene 4 is running
significantly slower than Lucene 3 (~50%). I'm pretty much out of ideas at
this point, and was wondering if anyone else had any suggestions on how to
make it work. I'm not even looking for a big improvement over 3.x anymore;
I'd be happy to just match it and stay on a current release of Lucene.
A bit of information on the system:
- We have one index that is broken into 20 shards - this provided the best
performance in both Lucene 3.x and Lucene 4.x
- The index currently contains ~150 million documents, all of which are
fairly simple and heavily normalized so there are a lot of duplicate
tokens. Only one field (an ID) is stored - the others are not retrievable.
- We have a fixed set of relatively simple queries that are populated with
user input and executed - they are comprised of multiple BooleanQueries,
TermQueries and TermRangeQueries. Some of them are nested, but only a
single level right now.
- We're not doing anything too advanced with results aside from iterating
through the scores and getting the ID fields
- We're using MMapDirectories pointing to index files in a tmpfs
- Our test machines have 94GB of RAM and 64 logical cores
General flow:
1) Request received by socket listener
2) Up to 4 Query objects are generated and populated with normalized user
input (all of the required input for a query must be present or it won't be
executed)
3) Queries are executed in parallel using the Fork/Join framework
3.a) Subqueries to each shard are executed in parallel using the
IndexSearcher w/ExecutorService
4) Aggregation and other simple post-processing
Other relevant info:
- Indexes were recreated for the 4.x system, but the data is the same. We
tried with the normal Lucene42 codec as well as an extended one that didn't
use compression (per a suggestion on the web)
- In 3.x we used a modified version of the ParallelMultisearcher, in 4.x
we're using the IndexSearcher with an ExecutorService and combining all of
our readers in a MultiReader
- In 3.x we used a ThreadPoolExecutor instead of Fork/Join (Fork/Join
performed better in my tests)
4.x Hot Spots:
Method | Self Time (%) | Self Time (ms)| Self Time (CPU in ms)
java.util.concurrent.CountDownLatch.await() | 11.29% | 140887.219 | 0.0
<- this is just from tcp threads waiting for the real work to finish - you
can ignore it
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.<init>()
| 9.74% | 121594.03 | 121594
org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum$Frame.<init>()
| 9.59% | 119680.956 | 119680
org.apache.lucene.codecs.lucene41.ForUtil.readBlock() | 6.91% | 86208.621
| 86208
org.apache.lucene.search.DisjunctionScorer.heapAdjust() | 6.68% |
83332.525 | 83332
java.util.concurrent.ExecutorCompletionService.take() | 5.29% | 66081.499
| 6153
org.apache.lucene.search.DisjunctionSucorer.afterNext() | 4.93% |
61560.872 | 61560
org.apache.lucene.search.Tercorer.advance() | 4.53% | 56530.752 | 56530
java.nio.DirectByteBuffer.get() | 3.96% | 49470.349 | 49470
org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.<init>()
| 2.97% | 37051.644 | 37051
org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.getFrame()
| 2.77% | 34576.54 | 34576
org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo() | 2.47% |
30767.711 | 30767
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.newTertate() |
2.23% | 27782.522 | 27782
java.net.ServerSocket.accept() | 2.19% | 27380.696 | 0.0
org.apache.lucene.search.DisjunctionSucorer.advance() | 1.82% | 22775.325
| 22775
org.apache.lucene.search.HitQueue.getSentinelObject() | 1.59% | 19869.871
| 19869
org.apache.lucene.store.ByteBufferIndexInput.buildSlice() | 1.43% |
17861.148 | 17861
org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.getArc()
| 1.35% | 16813.927 | 16813
org.apache.lucene.search.DisjunctionSucorer.countMatches() | 1.25% |
15603.283 | 15603
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs()
| 1.12% | 13929.646 | 13929
java.util.concurrent.locks.ReentrantLock.lock() | 1.05% | 13145.631 | 8618
org.apache.lucene.util.PriorityQueue.downHeap() | 1.00% | 12513.406 | 12513
java.util.TreeMap.get() | 0.89% | 11070.192 | 11070
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.docs() | 0.80% |
10026.117 | 10026
org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum$Frame.decodeMetaData()
| 0.62% | 7746.05 | 7746
org.apache.lucene.codecs.BlockTreeTerReader$FieldReader.iterator() | 0.60%
| 7482.395 | 7482
org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.seekExact()
| 0.55% | 6863.069 | 6863
org.apache.lucene.store.DataInput.clone() | 0.54% | 6721.357 | 6721
java.nio.DirectByteBufferR.duplicate() | 0.48% | 5930.226 | 5930
org.apache.lucene.util.fst.ByteSequenceOutputs.read() | 0.46% | 5708.354 |
5708
org.apache.lucene.util.fst.FST.findTargetArc() | 0.45% | 5601.63 | 5601
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.readTermsBlock() |
0.45% | 5567.914 | 5567
org.apache.lucene.store.ByteBufferIndexInput.toString() | 0.39% | 4889.302
| 4889
org.apache.lucene.codecs.lucene41.Lucene41SkipReader.<init>() | 0.33% |
4147.285 | 4147
org.apache.lucene.search.TermQuery$TermWeight.scorer() | 0.32% | 4045.912 |
4045
org.apache.lucene.codecs.MultiLevelSkipListReader.<init>() | 0.31% |
3890.399 | 3890
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock()
| 0.31% | 3886.194 | 3886
If there's any other information you could use, please let me know.
Thanks,
Matt
Re: Query performance in Lucene 4.x
Posted by Desidero <de...@gmail.com>.
I extended the IndexSearcher last night and set it up so it would make one
task per IndexReader instead of one per AtomicReaderContext. Performance
was pretty bad just like before, so it looks like I'm stuck merging
everything into one big segment.
I went through the documentation for the various merge policies and tried a
few different configurations, but couldn't find one that naturally caps the
number of segments at 1. The most promising options either had undocumented
limits in their setters or they didn't behave quite like I expected. I'll
spend some more time playing with it tonight, but in the meantime I don't
suppose anyone else knows a way to accomplish what I'm trying to do without
using forceMerge(1)?
On Tue, Oct 1, 2013 at 6:10 PM, Desidero <de...@gmail.com> wrote:
> Uwe,
>
> I was using a bounded thread pool.
>
> I don't know if the problem was the task overload or something about the
> actual efficiency of searching a single segment rather than iterating over
> multiple AtomicReaderContexts, but I'd lean toward task overload. I will do
> some testing tonight to find out for sure.
>
> Matt
> Hi,
>
> use a bounded thread pool.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Desidero [mailto:desidero@gmail.com]
> > Sent: Tuesday, October 01, 2013 11:37 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Query performance in Lucene 4.x
> >
> > For anyone who was wondering, this was actually resolved in a different
> > thread today. I misread the information in the
> > IndexSearcher(IndexReader,ExecutorService) constructor documentation - I
> > was under the impression that it was submitting a thread for each index
> > shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
> submitting
> > a task for each segment within each shard (20 shards * ~10 segments =
> ~200
> > tasks) which is horrible. Since my index changes infrequently, I'm using
> > forceMerge(1) before sending out updated indexes to the slave servers.
> > Without any extra tuning (threads, # of shards, etc) I've gone from ~2900
> > requests per minute to ~10k requests per minute.
> >
> > Thanks to Adrien and Mike for the clarification and Benson for bringing
> up
> > the question that led to my answer.
> >
> > I'm still pretty new to Lucene so I have a lot of poking around to do,
> but I'm
> > going to try to implement the "virtual segment" concept that Mike
> > mentioned. It'll be really helpful for those of us who want parallelism
> within
> > queries and don't want to forceMerge.
> >
> >
> > On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com> wrote:
> >
> > > Erick,
> > >
> > > Thank you for responding.
> > >
> > > I ran tests using both compressed fields and uncompressed fields, and
> > > it was significantly slower with uncompressed fields. I looked into
> > > the lazy field loading per your suggestion, but we don't get any
> > > values from the returned Documents until the result set has been
> > appropriately reduced.
> > > Since we only store one retrievable field and we always need to get
> > > it, it doesn't save any time loading it lazily.
> > >
> > > I'll try running a test without loading any fields just to see how it
> > > affects performance and let you know how that goes.
> > >
> > > Regards,
> > > Matt
> > >
> > >
> > > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> > <er...@gmail.com>wrote:
> > >
> > >> Hmmm, since 4.1, fields have been stored compressed by default.
> > >> I suppose it's possible that this is a result of
> > >> compressing/uncompressing.
> > >>
> > >> What happens if
> > >> 1> you enable lazy field loading
> > >> 2> don't load any fields?
> > >>
> > >> FWIW,
> > >> Erick
> > >>
> > >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com>
> > wrote:
> > >> > A quick update:
> > >> >
> > >> > In order to confirm that none of the standard migration changes had
> > >> > a negative effect on performance, I ported my Lucene 4.x version
> > >> > back to Lucene 3.6.2 and kept the newer API rather than using the
> > >> > custom ParallelMultiSearcher and other deprecated methods/classes.
> > >> >
> > >> > Performance in 3.6.2 is even faster than before (~2900 requests/min
> > >> with 4.x
> > >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> > >> > should be causing the difference. It seems to be something Lucene
> > >> > is doing under
> > >> the
> > >> > covers.
> > >> >
> > >> > Again, if there's any other information if I can provide to help
> > >> determine
> > >> > what's going on, please let me know.
> > >> >
> > >> > Thanks,
> > >> > Matt
> > >> >
> > >> >
> > >> >
> > >> > -------------------------------------------------------------------
> > >> > -- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>
> > >>
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Query performance in Lucene 4.x
Posted by Desidero <de...@gmail.com>.
You are correct in that I'm using a MultiReader over multiple IndexReaders
("shards") that contain one segment each to basically do what Lucene does
with a single IndexReader and multiple segments. It's done this way for two
reasons:
1) By using multiple single-segment "shards", I can completely control the
level of parallelism for my queries. It's possible to futz with
MergePolicies to control the number of segments in an index, but most of
the controls have variable outcomes (e.g. setting someAttribute allows for
up to n segments, but possibly less). I was hoping to avoid it, but I might
end up making a custom merge policy before all of this is over...
2) Unfortunately, the process we use to clean our data for indexing is
extremely CPU-intensive. Since we split the data into multiple shards, we
can re-create a shard in case a problem arises in less time because there's
less to index. This also allows us to exploit parallelism across machines
rather than just across threads to generate multiple shards simultaneously.
I generally use 128 logical CPU cores (2 servers) when reindexing for
tests, but we would benefit from using more if we had to do it quickly in
production for whatever reason.
As far as the IndexSearcher extension is concerned, it's actually pretty
simple. I've done some testing on my own and QA is going to get their hands
on it soon, so hopefully they won't find anything that I didn't!
On Wed, Oct 2, 2013 at 9:52 PM, Vitaly Funstein <vf...@gmail.com> wrote:
> Hmm, I guess your IndexSearcher is backed by a MultiReader which operates
> on these "shards" you're referring to, which are supposed to be
> single-segment indexes? If so, this topology sounds fairly equivalent, at
> least in concept but maybe similar in performance as well, to the regular
> case when you search a single index shard but allow multiple segments in
> it, and let Lucene manage segment creation/merging, instead of effectively
> disabling it.
>
> But this is all a speculation on my part, absent a concrete side by side
> comparison. Also, from your description, what you are doing to
> extend/replace internal plumbing in IndexSearcher sounds awfully complex
> and hard to envision... do make sure to run functional tests when it's all
> in place. :)
>
>
> On Wed, Oct 2, 2013 at 5:37 PM, Desidero <de...@gmail.com> wrote:
>
> > Vitaly,
> >
> > Thanks for your comments.
> >
> > Unfortunately, thread pool task overload is not the problem. When I
> > extended the IndexSearcher class last night, I had it create one task per
> > shard (20 tasks) instead of the default which turned out to be somewhere
> > around 320 (I didn't realize it created quite so many segments when I
> > recreated my index yesterday!) and performance was still awful as long as
> > there were multiple segments per shard.
> >
> > I understand that the fork/join framework doesn't work well with the
> > standard IndexSearcher's parallel search implementation, but I extended
> the
> > class and my implementation avoids the pitfalls as long as there are very
> > few (preferably one) segments per shard. The blocking in the original
> > SegmentWorkerCallable is still there, but I set it up so that if one
> worker
> > already has the lock, any others that try to lock it will just throw
> their
> > results in a separate queue which is polled by the thread that managed to
> > get the lock. After all the child tasks are complete, the queue is
> checked
> > one more time to be safe and then processing continues. There is
> basically
> > no waiting in that chunk of code (per the profiler). After doing some
> tests
> > with the ForkJoinPool and custom code as well as a standard
> > ThreadPoolExecutor, I found that the Fork/Join implementation was faster.
> >
> > The system I'm working on is much more sensitive to response time
> (latency)
> > than overall throughput, so we've had to focus on parallelism within
> > queries rather than across many queries. Batch processing performance is
> > still important, but we cannot sacrifice quick "online" responses. It
> would
> > be much easier to avoid this whole mess, but we cannot meet our
> performance
> > requirements without it.
> >
> > Regards,
> >
> > Matt
> >
> >
> > On Oct 2, 2013 3:36 PM, "Vitaly Funstein" <vf...@gmail.com> wrote:
> >
> > > Matt,
> > >
> > > I think you are mostly on track with suspecting thread pool task
> overload
> > > as the possible culprit here. First, the old school (prior to Java 7)
> > > ThreadPoolExecutor only accepts a BlockingQueue to use internally for
> > > worker tasks, instead of a concurrent variant (not sure why). So this
> > > internal work queue will become a significant point of contention when
> > > using the pool in a pattern similar to your use case, i.e. submitting
> > lots
> > > of tasks to the pool as fast as possible.
> > >
> > > Second, I am not too familiar with the internals of the fork/join pool
> > > implementation in 1.7 (if that's what you're using), but from reading
> the
> > > daunting javadoc for ForkJoinTask my rough guess is it's not terribly
> > well
> > > suited for use in IndexSearcher. In particular, one of the possible
> > > "non-compliant" behaviors is that a mutex lock is taken for each call()
> > > invocation that operates on an individual leaf slice. This is evident
> > from
> > > code inspection, and based on this, I am not sure what benefit, if any,
> > > multi-threaded search over a multi-segment index would provide in
> > general -
> > > regardless of the choice of thread pool implementation.
> > >
> > > I think a better strategy, as mentioned in another thread, is to
> optimize
> > > your system for multiple concurrent queries, rather than focusing on
> > > forcing each query to run across multiple threads/cores. With that
> > > approach, you could for instance, set up a non-blocking queue like
> > > ConcurrentLinkedQueue that would contain individual query tasks, then
> > use a
> > > fixed pool of worker threads to consume the queue in a loop and run
> them.
> > > In this scenario, you shouldn't need to pass an ExecutorService
> instance
> > to
> > > IndexSearcher.
> > >
> > > A strategy like that should provide for better query throughput,
> > regardless
> > > of whether each shard consists of a single segment or not - provided
> each
> > > of the queries is tied to a particular shard and can't search any
> others.
> > >
> > >
> > >
> > > On Tue, Oct 1, 2013 at 4:10 PM, Desidero <de...@gmail.com> wrote:
> > >
> > > > Uwe,
> > > >
> > > > I was using a bounded thread pool.
> > > >
> > > > I don't know if the problem was the task overload or something about
> > the
> > > > actual efficiency of searching a single segment rather than iterating
> > > over
> > > > multiple AtomicReaderContexts, but I'd lean toward task overload. I
> > will
> > > do
> > > > some testing tonight to find out for sure.
> > > >
> > > > Matt
> > > > Hi,
> > > >
> > > > use a bounded thread pool.
> > > >
> > > > Uwe
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > http://www.thetaphi.de
> > > > eMail: uwe@thetaphi.de
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Desidero [mailto:desidero@gmail.com]
> > > > > Sent: Tuesday, October 01, 2013 11:37 PM
> > > > > To: java-user@lucene.apache.org
> > > > > Subject: Re: Query performance in Lucene 4.x
> > > > >
> > > > > For anyone who was wondering, this was actually resolved in a
> > different
> > > > > thread today. I misread the information in the
> > > > > IndexSearcher(IndexReader,ExecutorService) constructor
> documentation
> > -
> > > I
> > > > > was under the impression that it was submitting a thread for each
> > index
> > > > > shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
> > > > submitting
> > > > > a task for each segment within each shard (20 shards * ~10
> segments =
> > > > ~200
> > > > > tasks) which is horrible. Since my index changes infrequently, I'm
> > > using
> > > > > forceMerge(1) before sending out updated indexes to the slave
> > servers.
> > > > > Without any extra tuning (threads, # of shards, etc) I've gone from
> > > ~2900
> > > > > requests per minute to ~10k requests per minute.
> > > > >
> > > > > Thanks to Adrien and Mike for the clarification and Benson for
> > bringing
> > > > up
> > > > > the question that led to my answer.
> > > > >
> > > > > I'm still pretty new to Lucene so I have a lot of poking around to
> > do,
> > > > but I'm
> > > > > going to try to implement the "virtual segment" concept that Mike
> > > > > mentioned. It'll be really helpful for those of us who want
> > parallelism
> > > > within
> > > > > queries and don't want to forceMerge.
> > > > >
> > > > >
> > > > > On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com>
> > wrote:
> > > > >
> > > > > > Erick,
> > > > > >
> > > > > > Thank you for responding.
> > > > > >
> > > > > > I ran tests using both compressed fields and uncompressed fields,
> > and
> > > > > > it was significantly slower with uncompressed fields. I looked
> into
> > > > > > the lazy field loading per your suggestion, but we don't get any
> > > > > > values from the returned Documents until the result set has been
> > > > > appropriately reduced.
> > > > > > Since we only store one retrievable field and we always need to
> get
> > > > > > it, it doesn't save any time loading it lazily.
> > > > > >
> > > > > > I'll try running a test without loading any fields just to see
> how
> > it
> > > > > > affects performance and let you know how that goes.
> > > > > >
> > > > > > Regards,
> > > > > > Matt
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> > > > > <er...@gmail.com>wrote:
> > > > > >
> > > > > >> Hmmm, since 4.1, fields have been stored compressed by default.
> > > > > >> I suppose it's possible that this is a result of
> > > > > >> compressing/uncompressing.
> > > > > >>
> > > > > >> What happens if
> > > > > >> 1> you enable lazy field loading
> > > > > >> 2> don't load any fields?
> > > > > >>
> > > > > >> FWIW,
> > > > > >> Erick
> > > > > >>
> > > > > >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com>
> > > > > wrote:
> > > > > >> > A quick update:
> > > > > >> >
> > > > > >> > In order to confirm that none of the standard migration
> changes
> > > had
> > > > > >> > a negative effect on performance, I ported my Lucene 4.x
> version
> > > > > >> > back to Lucene 3.6.2 and kept the newer API rather than using
> > the
> > > > > >> > custom ParallelMultiSearcher and other deprecated
> > methods/classes.
> > > > > >> >
> > > > > >> > Performance in 3.6.2 is even faster than before (~2900
> > > requests/min
> > > > > >> with 4.x
> > > > > >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> > > > > >> > should be causing the difference. It seems to be something
> > Lucene
> > > > > >> > is doing under
> > > > > >> the
> > > > > >> > covers.
> > > > > >> >
> > > > > >> > Again, if there's any other information if I can provide to
> help
> > > > > >> determine
> > > > > >> > what's going on, please let me know.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Matt
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > -------------------------------------------------------------------
> > > > > >> > -- To unsubscribe, e-mail:
> > > java-user-unsubscribe@lucene.apache.org
> > > > > >> > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > > > > >> >
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > >> For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > > > > >>
> > > > > >>
> > > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > >
> >
>
Re: Query performance in Lucene 4.x
Posted by Vitaly Funstein <vf...@gmail.com>.
Hmm, I guess your IndexSearcher is backed by a MultiReader which operates
on these "shards" you're referring to, which are supposed to be
single-segment indexes? If so, this topology sounds fairly equivalent, at
least in concept but maybe similar in performance as well, to the regular
case when you search a single index shard but allow multiple segments in
it, and let Lucene manage segment creation/merging, instead of effectively
disabling it.
But this is all a speculation on my part, absent a concrete side by side
comparison. Also, from your description, what you are doing to
extend/replace internal plumbing in IndexSearcher sounds awfully complex
and hard to envision... do make sure to run functional tests when it's all
in place. :)
On Wed, Oct 2, 2013 at 5:37 PM, Desidero <de...@gmail.com> wrote:
> Vitaly,
>
> Thanks for your comments.
>
> Unfortunately, thread pool task overload is not the problem. When I
> extended the IndexSearcher class last night, I had it create one task per
> shard (20 tasks) instead of the default which turned out to be somewhere
> around 320 (I didn't realize it created quite so many segments when I
> recreated my index yesterday!) and performance was still awful as long as
> there were multiple segments per shard.
>
> I understand that the fork/join framework doesn't work well with the
> standard IndexSearcher's parallel search implementation, but I extended the
> class and my implementation avoids the pitfalls as long as there are very
> few (preferably one) segments per shard. The blocking in the original
> SegmentWorkerCallable is still there, but I set it up so that if one worker
> already has the lock, any others that try to lock it will just throw their
> results in a separate queue which is polled by the thread that managed to
> get the lock. After all the child tasks are complete, the queue is checked
> one more time to be safe and then processing continues. There is basically
> no waiting in that chunk of code (per the profiler). After doing some tests
> with the ForkJoinPool and custom code as well as a standard
> ThreadPoolExecutor, I found that the Fork/Join implementation was faster.
>
> The system I'm working on is much more sensitive to response time (latency)
> than overall throughput, so we've had to focus on parallelism within
> queries rather than across many queries. Batch processing performance is
> still important, but we cannot sacrifice quick "online" responses. It would
> be much easier to avoid this whole mess, but we cannot meet our performance
> requirements without it.
>
> Regards,
>
> Matt
>
>
> On Oct 2, 2013 3:36 PM, "Vitaly Funstein" <vf...@gmail.com> wrote:
>
> > Matt,
> >
> > I think you are mostly on track with suspecting thread pool task overload
> > as the possible culprit here. First, the old school (prior to Java 7)
> > ThreadPoolExecutor only accepts a BlockingQueue to use internally for
> > worker tasks, instead of a concurrent variant (not sure why). So this
> > internal work queue will become a significant point of contention when
> > using the pool in a pattern similar to your use case, i.e. submitting
> lots
> > of tasks to the pool as fast as possible.
> >
> > Second, I am not too familiar with the internals of the fork/join pool
> > implementation in 1.7 (if that's what you're using), but from reading the
> > daunting javadoc for ForkJoinTask my rough guess is it's not terribly
> well
> > suited for use in IndexSearcher. In particular, one of the possible
> > "non-compliant" behaviors is that a mutex lock is taken for each call()
> > invocation that operates on an individual leaf slice. This is evident
> from
> > code inspection, and based on this, I am not sure what benefit, if any,
> > multi-threaded search over a multi-segment index would provide in
> general -
> > regardless of the choice of thread pool implementation.
> >
> > I think a better strategy, as mentioned in another thread, is to optimize
> > your system for multiple concurrent queries, rather than focusing on
> > forcing each query to run across multiple threads/cores. With that
> > approach, you could for instance, set up a non-blocking queue like
> > ConcurrentLinkedQueue that would contain individual query tasks, then
> use a
> > fixed pool of worker threads to consume the queue in a loop and run them.
> > In this scenario, you shouldn't need to pass an ExecutorService instance
> to
> > IndexSearcher.
> >
> > A strategy like that should provide for better query throughput,
> regardless
> > of whether each shard consists of a single segment or not - provided each
> > of the queries is tied to a particular shard and can't search any others.
> >
> >
> >
> > On Tue, Oct 1, 2013 at 4:10 PM, Desidero <de...@gmail.com> wrote:
> >
> > > Uwe,
> > >
> > > I was using a bounded thread pool.
> > >
> > > I don't know if the problem was the task overload or something about
> the
> > > actual efficiency of searching a single segment rather than iterating
> > over
> > > multiple AtomicReaderContexts, but I'd lean toward task overload. I
> will
> > do
> > > some testing tonight to find out for sure.
> > >
> > > Matt
> > > Hi,
> > >
> > > use a bounded thread pool.
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > > > -----Original Message-----
> > > > From: Desidero [mailto:desidero@gmail.com]
> > > > Sent: Tuesday, October 01, 2013 11:37 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: Query performance in Lucene 4.x
> > > >
> > > > For anyone who was wondering, this was actually resolved in a
> different
> > > > thread today. I misread the information in the
> > > > IndexSearcher(IndexReader,ExecutorService) constructor documentation
> -
> > I
> > > > was under the impression that it was submitting a thread for each
> index
> > > > shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
> > > submitting
> > > > a task for each segment within each shard (20 shards * ~10 segments =
> > > ~200
> > > > tasks) which is horrible. Since my index changes infrequently, I'm
> > using
> > > > forceMerge(1) before sending out updated indexes to the slave
> servers.
> > > > Without any extra tuning (threads, # of shards, etc) I've gone from
> > ~2900
> > > > requests per minute to ~10k requests per minute.
> > > >
> > > > Thanks to Adrien and Mike for the clarification and Benson for
> bringing
> > > up
> > > > the question that led to my answer.
> > > >
> > > > I'm still pretty new to Lucene so I have a lot of poking around to
> do,
> > > but I'm
> > > > going to try to implement the "virtual segment" concept that Mike
> > > > mentioned. It'll be really helpful for those of us who want
> parallelism
> > > within
> > > > queries and don't want to forceMerge.
> > > >
> > > >
> > > > On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com>
> wrote:
> > > >
> > > > > Erick,
> > > > >
> > > > > Thank you for responding.
> > > > >
> > > > > I ran tests using both compressed fields and uncompressed fields,
> and
> > > > > it was significantly slower with uncompressed fields. I looked into
> > > > > the lazy field loading per your suggestion, but we don't get any
> > > > > values from the returned Documents until the result set has been
> > > > appropriately reduced.
> > > > > Since we only store one retrievable field and we always need to get
> > > > > it, it doesn't save any time loading it lazily.
> > > > >
> > > > > I'll try running a test without loading any fields just to see how
> it
> > > > > affects performance and let you know how that goes.
> > > > >
> > > > > Regards,
> > > > > Matt
> > > > >
> > > > >
> > > > > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> > > > <er...@gmail.com>wrote:
> > > > >
> > > > >> Hmmm, since 4.1, fields have been stored compressed by default.
> > > > >> I suppose it's possible that this is a result of
> > > > >> compressing/uncompressing.
> > > > >>
> > > > >> What happens if
> > > > >> 1> you enable lazy field loading
> > > > >> 2> don't load any fields?
> > > > >>
> > > > >> FWIW,
> > > > >> Erick
> > > > >>
> > > > >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com>
> > > > wrote:
> > > > >> > A quick update:
> > > > >> >
> > > > >> > In order to confirm that none of the standard migration changes
> > had
> > > > >> > a negative effect on performance, I ported my Lucene 4.x version
> > > > >> > back to Lucene 3.6.2 and kept the newer API rather than using
> the
> > > > >> > custom ParallelMultiSearcher and other deprecated
> methods/classes.
> > > > >> >
> > > > >> > Performance in 3.6.2 is even faster than before (~2900
> > requests/min
> > > > >> with 4.x
> > > > >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> > > > >> > should be causing the difference. It seems to be something
> Lucene
> > > > >> > is doing under
> > > > >> the
> > > > >> > covers.
> > > > >> >
> > > > >> > Again, if there's any other information if I can provide to help
> > > > >> determine
> > > > >> > what's going on, please let me know.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Matt
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > -------------------------------------------------------------------
> > > > >> > -- To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> > > > >> > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > > > >> >
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >>
> > > > >>
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
>
Re: Query performance in Lucene 4.x
Posted by Desidero <de...@gmail.com>.
Vitaly,
Thanks for your comments.
Unfortunately, thread pool task overload is not the problem. When I
extended the IndexSearcher class last night, I had it create one task per
shard (20 tasks) instead of the default which turned out to be somewhere
around 320 (I didn't realize it created quite so many segments when I
recreated my index yesterday!) and performance was still awful as long as
there were multiple segments per shard.
I understand that the fork/join framework doesn't work well with the
standard IndexSearcher's parallel search implementation, but I extended the
class and my implementation avoids the pitfalls as long as there are very
few (preferably one) segments per shard. The blocking in the original
SegmentWorkerCallable is still there, but I set it up so that if one worker
already has the lock, any others that try to lock it will just throw their
results in a separate queue which is polled by the thread that managed to
get the lock. After all the child tasks are complete, the queue is checked
one more time to be safe and then processing continues. There is basically
no waiting in that chunk of code (per the profiler). After doing some tests
with the ForkJoinPool and custom code as well as a standard
ThreadPoolExecutor, I found that the Fork/Join implementation was faster.
The system I'm working on is much more sensitive to response time (latency)
than overall throughput, so we've had to focus on parallelism within
queries rather than across many queries. Batch processing performance is
still important, but we cannot sacrifice quick "online" responses. It would
be much easier to avoid this whole mess, but we cannot meet our performance
requirements without it.
Regards,
Matt
On Oct 2, 2013 3:36 PM, "Vitaly Funstein" <vf...@gmail.com> wrote:
> Matt,
>
> I think you are mostly on track with suspecting thread pool task overload
> as the possible culprit here. First, the old school (prior to Java 7)
> ThreadPoolExecutor only accepts a BlockingQueue to use internally for
> worker tasks, instead of a concurrent variant (not sure why). So this
> internal work queue will become a significant point of contention when
> using the pool in a pattern similar to your use case, i.e. submitting lots
> of tasks to the pool as fast as possible.
>
> Second, I am not too familiar with the internals of the fork/join pool
> implementation in 1.7 (if that's what you're using), but from reading the
> daunting javadoc for ForkJoinTask my rough guess is it's not terribly well
> suited for use in IndexSearcher. In particular, one of the possible
> "non-compliant" behaviors is that a mutex lock is taken for each call()
> invocation that operates on an individual leaf slice. This is evident from
> code inspection, and based on this, I am not sure what benefit, if any,
> multi-threaded search over a multi-segment index would provide in general -
> regardless of the choice of thread pool implementation.
>
> I think a better strategy, as mentioned in another thread, is to optimize
> your system for multiple concurrent queries, rather than focusing on
> forcing each query to run across multiple threads/cores. With that
> approach, you could for instance, set up a non-blocking queue like
> ConcurrentLinkedQueue that would contain individual query tasks, then use a
> fixed pool of worker threads to consume the queue in a loop and run them.
> In this scenario, you shouldn't need to pass an ExecutorService instance to
> IndexSearcher.
>
> A strategy like that should provide for better query throughput, regardless
> of whether each shard consists of a single segment or not - provided each
> of the queries is tied to a particular shard and can't search any others.
>
>
>
> On Tue, Oct 1, 2013 at 4:10 PM, Desidero <de...@gmail.com> wrote:
>
> > Uwe,
> >
> > I was using a bounded thread pool.
> >
> > I don't know if the problem was the task overload or something about the
> > actual efficiency of searching a single segment rather than iterating
> over
> > multiple AtomicReaderContexts, but I'd lean toward task overload. I will
> do
> > some testing tonight to find out for sure.
> >
> > Matt
> > Hi,
> >
> > use a bounded thread pool.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Desidero [mailto:desidero@gmail.com]
> > > Sent: Tuesday, October 01, 2013 11:37 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Query performance in Lucene 4.x
> > >
> > > For anyone who was wondering, this was actually resolved in a different
> > > thread today. I misread the information in the
> > > IndexSearcher(IndexReader,ExecutorService) constructor documentation -
> I
> > > was under the impression that it was submitting a thread for each index
> > > shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
> > submitting
> > > a task for each segment within each shard (20 shards * ~10 segments =
> > ~200
> > > tasks) which is horrible. Since my index changes infrequently, I'm
> using
> > > forceMerge(1) before sending out updated indexes to the slave servers.
> > > Without any extra tuning (threads, # of shards, etc) I've gone from
> ~2900
> > > requests per minute to ~10k requests per minute.
> > >
> > > Thanks to Adrien and Mike for the clarification and Benson for bringing
> > up
> > > the question that led to my answer.
> > >
> > > I'm still pretty new to Lucene so I have a lot of poking around to do,
> > but I'm
> > > going to try to implement the "virtual segment" concept that Mike
> > > mentioned. It'll be really helpful for those of us who want parallelism
> > within
> > > queries and don't want to forceMerge.
> > >
> > >
> > > On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com> wrote:
> > >
> > > > Erick,
> > > >
> > > > Thank you for responding.
> > > >
> > > > I ran tests using both compressed fields and uncompressed fields, and
> > > > it was significantly slower with uncompressed fields. I looked into
> > > > the lazy field loading per your suggestion, but we don't get any
> > > > values from the returned Documents until the result set has been
> > > appropriately reduced.
> > > > Since we only store one retrievable field and we always need to get
> > > > it, it doesn't save any time loading it lazily.
> > > >
> > > > I'll try running a test without loading any fields just to see how it
> > > > affects performance and let you know how that goes.
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > >
> > > > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> > > <er...@gmail.com>wrote:
> > > >
> > > >> Hmmm, since 4.1, fields have been stored compressed by default.
> > > >> I suppose it's possible that this is a result of
> > > >> compressing/uncompressing.
> > > >>
> > > >> What happens if
> > > >> 1> you enable lazy field loading
> > > >> 2> don't load any fields?
> > > >>
> > > >> FWIW,
> > > >> Erick
> > > >>
> > > >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com>
> > > wrote:
> > > >> > A quick update:
> > > >> >
> > > >> > In order to confirm that none of the standard migration changes
> had
> > > >> > a negative effect on performance, I ported my Lucene 4.x version
> > > >> > back to Lucene 3.6.2 and kept the newer API rather than using the
> > > >> > custom ParallelMultiSearcher and other deprecated methods/classes.
> > > >> >
> > > >> > Performance in 3.6.2 is even faster than before (~2900
> requests/min
> > > >> with 4.x
> > > >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> > > >> > should be causing the difference. It seems to be something Lucene
> > > >> > is doing under
> > > >> the
> > > >> > covers.
> > > >> >
> > > >> > Again, if there's any other information if I can provide to help
> > > >> determine
> > > >> > what's going on, please let me know.
> > > >> >
> > > >> > Thanks,
> > > >> > Matt
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> -------------------------------------------------------------------
> > > >> > -- To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> > > >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >> >
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >>
> > > >>
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
Re: Query performance in Lucene 4.x
Posted by Vitaly Funstein <vf...@gmail.com>.
Matt,
I think you are mostly on track with suspecting thread pool task overload
as the possible culprit here. First, the old school (prior to Java 7)
ThreadPoolExecutor only accepts a BlockingQueue to use internally for
worker tasks, instead of a concurrent variant (not sure why). So this
internal work queue will become a significant point of contention when
using the pool in a pattern similar to your use case, i.e. submitting lots
of tasks to the pool as fast as possible.
Second, I am not too familiar with the internals of the fork/join pool
implementation in 1.7 (if that's what you're using), but from reading the
daunting javadoc for ForkJoinTask my rough guess is it's not terribly well
suited for use in IndexSearcher. In particular, one of the possible
"non-compliant" behaviors is that a mutex lock is taken for each call()
invocation that operates on an individual leaf slice. This is evident from
code inspection, and based on this, I am not sure what benefit, if any,
multi-threaded search over a multi-segment index would provide in general -
regardless of the choice of thread pool implementation.
I think a better strategy, as mentioned in another thread, is to optimize
your system for multiple concurrent queries, rather than focusing on
forcing each query to run across multiple threads/cores. With that
approach, you could for instance, set up a non-blocking queue like
ConcurrentLinkedQueue that would contain individual query tasks, then use a
fixed pool of worker threads to consume the queue in a loop and run them.
In this scenario, you shouldn't need to pass an ExecutorService instance to
IndexSearcher.
A strategy like that should provide for better query throughput, regardless
of whether each shard consists of a single segment or not - provided each
of the queries is tied to a particular shard and can't search any others.
On Tue, Oct 1, 2013 at 4:10 PM, Desidero <de...@gmail.com> wrote:
> Uwe,
>
> I was using a bounded thread pool.
>
> I don't know if the problem was the task overload or something about the
> actual efficiency of searching a single segment rather than iterating over
> multiple AtomicReaderContexts, but I'd lean toward task overload. I will do
> some testing tonight to find out for sure.
>
> Matt
> Hi,
>
> use a bounded thread pool.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Desidero [mailto:desidero@gmail.com]
> > Sent: Tuesday, October 01, 2013 11:37 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Query performance in Lucene 4.x
> >
> > For anyone who was wondering, this was actually resolved in a different
> > thread today. I misread the information in the
> > IndexSearcher(IndexReader,ExecutorService) constructor documentation - I
> > was under the impression that it was submitting a thread for each index
> > shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
> submitting
> > a task for each segment within each shard (20 shards * ~10 segments =
> ~200
> > tasks) which is horrible. Since my index changes infrequently, I'm using
> > forceMerge(1) before sending out updated indexes to the slave servers.
> > Without any extra tuning (threads, # of shards, etc) I've gone from ~2900
> > requests per minute to ~10k requests per minute.
> >
> > Thanks to Adrien and Mike for the clarification and Benson for bringing
> up
> > the question that led to my answer.
> >
> > I'm still pretty new to Lucene so I have a lot of poking around to do,
> but I'm
> > going to try to implement the "virtual segment" concept that Mike
> > mentioned. It'll be really helpful for those of us who want parallelism
> within
> > queries and don't want to forceMerge.
> >
> >
> > On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com> wrote:
> >
> > > Erick,
> > >
> > > Thank you for responding.
> > >
> > > I ran tests using both compressed fields and uncompressed fields, and
> > > it was significantly slower with uncompressed fields. I looked into
> > > the lazy field loading per your suggestion, but we don't get any
> > > values from the returned Documents until the result set has been
> > appropriately reduced.
> > > Since we only store one retrievable field and we always need to get
> > > it, it doesn't save any time loading it lazily.
> > >
> > > I'll try running a test without loading any fields just to see how it
> > > affects performance and let you know how that goes.
> > >
> > > Regards,
> > > Matt
> > >
> > >
> > > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> > <er...@gmail.com>wrote:
> > >
> > >> Hmmm, since 4.1, fields have been stored compressed by default.
> > >> I suppose it's possible that this is a result of
> > >> compressing/uncompressing.
> > >>
> > >> What happens if
> > >> 1> you enable lazy field loading
> > >> 2> don't load any fields?
> > >>
> > >> FWIW,
> > >> Erick
> > >>
> > >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com>
> > wrote:
> > >> > A quick update:
> > >> >
> > >> > In order to confirm that none of the standard migration changes had
> > >> > a negative effect on performance, I ported my Lucene 4.x version
> > >> > back to Lucene 3.6.2 and kept the newer API rather than using the
> > >> > custom ParallelMultiSearcher and other deprecated methods/classes.
> > >> >
> > >> > Performance in 3.6.2 is even faster than before (~2900 requests/min
> > >> with 4.x
> > >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> > >> > should be causing the difference. It seems to be something Lucene
> > >> > is doing under
> > >> the
> > >> > covers.
> > >> >
> > >> > Again, if there's any other information if I can provide to help
> > >> determine
> > >> > what's going on, please let me know.
> > >> >
> > >> > Thanks,
> > >> > Matt
> > >> >
> > >> >
> > >> >
> > >> > -------------------------------------------------------------------
> > >> > -- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>
> > >>
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: Query performance in Lucene 4.x
Posted by Desidero <de...@gmail.com>.
Uwe,
I was using a bounded thread pool.
I don't know if the problem was the task overload or something about the
actual efficiency of searching a single segment rather than iterating over
multiple AtomicReaderContexts, but I'd lean toward task overload. I will do
some testing tonight to find out for sure.
Matt
Hi,
use a bounded thread pool.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Desidero [mailto:desidero@gmail.com]
> Sent: Tuesday, October 01, 2013 11:37 PM
> To: java-user@lucene.apache.org
> Subject: Re: Query performance in Lucene 4.x
>
> For anyone who was wondering, this was actually resolved in a different
> thread today. I misread the information in the
> IndexSearcher(IndexReader,ExecutorService) constructor documentation - I
> was under the impression that it was submitting a thread for each index
> shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
submitting
> a task for each segment within each shard (20 shards * ~10 segments = ~200
> tasks) which is horrible. Since my index changes infrequently, I'm using
> forceMerge(1) before sending out updated indexes to the slave servers.
> Without any extra tuning (threads, # of shards, etc) I've gone from ~2900
> requests per minute to ~10k requests per minute.
>
> Thanks to Adrien and Mike for the clarification and Benson for bringing up
> the question that led to my answer.
>
> I'm still pretty new to Lucene so I have a lot of poking around to do,
but I'm
> going to try to implement the "virtual segment" concept that Mike
> mentioned. It'll be really helpful for those of us who want parallelism
within
> queries and don't want to forceMerge.
>
>
> On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com> wrote:
>
> > Erick,
> >
> > Thank you for responding.
> >
> > I ran tests using both compressed fields and uncompressed fields, and
> > it was significantly slower with uncompressed fields. I looked into
> > the lazy field loading per your suggestion, but we don't get any
> > values from the returned Documents until the result set has been
> appropriately reduced.
> > Since we only store one retrievable field and we always need to get
> > it, it doesn't save any time loading it lazily.
> >
> > I'll try running a test without loading any fields just to see how it
> > affects performance and let you know how that goes.
> >
> > Regards,
> > Matt
> >
> >
> > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> <er...@gmail.com>wrote:
> >
> >> Hmmm, since 4.1, fields have been stored compressed by default.
> >> I suppose it's possible that this is a result of
> >> compressing/uncompressing.
> >>
> >> What happens if
> >> 1> you enable lazy field loading
> >> 2> don't load any fields?
> >>
> >> FWIW,
> >> Erick
> >>
> >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com>
> wrote:
> >> > A quick update:
> >> >
> >> > In order to confirm that none of the standard migration changes had
> >> > a negative effect on performance, I ported my Lucene 4.x version
> >> > back to Lucene 3.6.2 and kept the newer API rather than using the
> >> > custom ParallelMultiSearcher and other deprecated methods/classes.
> >> >
> >> > Performance in 3.6.2 is even faster than before (~2900 requests/min
> >> with 4.x
> >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> >> > should be causing the difference. It seems to be something Lucene
> >> > is doing under
> >> the
> >> > covers.
> >> >
> >> > Again, if there's any other information if I can provide to help
> >> determine
> >> > what's going on, please let me know.
> >> >
> >> > Thanks,
> >> > Matt
> >> >
> >> >
> >> >
> >> > -------------------------------------------------------------------
> >> > -- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Query performance in Lucene 4.x
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
use a bounded thread pool.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Desidero [mailto:desidero@gmail.com]
> Sent: Tuesday, October 01, 2013 11:37 PM
> To: java-user@lucene.apache.org
> Subject: Re: Query performance in Lucene 4.x
>
> For anyone who was wondering, this was actually resolved in a different
> thread today. I misread the information in the
> IndexSearcher(IndexReader,ExecutorService) constructor documentation - I
> was under the impression that it was submitting a thread for each index
> shard (MultiReader wraps 20 shards, so 20 tasks) but it was really submitting
> a task for each segment within each shard (20 shards * ~10 segments = ~200
> tasks) which is horrible. Since my index changes infrequently, I'm using
> forceMerge(1) before sending out updated indexes to the slave servers.
> Without any extra tuning (threads, # of shards, etc) I've gone from ~2900
> requests per minute to ~10k requests per minute.
>
> Thanks to Adrien and Mike for the clarification and Benson for bringing up
> the question that led to my answer.
>
> I'm still pretty new to Lucene so I have a lot of poking around to do, but I'm
> going to try to implement the "virtual segment" concept that Mike
> mentioned. It'll be really helpful for those of us who want parallelism within
> queries and don't want to forceMerge.
>
>
> On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com> wrote:
>
> > Erick,
> >
> > Thank you for responding.
> >
> > I ran tests using both compressed fields and uncompressed fields, and
> > it was significantly slower with uncompressed fields. I looked into
> > the lazy field loading per your suggestion, but we don't get any
> > values from the returned Documents until the result set has been
> appropriately reduced.
> > Since we only store one retrievable field and we always need to get
> > it, it doesn't save any time loading it lazily.
> >
> > I'll try running a test without loading any fields just to see how it
> > affects performance and let you know how that goes.
> >
> > Regards,
> > Matt
> >
> >
> > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> <er...@gmail.com>wrote:
> >
> >> Hmmm, since 4.1, fields have been stored compressed by default.
> >> I suppose it's possible that this is a result of
> >> compressing/uncompressing.
> >>
> >> What happens if
> >> 1> you enable lazy field loading
> >> 2> don't load any fields?
> >>
> >> FWIW,
> >> Erick
> >>
> >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com>
> wrote:
> >> > A quick update:
> >> >
> >> > In order to confirm that none of the standard migration changes had
> >> > a negative effect on performance, I ported my Lucene 4.x version
> >> > back to Lucene 3.6.2 and kept the newer API rather than using the
> >> > custom ParallelMultiSearcher and other deprecated methods/classes.
> >> >
> >> > Performance in 3.6.2 is even faster than before (~2900 requests/min
> >> with 4.x
> >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> >> > should be causing the difference. It seems to be something Lucene
> >> > is doing under
> >> the
> >> > covers.
> >> >
> >> > Again, if there's any other information if I can provide to help
> >> determine
> >> > what's going on, please let me know.
> >> >
> >> > Thanks,
> >> > Matt
> >> >
> >> >
> >> >
> >> > -------------------------------------------------------------------
> >> > -- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Query performance in Lucene 4.x
Posted by Desidero <de...@gmail.com>.
For anyone who was wondering, this was actually resolved in a different
thread today. I misread the information in the
IndexSearcher(IndexReader,ExecutorService) constructor documentation - I
was under the impression that it was submitting a thread for each index
shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
submitting a task for each segment within each shard (20 shards * ~10
segments = ~200 tasks) which is horrible. Since my index changes
infrequently, I'm using forceMerge(1) before sending out updated indexes to
the slave servers. Without any extra tuning (threads, # of shards, etc)
I've gone from ~2900 requests per minute to ~10k requests per minute.
Thanks to Adrien and Mike for the clarification and Benson for bringing up
the question that led to my answer.
I'm still pretty new to Lucene so I have a lot of poking around to do, but
I'm going to try to implement the "virtual segment" concept that Mike
mentioned. It'll be really helpful for those of us who want parallelism
within queries and don't want to forceMerge.
On Fri, Sep 27, 2013 at 9:55 AM, Desidero <de...@gmail.com> wrote:
> Erick,
>
> Thank you for responding.
>
> I ran tests using both compressed fields and uncompressed fields, and it
> was significantly slower with uncompressed fields. I looked into the lazy
> field loading per your suggestion, but we don't get any values from the
> returned Documents until the result set has been appropriately reduced.
> Since we only store one retrievable field and we always need to get it, it
> doesn't save any time loading it lazily.
>
> I'll try running a test without loading any fields just to see how it
> affects performance and let you know how that goes.
>
> Regards,
> Matt
>
>
> On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson <er...@gmail.com>wrote:
>
>> Hmmm, since 4.1, fields have been stored compressed by default.
>> I suppose it's possible that this is a result of
>> compressing/uncompressing.
>>
>> What happens if
>> 1> you enable lazy field loading
>> 2> don't load any fields?
>>
>> FWIW,
>> Erick
>>
>> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com> wrote:
>> > A quick update:
>> >
>> > In order to confirm that none of the standard migration changes had a
>> > negative effect on performance, I ported my Lucene 4.x version back to
>> > Lucene 3.6.2 and kept the newer API rather than using the custom
>> > ParallelMultiSearcher and other deprecated methods/classes.
>> >
>> > Performance in 3.6.2 is even faster than before (~2900 requests/min
>> with 4.x
>> > vs ~6200 requests/min with 3.6.2), so none of my code changes should be
>> > causing the difference. It seems to be something Lucene is doing under
>> the
>> > covers.
>> >
>> > Again, if there's any other information if I can provide to help
>> determine
>> > what's going on, please let me know.
>> >
>> > Thanks,
>> > Matt
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Query performance in Lucene 4.x
Posted by Desidero <de...@gmail.com>.
Erick,
Thank you for responding.
I ran tests using both compressed fields and uncompressed fields, and it
was significantly slower with uncompressed fields. I looked into the lazy
field loading per your suggestion, but we don't get any values from the
returned Documents until the result set has been appropriately reduced.
Since we only store one retrievable field and we always need to get it, it
doesn't save any time loading it lazily.
I'll try running a test without loading any fields just to see how it
affects performance and let you know how that goes.
Regards,
Matt
On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson <er...@gmail.com>wrote:
> Hmmm, since 4.1, fields have been stored compressed by default.
> I suppose it's possible that this is a result of compressing/uncompressing.
>
> What happens if
> 1> you enable lazy field loading
> 2> don't load any fields?
>
> FWIW,
> Erick
>
> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com> wrote:
> > A quick update:
> >
> > In order to confirm that none of the standard migration changes had a
> > negative effect on performance, I ported my Lucene 4.x version back to
> > Lucene 3.6.2 and kept the newer API rather than using the custom
> > ParallelMultiSearcher and other deprecated methods/classes.
> >
> > Performance in 3.6.2 is even faster than before (~2900 requests/min with
> 4.x
> > vs ~6200 requests/min with 3.6.2), so none of my code changes should be
> > causing the difference. It seems to be something Lucene is doing under
> the
> > covers.
> >
> > Again, if there's any other information if I can provide to help
> determine
> > what's going on, please let me know.
> >
> > Thanks,
> > Matt
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Query performance in Lucene 4.x
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, since 4.1, fields have been stored compressed by default.
I suppose it's possible that this is a result of compressing/uncompressing.
What happens if
1> you enable lazy field loading
2> don't load any fields?
FWIW,
Erick
On Thu, Sep 26, 2013 at 10:55 AM, Desidero <de...@gmail.com> wrote:
> A quick update:
>
> In order to confirm that none of the standard migration changes had a
> negative effect on performance, I ported my Lucene 4.x version back to
> Lucene 3.6.2 and kept the newer API rather than using the custom
> ParallelMultiSearcher and other deprecated methods/classes.
>
> Performance in 3.6.2 is even faster than before (~2900 requests/min with 4.x
> vs ~6200 requests/min with 3.6.2), so none of my code changes should be
> causing the difference. It seems to be something Lucene is doing under the
> covers.
>
> Again, if there's any other information if I can provide to help determine
> what's going on, please let me know.
>
> Thanks,
> Matt
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Query performance in Lucene 4.x
Posted by Desidero <de...@gmail.com>.
A quick update:
In order to confirm that none of the standard migration changes had a
negative effect on performance, I ported my Lucene 4.x version back to
Lucene 3.6.2 and kept the newer API rather than using the custom
ParallelMultiSearcher and other deprecated methods/classes.
Performance in 3.6.2 is even faster than before (~2900 requests/min with 4.x
vs ~6200 requests/min with 3.6.2), so none of my code changes should be
causing the difference. It seems to be something Lucene is doing under the
covers.
Again, if there's any other information if I can provide to help determine
what's going on, please let me know.
Thanks,
Matt
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org