You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Peter Keegan <pe...@gmail.com> on 2009/02/07 00:27:23 UTC
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization
bottleneck in MatchAlldocsQuery$MatchAllScorer
I just ran into the same bottleneck with ValueSourceScorer.
Here's a stack trace:
Name: QueryThread group 1,#4
State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned by:
QueryThread group 1,#8
Total blocked: 1,535,881 Total waited: 1,080
Stack trace:
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
org.apache.lucene.search.Searcher.search(Searcher.java:118)
Peter
On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA)
<ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
> Resolution: Fixed
>
> Committed revision 737513. Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
> > ------------------------------------------------------------------------
> >
> > Key: LUCENE-1316
> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
> > Project: Lucene - Java
> > Issue Type: Bug
> > Components: Query/Scoring
> > Affects Versions: 2.3
> > Environment: All
> > Reporter: Todd Feak
> > Assignee: Michael McCandless
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
> MatchAllDocsQuery.java
> >
> > Original Estimate: 1h
> > Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number of
> times as a potential synchronization bottleneck. However, the reason this
> bottleneck occurs is actually at a higher level that wasn't focused on (at
> least in the threads I read).
> > In every case I saw where a stack trace was provided to show the
> lock/block, higher in the stack you see the MatchAllScorer.next() method. In
> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly
> poor performance (order of magnitude) on our load tests for NOT queries, due
> to this bottleneck. The problem is that every single document is run through
> this isDeleted() method, which is synchronized. Having an optimized index
> exacerbates this issues, as there is only a single SegmentReader to
> synchronize on, causing a major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any deletions
> in the reader, much of this can be avoided. Especially in a read-only
> environment for production where you have slaves doing all the high load
> searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> > if (!reader.isDeleted(id)) {
> > TO:
> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major performance
> improvement. We also got the same query results. I don't believe this will
> improve the situation for indexes that have deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Michael McCandless <lu...@mikemccandless.com>.
Indeed!
I'll open an issue...
Thanks Peter.
Mike
Peter Keegan wrote:
> I just ran into the same bottleneck with ValueSourceScorer.
> Here's a stack trace:
>
> Name: QueryThread group 1,#4
> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83
> owned by: QueryThread group 1,#8
> Total blocked: 1,535,881 Total waited: 1,080
>
> Stack trace:
> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
> 663)
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.next(ValueSourceQuery.java:138)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
> org.apache.lucene.search.function.CustomScoreQuery
> $CustomScorer.skipTo(CustomScoreQuery.java:399)
> org
> .apache
> .lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
> org
> .apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:
> 51)
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org
> > wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
> Resolution: Fixed
>
> Committed revision 737513. Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery
> $MatchAllScorer
> >
> ------------------------------------------------------------------------
> >
> > Key: LUCENE-1316
> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
> > Project: Lucene - Java
> > Issue Type: Bug
> > Components: Query/Scoring
> > Affects Versions: 2.3
> > Environment: All
> > Reporter: Todd Feak
> > Assignee: Michael McCandless
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
> LUCENE_1316.patch, MatchAllDocsQuery.java
> >
> > Original Estimate: 1h
> > Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number
> of times as a potential synchronization bottleneck. However, the
> reason this bottleneck occurs is actually at a higher level that
> wasn't focused on (at least in the threads I read).
> > In every case I saw where a stack trace was provided to show the
> lock/block, higher in the stack you see the MatchAllScorer.next()
> method. In Solr paricularly, this scorer is used for "NOT" queries.
> We saw incredibly poor performance (order of magnitude) on our load
> tests for NOT queries, due to this bottleneck. The problem is that
> every single document is run through this isDeleted() method, which
> is synchronized. Having an optimized index exacerbates this issues,
> as there is only a single SegmentReader to synchronize on, causing a
> major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any
> deletions in the reader, much of this can be avoided. Especially in
> a read-only environment for production where you have slaves doing
> all the high load searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> > if (!reader.isDeleted(id)) {
> > TO:
> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major
> performance improvement. We also got the same query results. I
> don't believe this will improve the situation for indexes that have
> deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Michael McCandless <lu...@mikemccandless.com>.
Super, thanks Peter!
Mike
Peter Keegan wrote:
> Ok, I applied LUCENE-1316. (I had to patch MutliSegmentReader
> manually, probably because I'm running 2.3.2 and patch is for 2.4?)
> The patch fixes the sychronization bottleneck in ValueSourceQuery.
>
> Thank you Mike!
>
> Peter
>
> On Sat, Feb 7, 2009 at 10:38 AM, Michael McCandless <lucene@mikemccandless.com
> > wrote:
>
> Hi Peter,
>
> You'll definitely need to start from the full patch in LUCENE-1316,
> which fixes SegmentReader & Multi*Reader to use AllTermDocs when
> given a null Term.
>
> Mike
>
>
> Peter Keegan wrote:
>
> Hi Mike,
>
> Is this patch self contained or do I need the other patches from
> LUCENE-1316?
> I tried just this patch, but I get a NPE in MultiSegment Reader
> (current = null):
>
> at org.apache.lucene.index.MultiSegmentReader
> $MultiTermDocs.doc(MultiSegmentReader.java:531)
> at org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.score(ValueSourceQuery.java:142)
> at org.apache.lucene.search.function.CustomScoreQuery
> $CustomScorer.score(CustomScoreQuery.java:389)
> at org.apache.lucene.search.Scorer.score(Scorer.java:58)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless <lucene@mikemccandless.com
> > wrote:
>
> OK I just put the patch on there -- let me know. Thanks,
>
> Mike
>
>
> Peter Keegan wrote:
>
> Thanks, Mike. I'll be happy to help test the patch.
>
> Peter
>
> On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <lucene@mikemccandless.com
> > wrote:
>
> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>
> Mike
>
>
> Peter Keegan wrote:
>
> I just ran into the same bottleneck with ValueSourceScorer.
> Here's a stack trace:
>
> Name: QueryThread group 1,#4
> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83
> owned by: QueryThread group 1,#8
> Total blocked: 1,535,881 Total waited: 1,080
>
> Stack trace:
> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
> 663)
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.next(ValueSourceQuery.java:138)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
> org.apache.lucene.search.function.CustomScoreQuery
> $CustomScorer.skipTo(CustomScoreQuery.java:399)
> org
> .apache
> .lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
> org
> .apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:
> 51)
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org
> > wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
> Resolution: Fixed
>
> Committed revision 737513. Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery
> $MatchAllScorer
> >
> ------------------------------------------------------------------------
> >
> > Key: LUCENE-1316
> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
> > Project: Lucene - Java
> > Issue Type: Bug
> > Components: Query/Scoring
> > Affects Versions: 2.3
> > Environment: All
> > Reporter: Todd Feak
> > Assignee: Michael McCandless
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
> LUCENE_1316.patch, MatchAllDocsQuery.java
> >
> > Original Estimate: 1h
> > Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number
> of times as a potential synchronization bottleneck. However, the
> reason this bottleneck occurs is actually at a higher level that
> wasn't focused on (at least in the threads I read).
> > In every case I saw where a stack trace was provided to show the
> lock/block, higher in the stack you see the MatchAllScorer.next()
> method. In Solr paricularly, this scorer is used for "NOT" queries.
> We saw incredibly poor performance (order of magnitude) on our load
> tests for NOT queries, due to this bottleneck. The problem is that
> every single document is run through this isDeleted() method, which
> is synchronized. Having an optimized index exacerbates this issues,
> as there is only a single SegmentReader to synchronize on, causing a
> major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any
> deletions in the reader, much of this can be avoided. Especially in
> a read-only environment for production where you have slaves doing
> all the high load searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> > if (!reader.isDeleted(id)) {
> > TO:
> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major
> performance improvement. We also got the same query results. I
> don't believe this will improve the situation for indexes that have
> deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization
bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Peter Keegan <pe...@gmail.com>.
Ok, I applied LUCENE-1316. (I had to patch MutliSegmentReader manually,
probably because I'm running 2.3.2 and patch is for 2.4?)
The patch fixes the sychronization bottleneck in ValueSourceQuery.
Thank you Mike!
Peter
On Sat, Feb 7, 2009 at 10:38 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> Hi Peter,
>
> You'll definitely need to start from the full patch in LUCENE-1316, which
> fixes SegmentReader & Multi*Reader to use AllTermDocs when given a null
> Term.
>
> Mike
>
>
> Peter Keegan wrote:
>
> Hi Mike,
>>
>> Is this patch self contained or do I need the other patches from
>> LUCENE-1316?
>> I tried just this patch, but I get a NPE in MultiSegment Reader (current
>> = null):
>>
>> at
>> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.doc(MultiSegmentReader.java:531)
>> at
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.score(ValueSourceQuery.java:142)
>> at
>> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.score(CustomScoreQuery.java:389)
>> at org.apache.lucene.search.Scorer.score(Scorer.java:58)
>> at
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> Peter
>>
>>
>> On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> OK I just put the patch on there -- let me know. Thanks,
>>
>> Mike
>>
>>
>> Peter Keegan wrote:
>>
>> Thanks, Mike. I'll be happy to help test the patch.
>>
>> Peter
>>
>> On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>>
>> Mike
>>
>>
>> Peter Keegan wrote:
>>
>> I just ran into the same bottleneck with ValueSourceScorer.
>> Here's a stack trace:
>>
>> Name: QueryThread group 1,#4
>> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned
>> by: QueryThread group 1,#8
>> Total blocked: 1,535,881 Total waited: 1,080
>>
>> Stack trace:
>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
>> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
>>
>> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
>>
>> org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
>> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> Peter
>>
>>
>> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <
>> jira@apache.org> wrote:
>>
>> [
>> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> ]
>>
>> Michael McCandless resolved LUCENE-1316.
>> ----------------------------------------
>>
>> Resolution: Fixed
>>
>> Committed revision 737513. Thanks everyone!
>>
>> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
>> > ------------------------------------------------------------------------
>> >
>> > Key: LUCENE-1316
>> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
>> > Project: Lucene - Java
>> > Issue Type: Bug
>> > Components: Query/Scoring
>> > Affects Versions: 2.3
>> > Environment: All
>> > Reporter: Todd Feak
>> > Assignee: Michael McCandless
>> > Priority: Minor
>> > Fix For: 2.9
>> >
>> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
>> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
>> MatchAllDocsQuery.java
>> >
>> > Original Estimate: 1h
>> > Remaining Estimate: 1h
>> >
>> > The isDeleted() method on IndexReader has been mentioned a number of
>> times as a potential synchronization bottleneck. However, the reason this
>> bottleneck occurs is actually at a higher level that wasn't focused on (at
>> least in the threads I read).
>> > In every case I saw where a stack trace was provided to show the
>> lock/block, higher in the stack you see the MatchAllScorer.next() method. In
>> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly
>> poor performance (order of magnitude) on our load tests for NOT queries, due
>> to this bottleneck. The problem is that every single document is run through
>> this isDeleted() method, which is synchronized. Having an optimized index
>> exacerbates this issues, as there is only a single SegmentReader to
>> synchronize on, causing a major thread pileup waiting for the lock.
>> > By simply having the MatchAllScorer see if there have been any deletions
>> in the reader, much of this can be avoided. Especially in a read-only
>> environment for production where you have slaves doing all the high load
>> searching.
>> > I modified line 67 in the MatchAllDocsQuery
>> > FROM:
>> > if (!reader.isDeleted(id)) {
>> > TO:
>> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
>> > In our micro load test for NOT queries only, this was a major
>> performance improvement. We also got the same query results. I don't
>> believe this will improve the situation for indexes that have deletions.
>> > Please consider making this adjustment for a future bug fix release.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hi Peter,
You'll definitely need to start from the full patch in LUCENE-1316,
which fixes SegmentReader & Multi*Reader to use AllTermDocs when given
a null Term.
Mike
Peter Keegan wrote:
> Hi Mike,
>
> Is this patch self contained or do I need the other patches from
> LUCENE-1316?
> I tried just this patch, but I get a NPE in MultiSegment Reader
> (current = null):
>
> at org.apache.lucene.index.MultiSegmentReader
> $MultiTermDocs.doc(MultiSegmentReader.java:531)
> at org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.score(ValueSourceQuery.java:142)
> at org.apache.lucene.search.function.CustomScoreQuery
> $CustomScorer.score(CustomScoreQuery.java:389)
> at org.apache.lucene.search.Scorer.score(Scorer.java:58)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless <lucene@mikemccandless.com
> > wrote:
>
> OK I just put the patch on there -- let me know. Thanks,
>
> Mike
>
>
> Peter Keegan wrote:
>
> Thanks, Mike. I'll be happy to help test the patch.
>
> Peter
>
> On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <lucene@mikemccandless.com
> > wrote:
>
> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>
> Mike
>
>
> Peter Keegan wrote:
>
> I just ran into the same bottleneck with ValueSourceScorer.
> Here's a stack trace:
>
> Name: QueryThread group 1,#4
> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83
> owned by: QueryThread group 1,#8
> Total blocked: 1,535,881 Total waited: 1,080
>
> Stack trace:
> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
> 663)
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.next(ValueSourceQuery.java:138)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
> org.apache.lucene.search.function.CustomScoreQuery
> $CustomScorer.skipTo(CustomScoreQuery.java:399)
> org
> .apache
> .lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
> org
> .apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:
> 51)
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org
> > wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
> Resolution: Fixed
>
> Committed revision 737513. Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery
> $MatchAllScorer
> >
> ------------------------------------------------------------------------
> >
> > Key: LUCENE-1316
> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
> > Project: Lucene - Java
> > Issue Type: Bug
> > Components: Query/Scoring
> > Affects Versions: 2.3
> > Environment: All
> > Reporter: Todd Feak
> > Assignee: Michael McCandless
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
> LUCENE_1316.patch, MatchAllDocsQuery.java
> >
> > Original Estimate: 1h
> > Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number
> of times as a potential synchronization bottleneck. However, the
> reason this bottleneck occurs is actually at a higher level that
> wasn't focused on (at least in the threads I read).
> > In every case I saw where a stack trace was provided to show the
> lock/block, higher in the stack you see the MatchAllScorer.next()
> method. In Solr paricularly, this scorer is used for "NOT" queries.
> We saw incredibly poor performance (order of magnitude) on our load
> tests for NOT queries, due to this bottleneck. The problem is that
> every single document is run through this isDeleted() method, which
> is synchronized. Having an optimized index exacerbates this issues,
> as there is only a single SegmentReader to synchronize on, causing a
> major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any
> deletions in the reader, much of this can be avoided. Especially in
> a read-only environment for production where you have slaves doing
> all the high load searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> > if (!reader.isDeleted(id)) {
> > TO:
> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major
> performance improvement. We also got the same query results. I
> don't believe this will improve the situation for indexes that have
> deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization
bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Peter Keegan <pe...@gmail.com>.
Hi Mike,
Is this patch self contained or do I need the other patches from
LUCENE-1316?
I tried just this patch, but I get a NPE in MultiSegment Reader (current =
null):
at
org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.doc(MultiSegmentReader.java:531)
at
org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.score(ValueSourceQuery.java:142)
at
org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.score(CustomScoreQuery.java:389)
at org.apache.lucene.search.Scorer.score(Scorer.java:58)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.Searcher.search(Searcher.java:118)
Peter
On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> OK I just put the patch on there -- let me know. Thanks,
>
> Mike
>
>
> Peter Keegan wrote:
>
> Thanks, Mike. I'll be happy to help test the patch.
>>
>> Peter
>>
>> On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>>
>> Mike
>>
>>
>> Peter Keegan wrote:
>>
>> I just ran into the same bottleneck with ValueSourceScorer.
>> Here's a stack trace:
>>
>> Name: QueryThread group 1,#4
>> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned
>> by: QueryThread group 1,#8
>> Total blocked: 1,535,881 Total waited: 1,080
>>
>> Stack trace:
>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
>> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
>>
>> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
>>
>> org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
>> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> Peter
>>
>>
>> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <
>> jira@apache.org> wrote:
>>
>> [
>> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> ]
>>
>> Michael McCandless resolved LUCENE-1316.
>> ----------------------------------------
>>
>> Resolution: Fixed
>>
>> Committed revision 737513. Thanks everyone!
>>
>> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
>> > ------------------------------------------------------------------------
>> >
>> > Key: LUCENE-1316
>> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
>> > Project: Lucene - Java
>> > Issue Type: Bug
>> > Components: Query/Scoring
>> > Affects Versions: 2.3
>> > Environment: All
>> > Reporter: Todd Feak
>> > Assignee: Michael McCandless
>> > Priority: Minor
>> > Fix For: 2.9
>> >
>> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
>> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
>> MatchAllDocsQuery.java
>> >
>> > Original Estimate: 1h
>> > Remaining Estimate: 1h
>> >
>> > The isDeleted() method on IndexReader has been mentioned a number of
>> times as a potential synchronization bottleneck. However, the reason this
>> bottleneck occurs is actually at a higher level that wasn't focused on (at
>> least in the threads I read).
>> > In every case I saw where a stack trace was provided to show the
>> lock/block, higher in the stack you see the MatchAllScorer.next() method. In
>> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly
>> poor performance (order of magnitude) on our load tests for NOT queries, due
>> to this bottleneck. The problem is that every single document is run through
>> this isDeleted() method, which is synchronized. Having an optimized index
>> exacerbates this issues, as there is only a single SegmentReader to
>> synchronize on, causing a major thread pileup waiting for the lock.
>> > By simply having the MatchAllScorer see if there have been any deletions
>> in the reader, much of this can be avoided. Especially in a read-only
>> environment for production where you have slaves doing all the high load
>> searching.
>> > I modified line 67 in the MatchAllDocsQuery
>> > FROM:
>> > if (!reader.isDeleted(id)) {
>> > TO:
>> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
>> > In our micro load test for NOT queries only, this was a major
>> performance improvement. We also got the same query results. I don't
>> believe this will improve the situation for indexes that have deletions.
>> > Please consider making this adjustment for a future bug fix release.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK I just put the patch on there -- let me know. Thanks,
Mike
Peter Keegan wrote:
> Thanks, Mike. I'll be happy to help test the patch.
>
> Peter
>
> On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <lucene@mikemccandless.com
> > wrote:
>
> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>
> Mike
>
>
> Peter Keegan wrote:
>
> I just ran into the same bottleneck with ValueSourceScorer.
> Here's a stack trace:
>
> Name: QueryThread group 1,#4
> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83
> owned by: QueryThread group 1,#8
> Total blocked: 1,535,881 Total waited: 1,080
>
> Stack trace:
> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
> 663)
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.next(ValueSourceQuery.java:138)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
> org.apache.lucene.search.function.CustomScoreQuery
> $CustomScorer.skipTo(CustomScoreQuery.java:399)
> org
> .apache
> .lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
> org
> .apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:
> 51)
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org
> > wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
> Resolution: Fixed
>
> Committed revision 737513. Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery
> $MatchAllScorer
> >
> ------------------------------------------------------------------------
> >
> > Key: LUCENE-1316
> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
> > Project: Lucene - Java
> > Issue Type: Bug
> > Components: Query/Scoring
> > Affects Versions: 2.3
> > Environment: All
> > Reporter: Todd Feak
> > Assignee: Michael McCandless
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
> LUCENE_1316.patch, MatchAllDocsQuery.java
> >
> > Original Estimate: 1h
> > Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number
> of times as a potential synchronization bottleneck. However, the
> reason this bottleneck occurs is actually at a higher level that
> wasn't focused on (at least in the threads I read).
> > In every case I saw where a stack trace was provided to show the
> lock/block, higher in the stack you see the MatchAllScorer.next()
> method. In Solr paricularly, this scorer is used for "NOT" queries.
> We saw incredibly poor performance (order of magnitude) on our load
> tests for NOT queries, due to this bottleneck. The problem is that
> every single document is run through this isDeleted() method, which
> is synchronized. Having an optimized index exacerbates this issues,
> as there is only a single SegmentReader to synchronize on, causing a
> major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any
> deletions in the reader, much of this can be avoided. Especially in
> a read-only environment for production where you have slaves doing
> all the high load searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> > if (!reader.isDeleted(id)) {
> > TO:
> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major
> performance improvement. We also got the same query results. I
> don't believe this will improve the situation for indexes that have
> deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization
bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Peter Keegan <pe...@gmail.com>.
Thanks, Mike. I'll be happy to help test the patch.
Peter
On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>
> Mike
>
>
> Peter Keegan wrote:
>
> I just ran into the same bottleneck with ValueSourceScorer.
>> Here's a stack trace:
>>
>> Name: QueryThread group 1,#4
>> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned
>> by: QueryThread group 1,#8
>> Total blocked: 1,535,881 Total waited: 1,080
>>
>> Stack trace:
>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
>> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
>>
>> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
>>
>> org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
>> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> Peter
>>
>>
>> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <
>> jira@apache.org> wrote:
>>
>> [
>> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> ]
>>
>> Michael McCandless resolved LUCENE-1316.
>> ----------------------------------------
>>
>> Resolution: Fixed
>>
>> Committed revision 737513. Thanks everyone!
>>
>> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
>> > ------------------------------------------------------------------------
>> >
>> > Key: LUCENE-1316
>> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
>> > Project: Lucene - Java
>> > Issue Type: Bug
>> > Components: Query/Scoring
>> > Affects Versions: 2.3
>> > Environment: All
>> > Reporter: Todd Feak
>> > Assignee: Michael McCandless
>> > Priority: Minor
>> > Fix For: 2.9
>> >
>> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
>> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
>> MatchAllDocsQuery.java
>> >
>> > Original Estimate: 1h
>> > Remaining Estimate: 1h
>> >
>> > The isDeleted() method on IndexReader has been mentioned a number of
>> times as a potential synchronization bottleneck. However, the reason this
>> bottleneck occurs is actually at a higher level that wasn't focused on (at
>> least in the threads I read).
>> > In every case I saw where a stack trace was provided to show the
>> lock/block, higher in the stack you see the MatchAllScorer.next() method. In
>> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly
>> poor performance (order of magnitude) on our load tests for NOT queries, due
>> to this bottleneck. The problem is that every single document is run through
>> this isDeleted() method, which is synchronized. Having an optimized index
>> exacerbates this issues, as there is only a single SegmentReader to
>> synchronize on, causing a major thread pileup waiting for the lock.
>> > By simply having the MatchAllScorer see if there have been any deletions
>> in the reader, much of this can be avoided. Especially in a read-only
>> environment for production where you have slaves doing all the high load
>> searching.
>> > I modified line 67 in the MatchAllDocsQuery
>> > FROM:
>> > if (!reader.isDeleted(id)) {
>> > TO:
>> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
>> > In our micro load test for NOT queries only, this was a major
>> performance improvement. We also got the same query results. I don't
>> believe this will improve the situation for indexes that have deletions.
>> > Please consider making this adjustment for a future bug fix release.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
Mike
Peter Keegan wrote:
> I just ran into the same bottleneck with ValueSourceScorer.
> Here's a stack trace:
>
> Name: QueryThread group 1,#4
> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83
> owned by: QueryThread group 1,#8
> Total blocked: 1,535,881 Total waited: 1,080
>
> Stack trace:
> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
> 663)
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.next(ValueSourceQuery.java:138)
> org.apache.lucene.search.function.ValueSourceQuery
> $ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
> org.apache.lucene.search.function.CustomScoreQuery
> $CustomScorer.skipTo(CustomScoreQuery.java:399)
> org
> .apache
> .lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
> org
> .apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:
> 51)
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org
> > wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
> Resolution: Fixed
>
> Committed revision 737513. Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery
> $MatchAllScorer
> >
> ------------------------------------------------------------------------
> >
> > Key: LUCENE-1316
> > URL: https://issues.apache.org/jira/browse/LUCENE-1316
> > Project: Lucene - Java
> > Issue Type: Bug
> > Components: Query/Scoring
> > Affects Versions: 2.3
> > Environment: All
> > Reporter: Todd Feak
> > Assignee: Michael McCandless
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
> LUCENE_1316.patch, MatchAllDocsQuery.java
> >
> > Original Estimate: 1h
> > Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number
> of times as a potential synchronization bottleneck. However, the
> reason this bottleneck occurs is actually at a higher level that
> wasn't focused on (at least in the threads I read).
> > In every case I saw where a stack trace was provided to show the
> lock/block, higher in the stack you see the MatchAllScorer.next()
> method. In Solr paricularly, this scorer is used for "NOT" queries.
> We saw incredibly poor performance (order of magnitude) on our load
> tests for NOT queries, due to this bottleneck. The problem is that
> every single document is run through this isDeleted() method, which
> is synchronized. Having an optimized index exacerbates this issues,
> as there is only a single SegmentReader to synchronize on, causing a
> major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any
> deletions in the reader, much of this can be avoided. Especially in
> a read-only environment for production where you have slaves doing
> all the high load searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> > if (!reader.isDeleted(id)) {
> > TO:
> > if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major
> performance improvement. We also got the same query results. I
> don't believe this will improve the situation for indexes that have
> deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org