You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2009/02/04 21:29:59 UTC
[jira] Created: (LUCENE-1536) if a filter can support random access
API, we should use it
if a filter can support random access API, we should use it
-----------------------------------------------------------
Key: LUCENE-1536
URL: https://issues.apache.org/jira/browse/LUCENE-1536
Project: Lucene - Java
Issue Type: Improvement
Components: Search
Affects Versions: 2.4
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
I ran some performance tests, comparing applying a filter via
random-access API instead of current trunk's iterator API.
This was inspired by LUCENE-1476, where we realized deletions should
really be implemented just like a filter, but then in testing found
that switching deletions to iterator was a very sizable performance
hit.
Some notes on the test:
* Index is first 2M docs of Wikipedia. Test machine is Mac OS X
10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
* I test across multiple queries. 1-X means an OR query, eg 1-4
means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
AND 3 AND 4. "u s" means "united states" (phrase search).
* I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
95, 98, 99, 99.99999 (filter is non-null but all bits are set),
100 (filter=null, control)).
* Method high means I use random-access filter API in
IndexSearcher's main loop. Method low means I use random-access
filter API down in SegmentTermDocs (just like deleted docs
today).
* Baseline (QPS) is current trunk, where filter is applied as iterator up
"high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Issue Comment Edited: (LUCENE-1536) if a filter can
support random access API, we should use it
Posted by John Wang <jo...@gmail.com>.
Maybe I am not understanding the patch. But isn't casting from
Filter.getDocIdSet to OpenBitSet kinda dangerous and assuming Filter
constructing a Bitset something we want to move away from?
-John
On Mon, Apr 20, 2009 at 4:27 PM, Jason Rutherglen (JIRA) <ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700984#action_12700984]
>
> Jason Rutherglen edited comment on LUCENE-1536 at 4/20/09 4:26 PM:
> -------------------------------------------------------------------
>
> Perhaps we can go ahead with this patch given we're not sure how
> to do an optimized version of LUCENE-1518 yet. This patch
> entails passing the RandomAccessFilter to TermScorer, what's a
> good way to do this without rewriting too much of the Lucene API?
>
> * TermQuery.createWeight -> TermWeight.scorer instantiates the
> TermScorer which is where we need to pass in the filter? So we
> could somehow pass the filter in via multiple constructors? I
> didn't see a clean API way though.
>
> * Or we can add a new method to Scorer, something like
> getSequentialSubScorers? Which we then iterate over and if one
> is a TermScorer set the filter(s). This setting of the RAF would
> happen in IndexSearcher.doSearch.
>
> was (Author: jasonrutherglen):
> Perhaps we can go ahead with this patch given we're not sure how
> to do an optimized version of LUCENE-1345 yet. This patch
> entails passing the RandomAccessFilter to TermScorer, what's a
> good way to do this without rewriting too much of the Lucene API?
>
> * TermQuery.createWeight -> TermWeight.scorer instantiates the
> TermScorer which is where we need to pass in the filter? So we
> could somehow pass the filter in via multiple constructors? I
> didn't see a clean API way though.
>
> * Or we can add a new method to Scorer, something like
> getSequentialSubScorers? Which we then iterate over and if one
> is a TermScorer set the filter(s). This setting of the RAF would
> happen in IndexSearcher.doSearch.
>
> > if a filter can support random access API, we should use it
> > -----------------------------------------------------------
> >
> > Key: LUCENE-1536
> > URL: https://issues.apache.org/jira/browse/LUCENE-1536
> > Project: Lucene - Java
> > Issue Type: Improvement
> > Components: Search
> > Affects Versions: 2.4
> > Reporter: Michael McCandless
> > Assignee: Michael McCandless
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1536.patch
> >
> >
> > I ran some performance tests, comparing applying a filter via
> > random-access API instead of current trunk's iterator API.
> > This was inspired by LUCENE-1476, where we realized deletions should
> > really be implemented just like a filter, but then in testing found
> > that switching deletions to iterator was a very sizable performance
> > hit.
> > Some notes on the test:
> > * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> > 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> > * I test across multiple queries. 1-X means an OR query, eg 1-4
> > means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> > AND 3 AND 4. "u s" means "united states" (phrase search).
> > * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> > 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> > 100 (filter=null, control)).
> > * Method high means I use random-access filter API in
> > IndexSearcher's main loop. Method low means I use random-access
> > filter API down in SegmentTermDocs (just like deleted docs
> > today).
> > * Baseline (QPS) is current trunk, where filter is applied as iterator
> up
> > "high" (ie in IndexSearcher's search loop).
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Paul Elschot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679665#action_12679665 ]
Paul Elschot commented on LUCENE-1536:
--------------------------------------
For the skipToButNotNext() did you mean sth like LUCENE-1252 ?
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672005#action_12672005 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
OK I tested a different approach for matching when the filter is
relatively sparse filter (<= 10% of index size), by implementing a
simplistic prototype "random access" scorer API (JumpScorer), which
only exposes the method "boolean jump(int docID)" that returns true if
that doc matches, else false (and no next() under the hood).
I only implemented JumpScorer for pure AND/OR queries (ie no excluded
terms), so I only test for these queries.
I convert the filter to SortedVIntList up front, so iteration is fast.
Then during matching I iterate through each doc in the filter, and ask
the random-access scoring API to test whether it accepts the doc, and
collect it if so.
This only performs better when the filter is sparse relative to the
query. Once we merge Query/Filter, I think this optimization can more
generally be used whenever one sub-query is very restrictive compared
to the rest of the sub-queries.
This is basically the reverse of what I first tested, which was to
take a filter that can support random access API and "distribute" it
down to each TermQuery, which gives very good gains especially when
filter is in the middle of the sparse/dense range. Whereas this test
keeps the iterator API on the filter, but switches to a random access
API on the scorer.
Results:
||%tg Filter||Query||Hits||QPS||QPSNew||%tg change||
|1%|1-2| 5363| 47.4| 66.7| 40.7%|
|2%|1-2| 10675| 37.6| 50.6| 34.6%|
|5%|1-2| 26880| 28.6| 37.0| 29.4%|
|10%|1-2| 53673| 23.8| 26.2| 10.1%|
|1%|1-4| 6544| 26.9| 37.2| 38.3%|
|2%|1-4| 13062| 21.2| 29.2| 37.7%|
|5%|1-4| 32815| 16.1| 21.2| 31.7%|
|10%|1-4| 65491| 13.2| 15.2| 15.2%|
|1%|1-10| 8406| 13.0| 17.6| 35.4%|
|2%|1-10| 16756| 10.2| 14.2| 39.2%|
|5%|1-10| 41937| 7.7| 10.3| 33.8%|
|10%|1-10| 83828| 6.3| 7.8| 23.8%|
|1%|+1-2| 2308| 63.6| 82.9| 30.3%|
|2%|+1-2| 4621| 49.9| 60.7| 21.6%|
|5%|+1-2| 11706| 35.8| 47.6| 33.0%|
|10%|+1-2| 23272| 28.3| 35.5| 25.4%|
|1%|+1-4| 923| 34.4| 58.0| 68.6%|
|2%|+1-4| 1849| 28.5| 44.9| 57.5%|
|5%|+1-4| 4794| 22.0| 33.7| 53.2%|
|10%|+1-4| 9595| 19.8| 25.4| 28.3%|
|1%|+1-10| 292| 17.0| 36.6|115.3%|
|2%|+1-10| 579| 15.2| 30.2| 98.7%|
|5%|+1-10| 1517| 13.5| 22.2| 64.4%|
|10%|+1-10| 2999| 12.4| 17.4| 40.3%|
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705029#action_12705029 ]
Marvin Humphrey commented on LUCENE-1536:
-----------------------------------------
> it would be amajor change because you can only do this if
> you do not need the precise totalHits back.
Early termination (pruning) also messes with totalHits. I think
it would be good to get away from the idea that totalHits is
reliable side effect.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1536) if a filter can support random access
API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1536:
-------------------------------------
Attachment: LUCENE-1536.patch
* Added a IndexReader.termDocs(term, filter) method which
culminates in passing a RandomAccessDocIdSet to the scorers.
ConstantScoreQuery and FilterQuery need to AND the filters
together.
* Does explain need the filter?
* Spans aren't supported yet
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700984#action_12700984 ]
Jason Rutherglen commented on LUCENE-1536:
------------------------------------------
Perhaps we can go ahead with this patch given we're not sure how
to do an optimized version of LUCENE-1345 yet. This patch
entails passing the RandomAccessFilter to TermScorer, what's a
good way to do this without rewriting too much of the Lucene API?
* TermQuery.createWeight -> TermWeight.scorer instantiates the
TermScorer which is where we need to pass in the filter? So we
could somehow pass the filter in via multiple constructors? I
didn't see a clean API way though.
* Or we can add a new method to Scorer, something like
getSequentialSubScorers? Which we then iterate over and if one
is a TermScorer set the filter(s). This setting of the RAF would
happen in IndexSearcher.doSearch.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699573#action_12699573 ]
Uwe Schindler commented on LUCENE-1536:
---------------------------------------
And the switch for different densities:
OpenBitSet could calculate its density in isRandomAccess() and return true or false depending on the density factors above. The search code then would only check initially isRandomAccess() (before starting filtering) and then switch between iterator or random acess api.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705008#action_12705008 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
bq. Has anyone tested if this results in a performance degradation on SegmentTermDocs?
I haven't, and I'm also nervous.
One thing we could do is require a concrete impl (eg OpenBitSet) in order to use this optimization?
I also think we should require pre-multiplication of the deleted docs, and possibly inversion of the filter, rather than doing it on the fly per docID. Also, we shouldn't negate the deleted docs per doc when there's no RAF. We may want/need to switch to OpenBitSet for deleted docs, in order to not make 2 copies of the STD code.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670449#action_12670449 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
Test results:
||%tg Filter||Query||Method||Hits||QPS||QPSNew||%tg change||
|0%|1|low| 0|18992.3| 142.6|-99.2%|
|0%|1|high| 0|18992.3| 109.6|-99.4%|
|1%|1|low| 3863| 133.7| 135.3| 1.2%|
|1%|1|high| 3863| 133.7| 99.7|-25.4%|
|2%|1|low| 7714| 108.2| 133.7| 23.6%|
|2%|1|high| 7714| 108.2| 100.5| -7.1%|
|5%|1|low| 19333| 76.9| 128.6| 67.2%|
|5%|1|high| 19333| 76.9| 97.2| 26.4%|
|10%|1|low| 38673| 62.5| 119.1| 90.6%|
|10%|1|high| 38673| 62.5| 92.0| 47.2%|
|25%|1|low| 96670| 47.3| 102.3|116.3%|
|25%|1|high| 96670| 47.3| 90.5| 91.3%|
|50%|1|low| 193098| 40.0| 85.6|114.0%|
|50%|1|high| 193098| 40.0| 79.6| 99.0%|
|75%|1|low| 289765| 38.0| 82.8|117.9%|
|75%|1|high| 289765| 38.0| 79.0|107.9%|
|90%|1|low| 347762| 37.2| 82.7|122.3%|
|90%|1|high| 347762| 37.2| 72.8| 95.7%|
|95%|1|low| 367102| 36.5| 82.9|127.1%|
|95%|1|high| 367102| 36.5| 73.1|100.3%|
|98%|1|low| 378721| 37.3| 81.8|119.3%|
|98%|1|high| 378721| 37.3| 73.0| 95.7%|
|99%|1|low| 382572| 36.6| 83.3|127.6%|
|99%|1|high| 382572| 36.6| 71.8| 96.2%|
|99.99999%|1|low| 386435| 38.0| 83.8|120.5%|
|99.99999%|1|high| 386435| 38.0| 70.9| 86.6%|
|100%|1|low| 386435| 88.0| 89.1| 1.2%|
|100%|1|high| 386435| 88.0| 89.5| 1.7%|
|0%|1-2|low| 0|18808.1| 71.5|-99.6%|
|0%|1-2|high| 0|18808.1| 22.4|-99.9%|
|1%|1-2|low| 5363| 46.8| 65.2| 39.3%|
|1%|1-2|high| 5363| 46.8| 22.5|-51.9%|
|2%|1-2|low| 10675| 37.6| 61.4| 63.3%|
|2%|1-2|high| 10675| 37.6| 22.4|-40.4%|
|5%|1-2|low| 26880| 28.8| 53.4| 85.4%|
|5%|1-2|high| 26880| 28.8| 22.3|-22.6%|
|10%|1-2|low| 53673| 23.7| 48.1|103.0%|
|10%|1-2|high| 53673| 23.7| 21.8| -8.0%|
|25%|1-2|low| 133988| 19.9| 37.2| 86.9%|
|25%|1-2|high| 133988| 19.9| 21.0| 5.5%|
|50%|1-2|low| 267757| 17.2| 27.4| 59.3%|
|50%|1-2|high| 267757| 17.2| 20.3| 18.0%|
|75%|1-2|low| 401596| 16.9| 23.1| 36.7%|
|75%|1-2|high| 401596| 16.9| 20.3| 20.1%|
|90%|1-2|low| 481911| 17.0| 21.2| 24.7%|
|90%|1-2|high| 481911| 17.0| 20.6| 21.2%|
|95%|1-2|low| 508704| 17.1| 20.7| 21.1%|
|95%|1-2|high| 508704| 17.1| 20.7| 21.1%|
|98%|1-2|low| 524909| 17.3| 20.7| 19.7%|
|98%|1-2|high| 524909| 17.3| 20.7| 19.7%|
|99%|1-2|low| 530221| 17.4| 20.5| 17.8%|
|99%|1-2|high| 530221| 17.4| 20.8| 19.5%|
|99.99999%|1-2|low| 535584| 17.1| 20.3| 18.7%|
|99.99999%|1-2|high| 535584| 17.1| 20.3| 18.7%|
|100%|1-2|low| 535584| 21.0| 20.7| -1.4%|
|100%|1-2|high| 535584| 21.0| 20.8| -1.0%|
|0%|1-4|low| 0|17961.7| 42.2|-99.8%|
|0%|1-4|high| 0|17961.7| 11.9|-99.9%|
|1%|1-4|low| 6544| 27.1| 38.4| 41.7%|
|1%|1-4|high| 6544| 27.1| 12.0|-55.7%|
|2%|1-4|low| 13062| 21.4| 36.0| 68.2%|
|2%|1-4|high| 13062| 21.4| 11.9|-44.4%|
|5%|1-4|low| 32815| 16.1| 31.3| 94.4%|
|5%|1-4|high| 32815| 16.1| 11.8|-26.7%|
|10%|1-4|low| 65491| 13.3| 27.8|109.0%|
|10%|1-4|high| 65491| 13.3| 11.7|-12.0%|
|25%|1-4|low| 163600| 10.5| 21.0|100.0%|
|25%|1-4|high| 163600| 10.5| 11.5| 9.5%|
|50%|1-4|low| 327302| 9.9| 15.3| 54.5%|
|50%|1-4|high| 327302| 9.9| 11.2| 13.1%|
|75%|1-4|low| 490881| 9.7| 12.6| 29.9%|
|75%|1-4|high| 490881| 9.7| 11.1| 14.4%|
|90%|1-4|low| 588990| 9.6| 11.6| 20.8%|
|90%|1-4|high| 588990| 9.6| 11.1| 15.6%|
|95%|1-4|low| 621666| 9.6| 11.3| 17.7%|
|95%|1-4|high| 621666| 9.6| 11.2| 16.7%|
|98%|1-4|low| 641419| 9.4| 11.1| 18.1%|
|98%|1-4|high| 641419| 9.4| 11.2| 19.1%|
|99%|1-4|low| 647937| 9.8| 11.1| 13.3%|
|99%|1-4|high| 647937| 9.8| 11.2| 14.3%|
|99.99999%|1-4|low| 654481| 10.0| 11.1| 11.0%|
|99.99999%|1-4|high| 654481| 10.0| 11.2| 12.0%|
|100%|1-4|low| 654481| 11.3| 11.3| 0.0%|
|100%|1-4|high| 654481| 11.3| 11.2| -0.9%|
|0%|1-10|low| 0|15990.0| 22.5|-99.9%|
|0%|1-10|high| 0|15990.0| 5.8|-100.0%|
|1%|1-10|low| 8406| 13.1| 20.2| 54.2%|
|1%|1-10|high| 8406| 13.1| 5.8|-55.7%|
|2%|1-10|low| 16756| 10.2| 18.9| 85.3%|
|2%|1-10|high| 16756| 10.2| 5.8|-43.1%|
|5%|1-10|low| 41937| 7.7| 16.5|114.3%|
|5%|1-10|high| 41937| 7.7| 5.7|-26.0%|
|10%|1-10|low| 83828| 6.3| 14.5|130.2%|
|10%|1-10|high| 83828| 6.3| 5.7| -9.5%|
|25%|1-10|low| 209328| 5.3| 10.9|105.7%|
|25%|1-10|high| 209328| 5.3| 5.7| 7.5%|
|50%|1-10|low| 418668| 4.9| 7.8| 59.2%|
|50%|1-10|high| 418668| 4.9| 5.5| 12.2%|
|75%|1-10|low| 628338| 4.9| 6.4| 30.6%|
|75%|1-10|high| 628338| 4.9| 5.5| 12.2%|
|90%|1-10|low| 753838| 4.9| 5.9| 20.4%|
|90%|1-10|high| 753838| 4.9| 5.5| 12.2%|
|95%|1-10|low| 795729| 5.0| 5.7| 14.0%|
|95%|1-10|high| 795729| 5.0| 5.6| 12.0%|
|98%|1-10|low| 820910| 5.1| 5.6| 9.8%|
|98%|1-10|high| 820910| 5.1| 5.6| 9.8%|
|99%|1-10|low| 829260| 5.1| 5.6| 9.8%|
|99%|1-10|high| 829260| 5.1| 5.6| 9.8%|
|99.99999%|1-10|low| 837666| 5.2| 5.6| 7.7%|
|99.99999%|1-10|high| 837666| 5.2| 5.6| 7.7%|
|100%|1-10|low| 837666| 5.7| 5.7| 0.0%|
|100%|1-10|high| 837666| 5.7| 5.6| -1.8%|
|0%|+1-2|low| 0|18848.4| 138.4|-99.3%|
|0%|+1-2|high| 0|18848.4| 27.4|-99.9%|
|1%|+1-2|low| 2308| 63.1| 77.0| 22.0%|
|1%|+1-2|high| 2308| 63.1| 27.1|-57.1%|
|2%|+1-2|low| 4621| 50.3| 69.7| 38.6%|
|2%|+1-2|high| 4621| 50.3| 26.8|-46.7%|
|5%|+1-2|low| 11706| 36.1| 56.9| 57.6%|
|5%|+1-2|high| 11706| 36.1| 26.5|-26.6%|
|10%|+1-2|low| 23272| 28.4| 48.6| 71.1%|
|10%|+1-2|high| 23272| 28.4| 26.2| -7.7%|
|25%|+1-2|low| 58401| 23.7| 36.4| 53.6%|
|25%|+1-2|high| 58401| 23.7| 24.9| 5.1%|
|50%|+1-2|low| 117083| 20.9| 28.2| 34.9%|
|50%|+1-2|high| 117083| 20.9| 23.9| 14.4%|
|75%|+1-2|low| 176233| 19.3| 24.4| 26.4%|
|75%|+1-2|high| 176233| 19.3| 22.8| 18.1%|
|90%|+1-2|low| 211362| 18.6| 22.9| 23.1%|
|90%|+1-2|high| 211362| 18.6| 22.5| 21.0%|
|95%|+1-2|low| 222928| 18.5| 22.5| 21.6%|
|95%|+1-2|high| 222928| 18.5| 22.5| 21.6%|
|98%|+1-2|low| 230013| 18.3| 22.0| 20.2%|
|98%|+1-2|high| 230013| 18.3| 22.4| 22.4%|
|99%|+1-2|low| 232326| 18.3| 22.1| 20.8%|
|99%|+1-2|high| 232326| 18.3| 22.3| 21.9%|
|99.99999%|+1-2|low| 234634| 17.8| 21.9| 23.0%|
|99.99999%|+1-2|high| 234634| 17.8| 22.2| 24.7%|
|100%|+1-2|low| 234634| 22.9| 22.7| -0.9%|
|100%|+1-2|high| 234634| 22.9| 22.6| -1.3%|
|0%|+1-4|low| 0|17987.0| 137.9|-99.2%|
|0%|+1-4|high| 0|17987.0| 18.0|-99.9%|
|1%|+1-4|low| 923| 34.5| 58.4| 69.3%|
|1%|+1-4|high| 923| 34.5| 17.9|-48.1%|
|2%|+1-4|low| 1849| 28.7| 51.9| 80.8%|
|2%|+1-4|high| 1849| 28.7| 17.9|-37.6%|
|5%|+1-4|low| 4794| 22.1| 39.0| 76.5%|
|5%|+1-4|high| 4794| 22.1| 17.8|-19.5%|
|10%|+1-4|low| 9595| 19.9| 35.3| 77.4%|
|10%|+1-4|high| 9595| 19.9| 17.5|-12.1%|
|25%|+1-4|low| 24136| 17.3| 25.7| 48.6%|
|25%|+1-4|high| 24136| 17.3| 17.2| -0.6%|
|50%|+1-4|low| 48328| 16.1| 19.5| 21.1%|
|50%|+1-4|high| 48328| 16.1| 16.8| 4.3%|
|75%|+1-4|low| 72718| 15.8| 17.0| 7.6%|
|75%|+1-4|high| 72718| 15.8| 16.6| 5.1%|
|90%|+1-4|low| 87259| 15.3| 16.3| 6.5%|
|90%|+1-4|high| 87259| 15.3| 16.4| 7.2%|
|95%|+1-4|low| 92060| 15.4| 16.0| 3.9%|
|95%|+1-4|high| 92060| 15.4| 16.4| 6.5%|
|98%|+1-4|low| 95005| 15.4| 15.8| 2.6%|
|98%|+1-4|high| 95005| 15.4| 16.4| 6.5%|
|99%|+1-4|low| 95931| 15.1| 15.7| 4.0%|
|99%|+1-4|high| 95931| 15.1| 16.4| 8.6%|
|99.99999%|+1-4|low| 96854| 14.3| 15.9| 11.2%|
|99.99999%|+1-4|high| 96854| 14.3| 16.2| 13.3%|
|100%|+1-4|low| 96854| 16.6| 16.6| 0.0%|
|100%|+1-4|high| 96854| 16.6| 16.6| 0.0%|
|0%|"u s"|low| 0|19123.0| 124.9|-99.3%|
|0%|"u s"|high| 0|19123.0| 7.0|-100.0%|
|1%|"u s"|low| 3192| 23.5| 27.3| 16.2%|
|1%|"u s"|high| 3192| 23.5| 7.1|-69.8%|
|2%|"u s"|low| 6179| 17.8| 24.3| 36.5%|
|2%|"u s"|high| 6179| 17.8| 7.0|-60.7%|
|5%|"u s"|low| 15446| 12.7| 20.3| 59.8%|
|5%|"u s"|high| 15446| 12.7| 7.0|-44.9%|
|10%|"u s"|low| 30858| 10.1| 16.1| 59.4%|
|10%|"u s"|high| 30858| 10.1| 6.8|-32.7%|
|25%|"u s"|low| 77138| 7.7| 13.0| 68.8%|
|25%|"u s"|high| 77138| 7.7| 6.8|-11.7%|
|50%|"u s"|low| 154331| 6.7| 9.9| 47.8%|
|50%|"u s"|high| 154331| 6.7| 7.0| 4.5%|
|75%|"u s"|low| 231412| 6.3| 8.4| 33.3%|
|75%|"u s"|high| 231412| 6.3| 7.0| 11.1%|
|90%|"u s"|low| 277692| 5.8| 7.2| 24.1%|
|90%|"u s"|high| 277692| 5.8| 7.0| 20.7%|
|95%|"u s"|low| 293104| 5.8| 7.1| 22.4%|
|95%|"u s"|high| 293104| 5.8| 7.0| 20.7%|
|98%|"u s"|low| 302371| 5.8| 7.0| 20.7%|
|98%|"u s"|high| 302371| 5.8| 6.9| 19.0%|
|99%|"u s"|low| 305358| 5.8| 6.9| 19.0%|
|99%|"u s"|high| 305358| 5.8| 6.9| 19.0%|
|99.99999%|"u s"|low| 308550| 5.8| 6.8| 17.2%|
|99.99999%|"u s"|high| 308550| 5.8| 6.9| 19.0%|
|100%|"u s"|low| 308550| 7.0| 6.9| -1.4%|
|100%|"u s"|high| 308550| 7.0| 6.9| -1.4%|
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854612#action_12854612 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
With flex, you can now get the deleted docs from a reader (returns a Bits, yet another interface for bitsets), and, Docs/AndPositionsEnums require that you pass in the skipDocs. Ie they no longer skip deleted docs by default.
I think this makes this issue quite a bit simpler? EG OpenBitSets implements Bits (as of flex landing), so... we just need a way to pass this Bits down to the low level scorers that actually pull a postings list. But, we should only do this if the filter is not sparse.
Also: the filter must be inverted, and, ORd with the deleted docs.
This can result in enormous perf gains for searches doing filtering where the filter is relatively static (can be cached & shared across searches).
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699571#action_12699571 ]
Uwe Schindler commented on LUCENE-1536:
---------------------------------------
The empty docidset instance should *not* be random access :), so the only change would affect OpenBitSet to overwrite these two new methods from the default abstract class:
{code}
boolean isRandomAccess() { return true; }
boolean acceptDoc(int docid) { return get(docid); /* possibly inlined */ }
{code}
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699680#action_12699680 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
OK, if we do choose to couple, maybe we should name it "useRandomAccess()"?
Another filter optimization that'd be nice to get in is to somehow "know" that a filter has pre-incorporated deleted documents. This way, once we have a solution for the "push filter down to all TermScorers", we could have it only check the filter and not also deleted docs. (This is one of the optimizations in LUCENE-1594).
We might eventually want/need some sort of external FilterManager that would handle this (ie, convert a filter to sparse vs random-access as appropriate, multiply in deleted docs, handle caching, etc).
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705013#action_12705013 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
I don't think it should be the caller's job to getSequentialSubScorers
and push down a RAF? Rather, I think when requesting a scorer we
should pass in a RAF, requiring that the returned scorer factors it
in (passing on to its own sub-scorers if needed).
bq. One thing we could do is require a concrete impl (eg OpenBitSet) in order to use this optimization?
This may be too restrictive, because another use case (touched on
already in LUCENE-1593, but actually much more similar to this issue)
is when sorting by field.
EG say we are sorting by int ascending, and from the
FieldValueHitQueue we know the bottom value is 17. Then, within
TermScorer if we see a docID we should check if its value is greater
than 17 and skip it if so.
Very likely this will be a sizable performance gain, but it would be a
major change because you can only do this if you do not need the
precise totalHits back.
So... maybe we need to allow an abstract RandomAccesDocIdSet, to allow
this use case. But perhaps we should negate its API? Ie it exposes
"boolean reject(int docID)".
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699675#action_12699675 ]
Uwe Schindler commented on LUCENE-1536:
---------------------------------------
I coupled the density check inside the OpenBitSet, because the internals of OpenBitset are responsible for determining how fast a sequential vs. random approach is. Maybe someone invents an new hyper-bitset that can faster do sequential accesses even in sparingly filled bitsets (e.g. fragmented bitset, bitset with RDBMS-like "index"). In this case, it has the responsibility to say: if density is between this and this i would use sequential.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1536) if a filter can support random access
API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1536:
-------------------------------------
Attachment: LUCENE-1536.patch
The patch is a start at something that will work within the API
(I think).
* Created a RandomAccessDocIdSet class that BitVector and
OpenBitSet implement.
* SegmentTermDocs has a setter for RandomAccessDocIdSet with the
option of including the deletedDocs
* IndexSearcher.doSearch does the main work of setting the
RandomAccessDocIdSet on the TermScorers
* BooleanScorer2 and other classes implement
getSequentialSubScorers
* AndRandomAccessDocIdSet iterates over an array of
RandomAccessDocIdSets
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704590#action_12704590 ]
Yonik Seeley commented on LUCENE-1536:
--------------------------------------
Interesting stuff!
Has anyone tested if this results in a performance degradation on SegmentTermDocs?
This is very inner loop stuff, and it's replacing a "non virtual" BitVector.get() which can be easily inlined with two dispatches through base classes. Hopefully hotspot could handle it, but it's tough to figure out, esp in a real system where sometimes a user RAF will be used and sometimes not.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1536) if a filter can support random access
API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1536:
---------------------------------------
Attachment: LUCENE-1536.patch
Attaching patch I'm using to run tests. NOTE: this is nowhere near
committable. It's just a hack to allow testing the different ways of
applying filters.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679690#action_12679690 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
Ahh, that's actually something different but also a neat idea to
explore.
I want a way to skipTo(docID) without having it then internally do a
next() if the docID was not accepted. Basically a random-access
"accepts(int docID)" API, that's called only on increasing docIDs.
Implementing "accepts" for queries is often alot simpler than
implementing next/skipTo.
LUCENE-1252 wants a way to expose access to the two constraints within
a single query separately. EG a phrase search 1) must have all N
terms, and 2) must have them in the right positions. But if you could
check only 1), and if it passes next check the filter on the search,
and if it still passes go back and check 2), then that could give
better search performance.
I think there's decent room for improving search performance of
complex queries.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1536) if a filter can support random access
API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1536:
-------------------------------------
Attachment: LUCENE-1536.patch
* Filter has a parameter for included deletes. IndexSearcher
uses it for setting the RandomAccessDocIdSet on the Scorers
* MatchAllDocsScorer in addition to TermScorer properly supports
setting a RandomAccessDocIdSet
* Added more test cases
* AndRandomAccessDocIdSet.iterator() and BitVector.iterator()
needs to be implemented
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670452#action_12670452 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
It's a ridiculous amount of data to digest, but here are some initial
observations/thoughts:
* There are very sizable gains here by switching to random-access
low (ie, handling top-level filter the way we now handle deletes).
I'm especially interested in gains in the slowest queries. EG the
phrase query "united states" sees QPS gains from 16%-69%. The
10-clause OR query 1-10 sees QPS gains between 8% and 130%.
* Results are consistent with LUCENE-1476: random-access low gives
the best performance when filter density is >= 1%.
* High is worse than trunk up until ~25% density, which makes sense
since we are asking Scorer to do alot of work producing docIDs
that we then nix with the filter.
* Low is consistently better than high, though as filter density
gets higher the gap between them narrows. I'll drop high from
future tests.
* The gains are generally strongest in the "moderate" density range,
5-25%.
* The degenerate 0% case is clearly far far worse, which is expected
since the iterator scans the bits, finds none set, and quickly
ends the search. For very low density filters we should continue
to use iterator.
* The "control" 100% case (where filter is null) is about the same,
which is expected.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-1536) if a filter can support
random access API, we should use it
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699571#action_12699571 ]
Uwe Schindler edited comment on LUCENE-1536 at 4/16/09 2:27 AM:
----------------------------------------------------------------
The empty docidset instance should *not* be random access :), so the only change would affect OpenBitSet to overwrite these two new methods from the default abstract class:
{code}
boolean isRandomAccess() { return true; }
boolean acceptDoc(int docid) { return fastGet(docid); /* possibly inlined */ }
{code}
was (Author: thetaphi):
The empty docidset instance should *not* be random access :), so the only change would affect OpenBitSet to overwrite these two new methods from the default abstract class:
{code}
boolean isRandomAccess() { return true; }
boolean acceptDoc(int docid) { return get(docid); /* possibly inlined */ }
{code}
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1536) if a filter can support random access
API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1536:
-------------------------------------
Attachment: LUCENE-1536.patch
* The test case includes testConstantCoreQuery
* If LUCENE-1632 were updated to use the new DocIdSetIterator API,
it could be useful for the iterators of AndRandomAccessDocIdSet
and NotRandomAccessDocIdSet.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1536) if a filter can support random access
API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1536:
---------------------------------------
Fix Version/s: (was: 2.9)
3.1
Moving out.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699939#action_12699939 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
Ahh right, we should re-test performance of this after LUCENE-1518 is done.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699669#action_12699669 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
I like this approach!
But should we somehow decouple the density check vs the is random access check? Ie, isRandomAccess should return true or false based on the underlying datastructure. Then, somehow, I think the search code should determine whether a given docIdSet should be randomly accessed vs iterated? (I'm not sure how yet!)
Also, we somehow need the mechanism to "denormalize" the application of the filter from top to bottom, ie, each leaf TermQuery involved in the full query needs to know to apply the random access filter just like it applies deletes.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699558#action_12699558 ]
Uwe Schindler commented on LUCENE-1536:
---------------------------------------
How about DocIdSet adds a
{code}
boolean isRandomAccess() { return false; }
{code}
That is implemented to return false in the default abstract class for backwards compatibility.
If a DocIdSet is random access (backed by OpenBitSet or is the empty iterator), isRandomAccess() is overridden to return true and an additional method in DocIdSet is implemented, the default would be:
{code}
boolean acceptDoc(int docid) { throw new UnsupportedOperationException(); }
{code}
Both changes are backwards compatible, but filters using OpenBitSet would automatically be random access and support acceptDoc().
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705012#action_12705012 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
I don't think it should be the caller's job to getSequentialSubScorers
and push down a RAF? Rather, I think when requesting a scorer we
should pass in a RAF, requiring that the returned scorer factors it
in (passing on to its own sub-scorers if needed).
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722761#action_12722761 ]
Jason Rutherglen commented on LUCENE-1536:
------------------------------------------
{quote} since QueryWeight is new in 2.9, we are free to make
this an abstract method, and put the default only in
QueryWeightWrapper, {quote}
Sounds good.
{quote} we'd also want control on whether the Scorer should
apply deletes or not {quote}
Filter.hasDeletes can be used know whether to apply deletes or
not.
{quote}migrating to with LUCENE-1630{quote}
Looks this this will be committed soon, so I'll wait for that
then make the changes, then benchmark.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-1536) if a filter can support
random access API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700984#action_12700984 ]
Jason Rutherglen edited comment on LUCENE-1536 at 4/20/09 4:26 PM:
-------------------------------------------------------------------
Perhaps we can go ahead with this patch given we're not sure how
to do an optimized version of LUCENE-1518 yet. This patch
entails passing the RandomAccessFilter to TermScorer, what's a
good way to do this without rewriting too much of the Lucene API?
* TermQuery.createWeight -> TermWeight.scorer instantiates the
TermScorer which is where we need to pass in the filter? So we
could somehow pass the filter in via multiple constructors? I
didn't see a clean API way though.
* Or we can add a new method to Scorer, something like
getSequentialSubScorers? Which we then iterate over and if one
is a TermScorer set the filter(s). This setting of the RAF would
happen in IndexSearcher.doSearch.
was (Author: jasonrutherglen):
Perhaps we can go ahead with this patch given we're not sure how
to do an optimized version of LUCENE-1345 yet. This patch
entails passing the RandomAccessFilter to TermScorer, what's a
good way to do this without rewriting too much of the Lucene API?
* TermQuery.createWeight -> TermWeight.scorer instantiates the
TermScorer which is where we need to pass in the filter? So we
could somehow pass the filter in via multiple constructors? I
didn't see a clean API way though.
* Or we can add a new method to Scorer, something like
getSequentialSubScorers? Which we then iterate over and if one
is a TermScorer set the filter(s). This setting of the RAF would
happen in IndexSearcher.doSearch.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722538#action_12722538 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
bq. that requires adding a scorer(IndexReader, RAF) method to the Weight interface?
Well, to QueryWeight (abstract class, not interface) that we are migrating to with LUCENE-1630. We'd make a default impl, probably a static method somewhere, so that any external QueryWeight's "out there" would just work.
Though, since QueryWeight is new in 2.9, we are free to make this an abstract method, and put the default only in QueryWeightWrapper, if we could get this issue done for 2.9.
I think we'd also want control on whether the Scorer should apply deletes or not, so that for filters that are often re-used, we have the option to pre-fold deletes in.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722472#action_12722472 ]
Jason Rutherglen commented on LUCENE-1536:
------------------------------------------
{quote} I don't think it should be the caller's job to
getSequentialSubScorers and push down a RAF? Rather, I think
when requesting a scorer we should pass in a RAF, requiring that
the returned scorer factors it in (passing on to its own
sub-scorers if needed). {quote}
True, however that requires adding a scorer(IndexReader, RAF)
method to the Weight interface? Which means adding it to I think
20 classes or so. Which is fine, however is that definitely what
we want to do?
Also I forget if we created a patch or benchmarked AND NOT
deleteDocs?
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699892#action_12699892 ]
Jason Rutherglen commented on LUCENE-1536:
------------------------------------------
I thought we are going to get LUCENE-1518 working to compare the performance against passing the filter into TermDocs?
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1536) if a filter can support random access
API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1536:
---------------------------------------
Fix Version/s: 2.9
Given the sizable performance gains, I think we should try to do this for 2.9, if possible.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705257#action_12705257 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
{quote}
Early termination (pruning) also messes with totalHits. I think
it would be good to get away from the idea that totalHits is
reliable side effect.
{quote}
Agreed. Does KS/Lucy not guarantee accurate totalHits returned? Lucene today always returns the precise total hits.
If we can move away from that (optionally) then we can gain sizable performance.
But: we'd then presumably need to return an approx hit count.
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1536) if a filter can support random
access API, we should use it
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670455#action_12670455 ]
Michael McCandless commented on LUCENE-1536:
--------------------------------------------
Some ideas / further things to explore:
* Deletions, top-level filters, and BooleanQuery that "factors" to a
toplevel AND really should be handled by the same code.
* Even an AND'd filter on a sub-clause of a BooleanQuery can be
pushed down to the TermDocs under that tree.
* That common code should send a top-level filter down to the lowest
level, used by random access API, if the filter supports random
access (not all do) and it's not super sparse.
* I think one thing slowing down trunk is the lack of a
Scorer.skipToButNotNext API. We now ask the filter for its
next(), which gives us a filterDocID. Then we call
Scorer.skipTo(filterDocID). If the scorer does not match that
filterDocID, it internally does next(), which for an expensive
scorer is alot of likely wasted work: it advances to a docID that
the filter may not accept. If we had a "skipToButNotNext" API we
could avoid that wasted work. I'm curious what gains this change
alone would provide.
* I'm thinking (but haven't tested this) if the filter is relatively
sparse compared to the other iterators, it'd be better to convert
it to a sparse repr (eg SortedVIntList) and drive the search by
iteration through the filter, after fixing the above skipTo issue.
Maybe a "low iterator" access.
* We may need a "filter optimizer" utility class somewhere, somehow.
For filters you do not plan to re-use, you would not bother with
this. But for filters that will be re-used, you should 1) convert
them to sparse or non-sparse repr depending on their density, 2)
maybe invert them and make sparse if they are close to 100%
density, 3) maybe factor in deletions to the filter so there is
only a single top-level filter to apply.
* I'm not yet sure how to make this change cleanly to the APIs...
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.4
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
> * Index is first 2M docs of Wikipedia. Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
> * I test across multiple queries. 1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4. "u s" means "united states" (phrase search).
> * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.99999 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
> * Method high means I use random-access filter API in
> IndexSearcher's main loop. Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
> * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org