You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/05/12 05:13:41 UTC
[jira] Updated: (LUCENE-2410) Optimize PhraseQuery
[ https://issues.apache.org/jira/browse/LUCENE-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2410:
--------------------------------
Attachment: LUCENE-2410_rewrite.patch
just doing the easy part here, here's the rewrite patch.
I checked, MultiPhraseQuery already has it.
> Optimize PhraseQuery
> --------------------
>
> Key: LUCENE-2410
> URL: https://issues.apache.org/jira/browse/LUCENE-2410
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2410_rewrite.patch
>
>
> Looking the scorers for PhraseQuery, I think there are some speedups
> we could do:
> * The AND part of the scorer (which advances to the next doc that
> has all the terms), in PhraseScorer.doNext, should do the same
> optimizing as BooleanQuery's ConjunctionScorer, ie sort terms from
> rarest to most frequent. I don't think it should use a linked
> list/firstToLast() that it does today.
> * We do way too much work now when .score() is not called, because
> we go and find all occurrences of the phrase in the doc, whereas
> we should stop only after finding the first and then go and count
> the rest if .score() is called.
> * For the exact case, I think we can use two int arrays to find the
> matches. The first array holds the count of how many times a term
> in the phrase "matched" a phrase starting at that position. When
> that count == the number of terms in the phrase, it's a match.
> The 2nd is a "gen" array (holds docID when that count was last
> touched), to avoid clearing. Ie when incrementing the count, if
> the docID != gen, we reset count to 0. I think this'd be faster
> than the PQ we now use. Downside of this is if you have immense
> docs (position gets very large) we'd need 2 immense arrays.
> It'd be great to do LUCENE-1252 along with this, ie factor
> PhraseScorer into two AND'd sub-scorers (LUCENE-1252 is open for
> this). The first one should be ConjunctionScorer, and the 2nd one
> checks the positions (ie, either the exact or sloppy scorers). This
> would mean if the PhraseQuery is AND'd w/ other clauses (or, a filter
> is applied) we would save CPU by not checking the positions for a doc
> unless all other AND'd clauses accepted the doc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org