You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "benwtrent (via GitHub)" <gi...@apache.org> on 2023/01/30 16:55:17 UTC
[GitHub] [lucene] benwtrent opened a new pull request, #12118: Add `FeatureField` weight caching and non-scoring iterators
benwtrent opened a new pull request, #12118:
URL: https://github.com/apache/lucene/pull/12118
While `FeatureField` is a powerful tool in the scoring case, there are scenarios when caching should be allowed and scoring disabled.
A particular case is when the `FeatureField` is used in conjunction with learned-sparse retrieval. It is useful to iterate and calculate the entire matching doc set when combined with various other queries.
related to: https://github.com/apache/lucene/issues/11799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] benwtrent commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410445640
I like that @rmuir! Its keeps the nice API for FeatureFields and removes code duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410418901
So just rewrite it to a TermWeight in createWeight if scores are not needed? No need to duplicate the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410532359
does it make sense? From my perspective the reason to use `FeatureField` is for the WAND-skipping. So if you ask for it not to do scoring, it can't skip, and it defeats the entire purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] benwtrent commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410402325
@rmuir
> i don't understand this issue. The only purpose of this query is for scoring. If you don't want scores, drop the clause completely.
A `FeatureField` provides a useful extension point for learned-sparse retrieval models (see linked issue). These models provide multiple `feature` and `score` pairs. These fields will likely match relevant documents that are not previously matched by other means.
A perfectly valid (and powerful) query would be `BooleanQuery` with multiple `SHOULD` clauses containing `FeatureQuery` objects (obviously, with minimum should match > 0). Note that no other field is being queried. Dropping all those clauses would be a `match_all` and not accurately reflect the matching document set.
Being able to iterate the entire recall set for matching multiple `FeatureField` values will provide useful insight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410426974
example pseudocode:
```
@Override
public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)
throws IOException {
if (!scoreMode.needsScores()) {
// if scores are not needed, let TermWeight deal with optimizing that case.
TermQuery tq = new TermQuery(new Term(fieldName, featureName));
return searcher
.rewrite(tq)
.createWeight(searcher, ScoreMode.COMPLETE_NO_SCORES, boost);
}
...
}
```
This would ensure that it gets all the optimizations of TermQuery and keep the code simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410736154
For the record this need comes from implementing sparse retrieval similarly to what's discussed at #11799, so `FeatureField` no longer stores features but regular terms here. One option is to reuse `FeatureField` for this. Another option could be to reuse `TermQuery` by configuring the `Similarity`'s `SimScorer` to properly decode the frequency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410447591
stolen from SynonymQuery lol. and not sure about why it doesn't pass ScoreMode straight thru and instead hardcodes COMPLETE_NO_SCORES, seems wrong. but you got the idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410515203
I'm ok with changes but i still don't understand the use-case. Pulling all documents containing features, then calculating your own score throws away all the efficiency of FeatureField (e.g. early termination) and will be way too slow as the worst-case is scoring `O(maxdoc)` which could be e.g. a billion.
It would be better to look at `Rescorer` api, e.g. keep the scores ON for the FeatureField, but pull top 500 or 1000 and re-rank those with anything fancy. It keeps everything fast and bounded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1410148671
i don't understand this issue. The only purpose of this query is for scoring. If you don't want scores, drop the clause completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] benwtrent merged pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent merged PR #12118:
URL: https://github.com/apache/lucene/pull/12118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12118: Add `FeatureQuery` weight caching in non-scoring case
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12118:
URL: https://github.com/apache/lucene/pull/12118#issuecomment-1413950962
> For the record this need comes from implementing sparse retrieval similarly to what's discussed at #11799, so `FeatureField` no longer stores features but regular terms here. One option is to reuse `FeatureField` for this. Another option could be to reuse `TermQuery` by configuring the `Similarity`'s `SimScorer` to properly decode the frequency.
I think this is where my confusion is. Hopefully sparse retrieval is implemented in a way that doesn't require this codepath and still takes advantage of WAND-skipping.
But I think this change is OK for the use-case of faceting: if you are going to put FeatureQuery in a required clause for some reason (to me this is unrelated to any sparse retrieval, an app may just want a feature to be required for a particular search), then to support facets that reflect that, you need to disable scores to count correctly. Maybe add a comment above it in the code, something like `// e.g. for faceting` so that it doesn't cause confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org