You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/06/13 13:37:35 UTC
[GitHub] [solr] eliaporciani opened a new pull request, #904: SOLR-16246: Introduced pre-filtering in KnnQParser
eliaporciani opened a new pull request, #904:
URL: https://github.com/apache/solr/pull/904
Implementation of pre-filtering in KnnQParser.
https://issues.apache.org/jira/browse/SOLR-16246
# Description
With this change, it is possible to use filter queries alongside knnQParser:
KnnSearch searches for the topK documents that have passed the filter.
# Solution
With this change, it is possible to use filter queries for pre-filtering documents in knnQParser:
KnnSearch searches for the topK documents that have passed the filter.
The implementation leverages the new features added with https://issues.apache.org/jira/browse/LUCENE-10382.
Before this change, the results of the filter query were intersected with the result of the topK vector retrieval.
# Tests
Changed tests in KnnQParserTest class in charge of checking the correct behavior of filter queries.
# Checklist
Please review the following and check all that apply:
- [v] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [v] I have created a Jira issue and added the issue ID to my pull request title.
- [v] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [v] I have developed this patch against the `main` branch.
- [v] I have run `./gradlew check`.
- [v] I have added tests for my changes.
- [v] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r905195851
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
+ .map(
+ filterQuery ->
+ denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, filterQuery))
+ .orElse(
+ denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK));
+ }
+
+ private Optional<? extends Query> getFilterQuery() throws SolrException {
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] alessandrobenedetti merged pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
alessandrobenedetti merged PR #904:
URL: https://github.com/apache/solr/pull/904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r911773738
##########
solr/core/src/java/org/apache/solr/search/QueryUtils.java:
##########
@@ -215,4 +221,38 @@ public static Query combineQueryAndFilter(Query scoreQuery, Query filterQuery) {
}
}
}
+
+ /**
+ * Parse the filter queries in Solr request
+ *
+ * @param req Solr request
+ * @param fixNegativeQueries if true, negative queries are rewritten by adding a MatchAllDocs
+ * query clause
+ * @return and array of Query. If the request does not contain filter queries, returns an empty
+ * list.
+ * @throws SyntaxError if an error occurs during parsing
+ */
+ public static List<Query> parseFilterQueries(SolrQueryRequest req, boolean fixNegativeQueries)
+ throws SyntaxError {
+
+ String[] filterQueriesStr = req.getParams().getParams(CommonParams.FQ);
+
+ if (filterQueriesStr != null) {
+ List<Query> filters = new ArrayList<>(filterQueriesStr.length);
+ for (String fq : filterQueriesStr) {
+ if (fq != null && fq.trim().length() != 0) {
+ QParser fqp = QParser.getParser(fq, req);
+ fqp.setIsFilter(true);
Review Comment:
I think that adding this kind of parameter might make the method more confused and would discourage future usage.
The method parseFilterQueries aims to parse filter queries and setIsFilter(true) is correct in this scenario. Moreover, setIsFilter(true) is not harmful as it is needed for optimizing the query execution for queries that do not need scoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r905196136
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
Review Comment:
Removed te if condition as you suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] cpoerschke commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
cpoerschke commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r900359934
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
+ .map(
+ filterQuery ->
+ denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, filterQuery))
+ .orElse(
+ denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK));
+ }
+
+ private Optional<? extends Query> getFilterQuery() throws SolrException {
+ if (!isFilter()) {
+ String[] filterQueries = req.getParams().getParams(CommonParams.FQ);
+ return ofNullable(filterQueries).filter(fq -> fq.length != 0).map(this::computeFilterQuery);
+ }
+ return Optional.empty();
+ }
+
+ private Query computeFilterQuery(String[] filterQueries) throws SolrException {
+
+ List<Query> filters = new ArrayList<>(filterQueries.length);
+
+ for (String filterQuery : filterQueries) {
+ if (filterQuery != null && filterQuery.trim().length() != 0) {
+ QParser filterParser;
+ try {
+ filterParser = QParser.getParser(filterQuery, super.getReq());
Review Comment:
subjective: https://github.com/apache/solr/blob/releases/solr/9.0.0/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L199 and other places have similar code, perhaps a utility method could be factored out somehow to reduce code duplication e.g. in https://github.com/apache/solr/blob/releases/solr/9.0.0/solr/core/src/java/org/apache/solr/search/QueryUtils.java something like `public static List<Query> parseFilterQueries(String[] filterQueries, SolrQueryRequest req)` or so.
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
+ .map(
+ filterQuery ->
+ denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, filterQuery))
+ .orElse(
+ denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK));
+ }
+
+ private Optional<? extends Query> getFilterQuery() throws SolrException {
+ if (!isFilter()) {
+ String[] filterQueries = req.getParams().getParams(CommonParams.FQ);
+ return ofNullable(filterQueries).filter(fq -> fq.length != 0).map(this::computeFilterQuery);
+ }
+ return Optional.empty();
+ }
+
+ private Query computeFilterQuery(String[] filterQueries) throws SolrException {
+
+ List<Query> filters = new ArrayList<>(filterQueries.length);
+
+ for (String filterQuery : filterQueries) {
+ if (filterQuery != null && filterQuery.trim().length() != 0) {
+ QParser filterParser;
+ try {
+ filterParser = QParser.getParser(filterQuery, super.getReq());
+ filterParser.setIsFilter(true);
+ filters.add(filterParser.getQuery());
+ } catch (SyntaxError e) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "error in parsing filter queries:" + e.getMessage());
+ }
+ }
+ }
+
+ BooleanQuery.Builder builder = new BooleanQuery.Builder();
+ for (Query query : filters) {
+ builder.add(query, BooleanClause.Occur.FILTER);
+ }
+
+ return builder.build();
Review Comment:
If `filters` contained only one element, could that be returned directly?
##########
solr/core/src/java/org/apache/solr/schema/DenseVectorField.java:
##########
@@ -276,6 +276,11 @@ public Query getKnnVectorQuery(String fieldName, float[] vectorToSearch, int top
return new KnnVectorQuery(fieldName, vectorToSearch, topK);
}
+ public Query getKnnVectorQuery(
Review Comment:
Maybe keep the original one for at least one release e.g. mark it as deprecated now for 9.1 and remove in a future 9.2 or higher release with backwards compatibility in mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r905114621
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
+ .map(
+ filterQuery ->
+ denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, filterQuery))
+ .orElse(
+ denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK));
+ }
+
+ private Optional<? extends Query> getFilterQuery() throws SolrException {
+ if (!isFilter()) {
+ String[] filterQueries = req.getParams().getParams(CommonParams.FQ);
+ return ofNullable(filterQueries).filter(fq -> fq.length != 0).map(this::computeFilterQuery);
+ }
+ return Optional.empty();
+ }
+
+ private Query computeFilterQuery(String[] filterQueries) throws SolrException {
+
+ List<Query> filters = new ArrayList<>(filterQueries.length);
+
+ for (String filterQuery : filterQueries) {
+ if (filterQuery != null && filterQuery.trim().length() != 0) {
+ QParser filterParser;
+ try {
+ filterParser = QParser.getParser(filterQuery, super.getReq());
+ filterParser.setIsFilter(true);
+ filters.add(filterParser.getQuery());
+ } catch (SyntaxError e) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "error in parsing filter queries:" + e.getMessage());
+ }
+ }
+ }
+
+ BooleanQuery.Builder builder = new BooleanQuery.Builder();
+ for (Query query : filters) {
+ builder.add(query, BooleanClause.Occur.FILTER);
+ }
+
+ return builder.build();
Review Comment:
Good point. I changed the code in order to return it direcly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r903975160
##########
solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java:
##########
@@ -53,18 +53,7 @@
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.schema.IndexSchema;
import org.apache.solr.schema.SchemaField;
-import org.apache.solr.search.DocIterator;
-import org.apache.solr.search.DocList;
-import org.apache.solr.search.DocListAndSet;
-import org.apache.solr.search.QParser;
-import org.apache.solr.search.QParserPlugin;
-import org.apache.solr.search.QueryParsing;
-import org.apache.solr.search.ReturnFields;
-import org.apache.solr.search.SolrIndexSearcher;
-import org.apache.solr.search.SolrQueryTimeoutImpl;
-import org.apache.solr.search.SolrReturnFields;
-import org.apache.solr.search.SortSpec;
-import org.apache.solr.search.SyntaxError;
+import org.apache.solr.search.*;
Review Comment:
As a project we prefer to list all dependencies instead of using the `*`. Could you revert this for this file and the other that has the same change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] alessandrobenedetti commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
alessandrobenedetti commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r914920701
##########
solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc:
##########
@@ -279,13 +279,12 @@ The `knn` query parser can be used with filter queries:
[IMPORTANT]
====
-When using `knn` in these scenarios make sure you have a clear understanding of how filter queries work in Apache Solr:
-The Ranked List of document IDs resulting from the main query `q` is intersected with the set of document IDs deriving from each filter query `fq`.
+Filter queries are executed as pre-filtering for the main query. This means that in the first scenario, the query id:(1,2,3)
Review Comment:
This documentation change is not clear:
Maybe you meant:
"
Filter queries are executed as pre-filters: the main query refines the sub-set of search results derived from the application of all the filter queries combined as 'MUST' clauses(boolean AND).
This means that in
[source,text]
&q=id:(1 2 3)&fq={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
The results are prefiltered by the topK knn retrieval and then only the documents from this subset, matching the query 'q=id:(1 2 3)' are returned.
In
[source,text]
&q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]&fq=id:(1 2 3)
The results are prefiltered by the fq=id:(1 2 3) and then only the documents from this subset are considered as candidates for the topK knn retrieval.
"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] alessandrobenedetti commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
alessandrobenedetti commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r905193102
##########
solr/core/src/java/org/apache/solr/schema/DenseVectorField.java:
##########
@@ -276,6 +276,11 @@ public Query getKnnVectorQuery(String fieldName, float[] vectorToSearch, int top
return new KnnVectorQuery(fieldName, vectorToSearch, topK);
}
+ public Query getKnnVectorQuery(
Review Comment:
Hi @cpoerschke , thanks for the suggestion, I would normally agree but being an "experimental" feature I think we can just keep it simple and do that here in this pull request.
In this way, we'll avoid the complexity of having to create another pull request immediately after (and it's a matter of 1 version or another anyway)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] cpoerschke commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
cpoerschke commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r906238869
##########
solr/core/src/java/org/apache/solr/search/QueryUtils.java:
##########
@@ -215,4 +221,38 @@ public static Query combineQueryAndFilter(Query scoreQuery, Query filterQuery) {
}
}
}
+
+ /**
+ * Parse the filter queries in Solr request
+ *
+ * @param req Solr request
+ * @param fixNegativeQueries if true, negative queries are rewritten by adding a MatchAllDocs
+ * query clause
+ * @return and array of Query. If the request does not contain filter queries, returns an empty
+ * list.
+ * @throws SyntaxError if an error occurs during parsing
+ */
+ public static List<Query> parseFilterQueries(SolrQueryRequest req, boolean fixNegativeQueries)
+ throws SyntaxError {
+
+ String[] filterQueriesStr = req.getParams().getParams(CommonParams.FQ);
+
+ if (filterQueriesStr != null) {
+ List<Query> filters = new ArrayList<>(filterQueriesStr.length);
+ for (String fq : filterQueriesStr) {
+ if (fq != null && fq.trim().length() != 0) {
+ QParser fqp = QParser.getParser(fq, req);
+ fqp.setIsFilter(true);
Review Comment:
Wondering if this could also be under a flag since not all existing callers currently do the `setIsFilter` call.
```suggestion
if (setIsFilter) {
fqp.setIsFilter(true);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r905117282
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
+ .map(
+ filterQuery ->
+ denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, filterQuery))
+ .orElse(
+ denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK));
+ }
+
+ private Optional<? extends Query> getFilterQuery() throws SolrException {
+ if (!isFilter()) {
+ String[] filterQueries = req.getParams().getParams(CommonParams.FQ);
+ return ofNullable(filterQueries).filter(fq -> fq.length != 0).map(this::computeFilterQuery);
+ }
+ return Optional.empty();
+ }
+
+ private Query computeFilterQuery(String[] filterQueries) throws SolrException {
+
+ List<Query> filters = new ArrayList<>(filterQueries.length);
+
+ for (String filterQuery : filterQueries) {
+ if (filterQuery != null && filterQuery.trim().length() != 0) {
+ QParser filterParser;
+ try {
+ filterParser = QParser.getParser(filterQuery, super.getReq());
Review Comment:
I did some refactoring in the way you suggested. Now there is a method for parsing filter queries in QueryUtils.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r911773738
##########
solr/core/src/java/org/apache/solr/search/QueryUtils.java:
##########
@@ -215,4 +221,38 @@ public static Query combineQueryAndFilter(Query scoreQuery, Query filterQuery) {
}
}
}
+
+ /**
+ * Parse the filter queries in Solr request
+ *
+ * @param req Solr request
+ * @param fixNegativeQueries if true, negative queries are rewritten by adding a MatchAllDocs
+ * query clause
+ * @return and array of Query. If the request does not contain filter queries, returns an empty
+ * list.
+ * @throws SyntaxError if an error occurs during parsing
+ */
+ public static List<Query> parseFilterQueries(SolrQueryRequest req, boolean fixNegativeQueries)
+ throws SyntaxError {
+
+ String[] filterQueriesStr = req.getParams().getParams(CommonParams.FQ);
+
+ if (filterQueriesStr != null) {
+ List<Query> filters = new ArrayList<>(filterQueriesStr.length);
+ for (String fq : filterQueriesStr) {
+ if (fq != null && fq.trim().length() != 0) {
+ QParser fqp = QParser.getParser(fq, req);
+ fqp.setIsFilter(true);
Review Comment:
I think that adding this kind of parameter might make the method more confused and would discourage future usage.
setIsFilter(true) is needed for optimizing queries that do not need scoring but are not necessary. In the case of KnnQParser, it is needed also for understanding if it is necessary to parse recursively the filter queries (and avoid infinite recursion).
I double-checked and scoring is not involved in any scenario involved in this refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] alessandrobenedetti commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
alessandrobenedetti commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r914906892
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +78,41 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, getFilterQuery());
+ }
+
+ private Query getFilterQuery() throws SolrException {
+ if (!isFilter()) {
+ String[] filterQueries = req.getParams().getParams(CommonParams.FQ);
+ if (filterQueries != null && filterQueries.length != 0) {
+ return computeFilterQuery(filterQueries);
+ }
+ }
+ return null;
+ }
+
+ private Query computeFilterQuery(String[] filterQueries) throws SolrException {
Review Comment:
do we really need two private methods:
getFilterQuery()
computeFilterQuery(String[] filterQueries)
Can't we just have one "buildFilterQuery()" ?
They don't appear to be that complex to require a split, but I am happy to reconsider if there was some deep motivation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r904033703
##########
solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java:
##########
@@ -53,18 +53,7 @@
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.schema.IndexSchema;
import org.apache.solr.schema.SchemaField;
-import org.apache.solr.search.DocIterator;
-import org.apache.solr.search.DocList;
-import org.apache.solr.search.DocListAndSet;
-import org.apache.solr.search.QParser;
-import org.apache.solr.search.QParserPlugin;
-import org.apache.solr.search.QueryParsing;
-import org.apache.solr.search.ReturnFields;
-import org.apache.solr.search.SolrIndexSearcher;
-import org.apache.solr.search.SolrQueryTimeoutImpl;
-import org.apache.solr.search.SolrReturnFields;
-import org.apache.solr.search.SortSpec;
-import org.apache.solr.search.SyntaxError;
+import org.apache.solr.search.*;
Review Comment:
Didn't recognize that IntelliJ changed it automatically. I've restored the dependency list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] alessandrobenedetti commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
alessandrobenedetti commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r900200280
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
Review Comment:
the if condition is not necessary, as the Lucene method accepts the null filter query
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +81,49 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return getFilterQuery()
+ .map(
+ filterQuery ->
+ denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, filterQuery))
+ .orElse(
+ denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK));
+ }
+
+ private Optional<? extends Query> getFilterQuery() throws SolrException {
Review Comment:
don't think we need an Optional here, let's discuss
##########
solr/core/src/java/org/apache/solr/schema/DenseVectorField.java:
##########
@@ -276,6 +276,11 @@ public Query getKnnVectorQuery(String fieldName, float[] vectorToSearch, int top
return new KnnVectorQuery(fieldName, vectorToSearch, topK);
}
+ public Query getKnnVectorQuery(
Review Comment:
It's not necessary to keep the original one, KnnVectorQuery accepts 'null' for empty filter query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] alessandrobenedetti commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
alessandrobenedetti commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r914902058
##########
solr/core/src/java/org/apache/solr/search/QueryUtils.java:
##########
@@ -215,4 +221,38 @@ public static Query combineQueryAndFilter(Query scoreQuery, Query filterQuery) {
}
}
}
+
+ /**
+ * Parse the filter queries in Solr request
+ *
+ * @param req Solr request
+ * @param fixNegativeQueries if true, negative queries are rewritten by adding a MatchAllDocs
+ * query clause
+ * @return and array of Query. If the request does not contain filter queries, returns an empty
+ * list.
+ * @throws SyntaxError if an error occurs during parsing
+ */
+ public static List<Query> parseFilterQueries(SolrQueryRequest req, boolean fixNegativeQueries)
+ throws SyntaxError {
+
+ String[] filterQueriesStr = req.getParams().getParams(CommonParams.FQ);
+
+ if (filterQueriesStr != null) {
+ List<Query> filters = new ArrayList<>(filterQueriesStr.length);
+ for (String fq : filterQueriesStr) {
+ if (fq != null && fq.trim().length() != 0) {
+ QParser fqp = QParser.getParser(fq, req);
+ fqp.setIsFilter(true);
Review Comment:
Yes, @cpoerschke , I have done some investigations and it seems that the callers that were not setting it, should have done that, so it shouldn't be parametric
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r911773738
##########
solr/core/src/java/org/apache/solr/search/QueryUtils.java:
##########
@@ -215,4 +221,38 @@ public static Query combineQueryAndFilter(Query scoreQuery, Query filterQuery) {
}
}
}
+
+ /**
+ * Parse the filter queries in Solr request
+ *
+ * @param req Solr request
+ * @param fixNegativeQueries if true, negative queries are rewritten by adding a MatchAllDocs
+ * query clause
+ * @return and array of Query. If the request does not contain filter queries, returns an empty
+ * list.
+ * @throws SyntaxError if an error occurs during parsing
+ */
+ public static List<Query> parseFilterQueries(SolrQueryRequest req, boolean fixNegativeQueries)
+ throws SyntaxError {
+
+ String[] filterQueriesStr = req.getParams().getParams(CommonParams.FQ);
+
+ if (filterQueriesStr != null) {
+ List<Query> filters = new ArrayList<>(filterQueriesStr.length);
+ for (String fq : filterQueriesStr) {
+ if (fq != null && fq.trim().length() != 0) {
+ QParser fqp = QParser.getParser(fq, req);
+ fqp.setIsFilter(true);
Review Comment:
I think that adding this kind of parameter might make the method more confused and would discourage future usage.
setIsFilter(true) is needed for optimizing queries that do not need scoring but are not necessary. In the case of KnnQParser, it is also needed for understanding if it is necessary to parse recursively the filter queries (and avoid infinite recursion).
I double-checked and scoring is not involved in any scenario involved in this refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r923375406
##########
solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java:
##########
@@ -72,7 +78,41 @@ public Query parse() {
DenseVectorField denseVectorType = (DenseVectorField) fieldType;
float[] parsedVectorToSearch = parseVector(vectorToSearch, denseVectorType.getDimension());
- return denseVectorType.getKnnVectorQuery(schemaField.getName(), parsedVectorToSearch, topK);
+
+ return denseVectorType.getKnnVectorQuery(
+ schemaField.getName(), parsedVectorToSearch, topK, getFilterQuery());
+ }
+
+ private Query getFilterQuery() throws SolrException {
+ if (!isFilter()) {
+ String[] filterQueries = req.getParams().getParams(CommonParams.FQ);
+ if (filterQueries != null && filterQueries.length != 0) {
+ return computeFilterQuery(filterQueries);
+ }
+ }
+ return null;
+ }
+
+ private Query computeFilterQuery(String[] filterQueries) throws SolrException {
Review Comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r923375158
##########
solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc:
##########
@@ -279,13 +279,12 @@ The `knn` query parser can be used with filter queries:
[IMPORTANT]
====
-When using `knn` in these scenarios make sure you have a clear understanding of how filter queries work in Apache Solr:
-The Ranked List of document IDs resulting from the main query `q` is intersected with the set of document IDs deriving from each filter query `fq`.
+Filter queries are executed as pre-filtering for the main query. This means that in the first scenario, the query id:(1,2,3)
Review Comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r911773738
##########
solr/core/src/java/org/apache/solr/search/QueryUtils.java:
##########
@@ -215,4 +221,38 @@ public static Query combineQueryAndFilter(Query scoreQuery, Query filterQuery) {
}
}
}
+
+ /**
+ * Parse the filter queries in Solr request
+ *
+ * @param req Solr request
+ * @param fixNegativeQueries if true, negative queries are rewritten by adding a MatchAllDocs
+ * query clause
+ * @return and array of Query. If the request does not contain filter queries, returns an empty
+ * list.
+ * @throws SyntaxError if an error occurs during parsing
+ */
+ public static List<Query> parseFilterQueries(SolrQueryRequest req, boolean fixNegativeQueries)
+ throws SyntaxError {
+
+ String[] filterQueriesStr = req.getParams().getParams(CommonParams.FQ);
+
+ if (filterQueriesStr != null) {
+ List<Query> filters = new ArrayList<>(filterQueriesStr.length);
+ for (String fq : filterQueriesStr) {
+ if (fq != null && fq.trim().length() != 0) {
+ QParser fqp = QParser.getParser(fq, req);
+ fqp.setIsFilter(true);
Review Comment:
I think that adding this kind of parameter might make the method more confused and would discourage future usage.
setIsFilter(true) is useful for optimizing queries that do not need scoring. In the case of KnnQParser, it is also needed for understanding if it is necessary to parse recursively the filter queries (and avoid infinite recursion).
I double-checked and scoring is not involved in any scenario involved in this refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] eliaporciani commented on a diff in pull request #904: SOLR-16246: Introduced pre-filtering in KnnQParser
Posted by GitBox <gi...@apache.org>.
eliaporciani commented on code in PR #904:
URL: https://github.com/apache/solr/pull/904#discussion_r911773738
##########
solr/core/src/java/org/apache/solr/search/QueryUtils.java:
##########
@@ -215,4 +221,38 @@ public static Query combineQueryAndFilter(Query scoreQuery, Query filterQuery) {
}
}
}
+
+ /**
+ * Parse the filter queries in Solr request
+ *
+ * @param req Solr request
+ * @param fixNegativeQueries if true, negative queries are rewritten by adding a MatchAllDocs
+ * query clause
+ * @return and array of Query. If the request does not contain filter queries, returns an empty
+ * list.
+ * @throws SyntaxError if an error occurs during parsing
+ */
+ public static List<Query> parseFilterQueries(SolrQueryRequest req, boolean fixNegativeQueries)
+ throws SyntaxError {
+
+ String[] filterQueriesStr = req.getParams().getParams(CommonParams.FQ);
+
+ if (filterQueriesStr != null) {
+ List<Query> filters = new ArrayList<>(filterQueriesStr.length);
+ for (String fq : filterQueriesStr) {
+ if (fq != null && fq.trim().length() != 0) {
+ QParser fqp = QParser.getParser(fq, req);
+ fqp.setIsFilter(true);
Review Comment:
I think that adding this kind of parameter might make the method more confused and would discourage future usage.
setIsFilter(true) is needed for optimizing queries that do not need scoring. In the case of KnnQParser, it is also needed for understanding if it is necessary to parse recursively the filter queries (and avoid infinite recursion).
I double-checked and scoring is not involved in any scenario involved in this refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org