You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "ercsonusharma (via GitHub)" <gi...@apache.org> on 2025/07/04 09:19:23 UTC
[PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
ercsonusharma opened a new pull request, #3418:
URL: https://github.com/apache/solr/pull/3418
https://issues.apache.org/jira/browse/SOLR-17319
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA.
To create a Jira issue, you will need to create an account there first.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
This feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion). It also help resolve the issues being discussed w.r.t the [ previous PR](https://github.com/apache/solr/pull/2489), mainly around across shard documents merging. It provides more flexibility in terms of querying extending JSON Query DSL ultimately enabling Hybrid Search in a pure way solving the shortcomings.
# Solution
* Extended the QueryComponent to create new CombinedQueryComponent and ResponseBuilder to create new CombinedQueryResponseBuilder supports multiple response builders to hold the state and execute multiple queries.
* In JSON Query DSL, a parameter is added to identity Combined Query request and basis that it invokes the new CombinedQueryComponent
* CombinedQueryComponent have multiple response builders assigned for each query. These queries are first executed at the SolrSearchIndexer level and combined them using RRF for now.
* At Shard level also, the responses for the multiple queries are merged.
# Tests
* Added tests for testing the RRF logic independently.
* Added tests for across search index and distributed requests.
* Added tests to assert existing behaviour of search handler's QueryComponent as well as for the newly added CombinedQueryComponent basis the flag in json query DSL.
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
- [x] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [x] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231192975
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
Review Comment:
observation: use of `computeIfAbsent` here means that
* any configured combiner silently takes precedence over the in-built one, and
* that if the same combiner key is configured more than once then the second and subsequent ones will be silently loaded and then ignored
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2282280820
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
Review Comment:
@ercsonusharma could you explain to me what totalMatches represents and why it is computed in this way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2201488314
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
Review Comment:
yes, and `(CombinedQuery)ResponseBuilder` builder is already populated with all the parameters including highlights query [here](https://github.com/apache/solr/pull/3418/files#diff-1ca21c13d967412ea6227fe257616a9cb2ac45012b207373a526d3767a46b588R67).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198424564
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
Review Comment:
Early termination is not applicable in this context, as the complete set of documents is required for the RRF (Reciprocal Rank Fusion) algorithm to function correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2352932992
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -21,39 +21,59 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.util.SimpleOrderedMap;
import org.junit.BeforeClass;
import org.junit.Test;
/**
* The CombinedQueryComponentTest class is an integration test suite for the CombinedQueryComponent
- * in Solr. It verifies the functionality of the component by performing few basic queries and
- * validating the responses including limitations and combiner plugin.
+ * in Solr. It verifies the functionality of the component by performing few basic queries in single
+ * sharded mode and validating the responses including limitations and combiner plugin.
*/
-public class CombinedQueryComponentTest extends SolrTestCaseJ4 {
+public class CombinedQueryComponentTest extends BaseDistributedSearchTestCase {
Review Comment:
Since, the algorithm doesn't support non-distributed request anymore, it didn't make sense to add test cases by extending SolrTestCaseJ4 so I have changed it. However, this class can be merged with other CombinedQueryTest but the key idea was: `The class verifies the functionality of the component by performing few basic queries in single sharded mode and validating the responses including limitations and combiner plugin.`
If you feel it should be merged to other test classes, I am open to making those changes when we finalise [this one](https://github.com/apache/solr/pull/3418#discussion_r2331983004)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2289322587
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
Users have used distributed search in Solr long before SolrCloud and they can certainly continue to use it. SolrCloud-ness is unrelated to distributed search-ness. You could have a single or multi-shard collection in SolrCloud, and likewise in standalone you can have a single core or do distributed search across more than one (`shards=` param).
I'm left confused if there's actually a limitation. Putting aside grouping and cursorMark; I understand those may not work. You seem to point specifically to "distrib". For a single-sharded collection (or single core standalone mode), distrib will be false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318888403
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -905,7 +906,84 @@ protected boolean addFL(StringBuilder fl, String field, boolean additionalAdded)
return true;
}
+ protected abstract static class ShardDocQueue {
+ public abstract boolean push(ShardDoc shardDoc);
+
+ public abstract Map<Object, ShardDoc> resultIds(int offset);
+ }
+ ;
+
+ protected static class ShardDocQueueFactory
+ implements BiFunction<SortField[], Integer, ShardDocQueue> {
+
+ private final SolrIndexSearcher searcher;
+
+ public ShardDocQueueFactory(SolrIndexSearcher searcher) {
+ this.searcher = searcher;
+ }
+
+ @Override
+ public ShardDocQueue apply(SortField[] sortFields, Integer size) {
+ return new ShardDocQueue() {
+
+ // id to shard mapping, to eliminate any accidental dups
+ private final HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ private final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(sortFields, size, searcher);
+
+ @Override
+ public boolean push(ShardDoc shardDoc) {
+ final String prevShard = uniqueDoc.put(shardDoc.id, shardDoc.shard);
+ if (prevShard != null) {
+ // duplicate detected
+
+ // For now, just always use the first encountered since we can't currently
+ // remove the previous one added to the priority queue. If we switched
+ // to the Java5 PriorityQueue, this would be easier.
+ return false;
+ // make which duplicate is used deterministic based on shard
+ // if (prevShard.compareTo(shardDoc.shard) >= 0) {
+ // TODO: remove previous from priority queue
+ // return false;
+ // }
+ }
+
+ queue.insertWithOverflow(shardDoc);
+ return true;
+ }
+
+ @Override
+ public Map<Object, ShardDoc> resultIds(int offset) {
+ final Map<Object, ShardDoc> resultIds = new HashMap<>();
+
+ // The queue now has 0 -> queuesize docs, where queuesize <= start + rows
+ // So we want to pop the last documents off the queue to get
+ // the docs offset -> queuesize
+ int resultSize = queue.size() - offset;
+ resultSize = Math.max(0, resultSize); // there may not be any docs in range
+
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+
+ return resultIds;
+ }
+ };
+ }
+ }
+ ;
+
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ implementMergeIds(rb, sreq, new ShardDocQueueFactory(rb.req.getSearcher()));
+ }
+
+ private void implementMergeIds(
Review Comment:
> > though in that case no need for a factory perhaps:
>
> Simple!
https://github.com/apache/solr/pull/3418/commits/a52dd22072b98a91d6e7d065c4df843e340549cd for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "atris (via GitHub)" <gi...@apache.org>.
atris commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2281542076
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
Review Comment:
Thinking about this - a ResponseBuilder per query might be too memory intensive. Especially for LLM reformulated use cases where we might be fanning out. Is there a way to use a single ResponseBuilder with tracking for QueryResults?
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.facet.FacetModule;
+
+/**
+ * The CombinedQuerySearchHandler class extends the SearchHandler and provides custom behavior for
+ * handling combined queries. It overrides methods to create a response builder based on the {@link
+ * CombinerParams#COMBINER} parameter and to define the default components included in the search
+ * configuration.
+ */
+public class CombinedQuerySearchHandler extends SearchHandler {
+
+ /**
+ * Overrides the default response builder creation method. This method checks if the {@link
+ * CombinerParams#COMBINER} parameter is set to true in the request. If it is, it returns an
+ * instance of {@link CombinedQueryResponseBuilder}, otherwise, it returns an instance of {@link
+ * ResponseBuilder}.
+ *
+ * @param req the SolrQueryRequest object
+ * @param rsp the SolrQueryResponse object
+ * @param components the list of SearchComponent objects
+ * @return the appropriate ResponseBuilder instance based on the CombinerParams.COMBINER parameter
+ */
+ @Override
+ protected ResponseBuilder newResponseBuilder(
+ SolrQueryRequest req, SolrQueryResponse rsp, List<SearchComponent> components) {
+ if (req.getParams().getBool(CombinerParams.COMBINER, false)) {
+ return new CombinedQueryResponseBuilder(req, rsp, components);
+ }
+ return super.newResponseBuilder(req, rsp, components);
+ }
+
+ /**
+ * Overrides the default components and returns a list of component names that are included in the
+ * default configuration.
+ *
+ * @return a list of component names
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ protected List<String> getDefaultComponents() {
+ List<String> names = new ArrayList<>(9);
+ names.add(CombinedQueryComponent.COMPONENT_NAME);
Review Comment:
Is adding CombinedQueryComponent as the first component breaking any dependency chains?
##########
solr/core/src/test/org/apache/solr/search/combine/ReciprocalRankFusionTest.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import static org.apache.solr.common.params.CombinerParams.COMBINER_RRF_K;
+import static org.apache.solr.common.params.CombinerParams.RECIPROCAL_RANK_FUSION;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The ReciprocalRankFusionTest class is a unit test suite for the {@link ReciprocalRankFusion}
+ * class. It verifies the correctness of the fusion algorithm and its supporting methods.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class ReciprocalRankFusionTest extends SolrTestCaseJ4 {
+
+ public static ReciprocalRankFusion reciprocalRankFusion;
+
+ /**
+ * Initializes the test environment by setting up the {@link ReciprocalRankFusion} instance with
+ * specific parameters.
+ */
+ @BeforeClass
+ public static void beforeClass() {
+ NamedList<?> args = new NamedList<>(Map.of("k", "20"));
+ reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(args);
+ }
+
+ /** Tests the functionality of combining using RRF across local search indices. */
+ @Test
+ public void testSearcherCombine() {
+ List<QueryResult> rankedList = getQueryResults();
+ SolrParams solrParams = params(COMBINER_RRF_K, "10");
+ QueryResult result = reciprocalRankFusion.combine(rankedList, solrParams);
+ assertEquals(20, reciprocalRankFusion.getK());
+ assertEquals(3, result.getDocList().size());
+ }
+
+ private static List<QueryResult> getQueryResults() {
+ QueryResult r1 = new QueryResult();
+ r1.setDocList(
+ new DocSlice(
+ 0,
+ 2,
+ new int[] {1, 2},
+ new float[] {0.67f, 0, 62f},
Review Comment:
Is this intended to be 0.62f?
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed and grouping query. Using Cursors is not advisable, as it may produce inconsistent results.
Review Comment:
This IMO is quite hand wavy. Lets either support cursors with full functionality, or not and explicitly fail queries where this happens.
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed and grouping query. Using Cursors is not advisable, as it may produce inconsistent results.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner.enable` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
+
+```
+{
+ "queries": {
+ "lexical1": {
+ "lucene": {
+ "query": "title:sales"
+ }
+ },
+ "lexical2": {
+ "lucene": {
+ "query": "title:report"
+ }
+ },
+ "vector": {
+ "knn": {
+ "f": "vector",
+ "topK" :5,
+ "query": "[0.1,-0.34,0.89,0.02]"
+ }
+ }
+ },
+ "limit": 5,
+ "fields": ["id", "score", "title"],
+ "params": {
+ "combiner": true,
Review Comment:
Parameter is combiner.enable but documentation shows combiner:true. Lets stick with one naming convention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322141633
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -205,7 +203,7 @@ public void testMultipleQueryWithSort() throws Exception {
}
/**
- * Tests the hybrid query functionality of the system.
+ * Tests the hybrid query functionality of the system with various setting of pagination.
Review Comment:
thanks for improving this test :-)
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -69,21 +68,24 @@ public int getK() {
@Override
public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
- List<DocListAndSet> docListAndSet = getDocListsAndSetFromQueryResults(rankedLists);
QueryResult combinedResult = new QueryResult();
- combineResults(combinedResult, docListAndSet, false, kVal);
+ combineResults(combinedResult, rankedLists, false, kVal);
return combinedResult;
}
- private static List<DocListAndSet> getDocListsAndSetFromQueryResults(
- List<QueryResult> rankedLists) {
- List<DocListAndSet> docLists = new ArrayList<>(rankedLists.size());
- for (QueryResult rankedList : rankedLists) {
- docLists.add(rankedList.getDocListAndSet());
- }
- return docLists;
- }
-
+ /**
+ * Merges per-shard ranked results using Reciprocal Rank Fusion (RRF).
Review Comment:
Reading this sentence, I think okay this is a typical application of RRF. And the code does that, so it's technically not wrong. But lets be real with the reader; in practice this is going to interleave the results because docs uniquely only appear in one shard. Say that. And say that you change the score to the RRF score.
As an aside, it'd probably be ideal to secondarily sort based on doc ID so we have more reproducibility. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322185934
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
Review Comment:
Yeah, that's for SolrCloud and/or distributed search. But, for User Managed mode where we don't pass `shards` param, the pagination would not work for now and that's the limitation I was trying to establish [here](https://github.com/apache/solr/pull/3418#discussion_r2312593325).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322683455
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.plugin.NamedListInitializedPlugin;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * shard documents. It provides a framework for different algorithms to be implemented for merging
+ * ranked lists and shard documents.
+ */
+public abstract class QueryAndResponseCombiner implements NamedListInitializedPlugin {
+ /**
+ * Combines multiple ranked lists into a single QueryResult.
+ *
+ * @param rankedLists a list of ranked lists to be combined
+ * @param solrParams params to be used when provided at query time
+ * @return a new QueryResult containing the combined results
+ * @throws IllegalArgumentException if the input list is empty
+ */
+ public abstract QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams);
+
+ /**
+ * Combines shard documents based on the provided map.
+ *
+ * @param shardDocMap a map where keys represent shard IDs and values are lists of ShardDocs for
+ * each shard
+ * @param solrParams params to be used when provided at query time
+ * @return a combined list of ShardDocs from all shards
+ */
+ public abstract List<ShardDoc> combine(
+ Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams);
+
+ /**
+ * Retrieves a list of explanations for the given queries and results.
+ *
+ * @param queryKeys the keys associated with the queries
+ * @param queries the list of queries for which explanations are requested
+ * @param queryResult the list of QueryResult corresponding to each query
+ * @param searcher the SolrIndexSearcher used to perform the search
+ * @param schema the IndexSchema used to interpret the search results
+ * @param solrParams params to be used when provided at query time
+ * @return a list of explanations for the given queries and results
+ * @throws IOException if an I/O error occurs during the explanation retrieval process
+ */
+ public abstract NamedList<Explanation> getExplanations(
Review Comment:
I'm suggest insisting that in the interface. In Solr I think we want to get away from NamedList usages that are not SimpleOrderedMap. Non-SOM is only for when keys may repeat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2591417528
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
I was thinking the CombinedQuery* classes would come here too, thus all classes associated with this are together. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2201074161
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
Review Comment:
> ... so updated ResponseBuilders are not propagated there. ...
So to perhaps illustrate with an example, https://github.com/apache/solr/blob/releases/solr/9.8.1/solr/core/src/java/org/apache/solr/handler/component/HighlightComponent.java#L97 sets the `rb.doHighlights` flag and this would be on on the `(CombinedQuery)ResponseBuilder` builder but not `CombinedQueryResponseBuilder.responseBuilders` builders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2199751152
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -1234,6 +1230,34 @@ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
}
}
+ /**
+ * Creates a map of shard results based on the provided parameters.
+ *
+ * @param rb ResponseBuilder to retrieve any request info
+ * @param resultSize the desired size of the result map
+ * @param queue the queue containing sorted hits from different shards
+ * @param shardDocMap a map of shard documents indexed by shard ID
+ * @param responseDocs the final SolrDocumentList to be returned or set in the responseBuilder
+ * @return a map of shard documents, where the keys are the shard IDs as strings, and the values
+ * are the corresponding ShardDoc objects
+ */
+ protected Map<Object, ShardDoc> createShardResult(
Review Comment:
It should be String but the [ResponseBuilder](https://github.com/ercsonusharma/solr/blob/9b0c76ef88a204829bbe1a2cb1cdec938d1b8586/solr/core/src/java/org/apache/solr/handler/component/ResponseBuilder.java#L169-L169) has type Object so had to keep it Object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2289235236
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
> rather duplicating the code to some extent as it may impact the existing component as recommended by Atri.
@atris are you against factoring out duplicative code in QueryComponent to try to express an algorithm once?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2316087608
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -905,7 +906,84 @@ protected boolean addFL(StringBuilder fl, String field, boolean additionalAdded)
return true;
}
+ protected abstract static class ShardDocQueue {
+ public abstract boolean push(ShardDoc shardDoc);
+
+ public abstract Map<Object, ShardDoc> resultIds(int offset);
+ }
+ ;
+
+ protected static class ShardDocQueueFactory
+ implements BiFunction<SortField[], Integer, ShardDocQueue> {
+
+ private final SolrIndexSearcher searcher;
+
+ public ShardDocQueueFactory(SolrIndexSearcher searcher) {
+ this.searcher = searcher;
+ }
+
+ @Override
+ public ShardDocQueue apply(SortField[] sortFields, Integer size) {
+ return new ShardDocQueue() {
+
+ // id to shard mapping, to eliminate any accidental dups
+ private final HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ private final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(sortFields, size, searcher);
+
+ @Override
+ public boolean push(ShardDoc shardDoc) {
+ final String prevShard = uniqueDoc.put(shardDoc.id, shardDoc.shard);
+ if (prevShard != null) {
+ // duplicate detected
+
+ // For now, just always use the first encountered since we can't currently
+ // remove the previous one added to the priority queue. If we switched
+ // to the Java5 PriorityQueue, this would be easier.
+ return false;
+ // make which duplicate is used deterministic based on shard
+ // if (prevShard.compareTo(shardDoc.shard) >= 0) {
+ // TODO: remove previous from priority queue
+ // return false;
+ // }
+ }
+
+ queue.insertWithOverflow(shardDoc);
+ return true;
+ }
+
+ @Override
+ public Map<Object, ShardDoc> resultIds(int offset) {
+ final Map<Object, ShardDoc> resultIds = new HashMap<>();
+
+ // The queue now has 0 -> queuesize docs, where queuesize <= start + rows
+ // So we want to pop the last documents off the queue to get
+ // the docs offset -> queuesize
+ int resultSize = queue.size() - offset;
+ resultSize = Math.max(0, resultSize); // there may not be any docs in range
+
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+
+ return resultIds;
+ }
+ };
+ }
+ }
+ ;
+
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ implementMergeIds(rb, sreq, new ShardDocQueueFactory(rb.req.getSearcher()));
+ }
+
+ private void implementMergeIds(
Review Comment:
I don't think I get the point of the "implementMergeIds" method separation unless it could get called in more situations (that I don't see in the PR interface).
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
Review Comment:
I recommend only using "Now" for a temporal meaning. Here, I suggest one of these possible names: `iRb` `thisRb`, or maybe rename the `rb` in scope allowing this one here to use simply `rb`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322105169
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -213,7 +214,7 @@ private void prepareCombinedResponseBuilder(
combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
crb.setResult(combinedQueryResult);
- if (rb.isDebug()) {
+ if (rb.isDebugQuery()) {
String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
NamedList<Explanation> explanations =
Review Comment:
the explainations are for "isDebugResults". If you want to add the queries, then _that_ is "isDebugQuery"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315974090
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of lexical and vector queries using the combiner component.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("nullfirst", String.valueOf(i % 3));
Review Comment:
Didn't intend adding an extra field in the existing schema just for testing purpose. But, can be done easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2319184435
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
> and we can adjust the response to indicate the inexactness if more were underlying results were present.
True, it's already being marked as part of this [code](https://github.com/apache/solr/pull/3418/files#diff-9ee2d3ca53340b37b597cf2666844ce73ea534416ead953dddc15c5399458282R173)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231297175
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * shard documents. It provides a framework for different algorithms to be implemented for merging
+ * ranked lists and shard documents.
+ */
+public abstract class QueryAndResponseCombiner {
Review Comment:
```suggestion
public abstract class QueryAndResponseCombiner implements NamedListInitializedPlugin {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2288385230
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
Agreed. I think it's confusing and I don't find a right term yet. But, the [code line](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L493) is currently using [isDistrib(SolrQueryRequest req)](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L307) to determine this behavior. Based on your suggestion, I think `Standalone (AKA user-managed)` might be the right term to adopt — does that sound accurate to you?
I am using `shards` param to query in my test case which uses `BaseDistributedSearchTestCase` (or I could have used distrib=true).
> Your response didn't indicate if this would work for a SolrCloud single-shard collection. Will it?
Yes, it would work for single shard or multi shard both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2206368642
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
Review Comment:
Moved all the logic to CombinedQueryComponent with no change in QueryComponent. Also, have put a limit on the number of combined queries. However, I cannot extend SearchComponent because I still need many methods in QueryComponent. Hence, I have been using the parent class methods, which do not use any global parameters of the class and won't affect the QueryComponent at all IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3621680527
Please take a look Sonu. I also refactored SearchHandler a bit to have a clearer flow / readability and also extensibility. I'd like to merge the refactoring portions of this PR in one go, then the rest of this here. Like... I could extract all that to be under SOLR-17982.
A TODO I didn't get to tonight is testing the different forceDistrib API with a custom SearchHandler subclass. Both testForceDistrib methods are there but need to be modified. I suspect shards.info may reflect what we want to see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3650202448
#3931 should merge first (say in ~2 days), then this one will be simpler and it's ready to go. I expect I should merge this one by the end of next week..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3299328167
> I have doubts that it makes sense to include "Way 1" if, I imagine, it increases the complexity/documentation matters and more fundamentally... why would someone choose it.
I don't think it introduces any additional complexity in terms of code or documentation (adding a parameter combiner.method), rather it reuses several piece of code.
IMO, If the data is independently sharded and shard-level scoring doesn't matter in overall query relevance, it may not make sense to first combine shard results based on their original scores and then apply RRF per query (as done in Way 2). In such cases, users may prefer Way 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2436004520
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
Review Comment:
For an isShard=true request then. Okay; but then why wouldn't the logic just pass-through as normal? It's not clear why CombinedQueryComponent has any logic to accomplish at this phase. "Way 2" is only at aggregation (disbtrib=true), in my head anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3402817819
@dsmiley - Just checking if you have any concerns apart from a few mentioned above, which I have resolved recently. Not sure if you had a chance to review it completely yet. Would you be able to advise on the best path forward?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3341358574
@dsmiley I had a chance to take a look at Christine's POC changes and have updated my findings there.
>I'll need to come at this afresh (not just looking at deltas from previous review)
Just checking in—have you had a chance to take a look at this yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2486775121
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
+ }
+
+ /**
+ * Test multiple query execution with sort.
+ *
+ * @throws Exception the exception
+ */
+ @Test
+ public void testMultipleQueryWithSort() throws Exception {
Review Comment:
Wouldn't a boolean OR of both subqueries without CombinedQueryComponent count as "built in support for joining two queries"? "Consistency of the query structure".... huh?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "alessandrobenedetti (via GitHub)" <gi...@apache.org>.
alessandrobenedetti commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3056668948
Hi @ercsonusharma , thanks for resurrecting this, didn't have time to dedicate to the feature in the last few months, good to see some movement!
In the next couple of weeks, I should be able to give it a go and review it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2261039412
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
Review Comment:
I used the same naming convention as used in QueryComponent as I copied the code from there. But, I can refactor it to name differently if you want.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
Review Comment:
I used the same naming convention as used in QueryComponent as I replicated the code from there. But, I can refactor it to name differently if you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2261069585
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
This method is present in QueryComponent as well. The primary purpose of this method is to merge the responses received from all the shards basis score and sortValues for QueryComponent. And then, set the responses like numFound, docs etc to the SolrDocumentList. The way it varies in CombinedQueryComponent, is the way it ranks the responses by taking the positions of the shard documents into consideration first (in case of RRF).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2263682221
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
Review Comment:
It's *very* common for `srsp` to be named this. I think consistency with common names throughtout the codebase is good, even if it isn't the name we might have chosen on a green field project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322092316
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_START;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
+ * It verifies the functionality of the component by performing various queries and validating the
+ * responses.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
Review Comment:
Feel free to at-mention me to look into a failure.
Solr tests are rather stringent to enforce best practices like ensuring threads aren't running when a test finishes. When this is violated, usually it means the test hasn't cleaned up (closed) something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318852609
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
This is also true for a single query which are spread over multiple shards. But, yes we can add this as clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315036487
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors. In User-Managed (aka Standalone) Mode, the `shards` parameter must be provided to enable this feature.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
+
+```
+{
+ "queries": {
+ "lexical1": {
+ "lucene": {
+ "query": "title:sales"
+ }
+ },
+ "lexical2": {
+ "lucene": {
+ "query": "title:report"
+ }
+ },
+ "vector": {
+ "knn": {
+ "f": "vector",
+ "topK" :5,
+ "query": "[0.1,-0.34,0.89,0.02]"
+ }
+ }
+ },
+ "limit": 5,
+ "fields": ["id", "score", "title"],
+ "params": {
+ "combiner": true,
+ "combiner.query": ["lexical1", "vector"],
+ "combiner.algorithm": "rrf",
+ "combiner.rrf.k": "15"
+ }
+}
+```
+
+== Search Handler Configuration
+
+Combined Query Feature has a separate handler with class `solr.CombinedQuerySearchHandler` which can be configured as below:
+
+```
+<requestHandler name="/search" class="solr.CombinedQuerySearchHandler">
+<bool name="httpCaching">true</bool>
Review Comment:
It's not needed. removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2316149147
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -905,7 +906,84 @@ protected boolean addFL(StringBuilder fl, String field, boolean additionalAdded)
return true;
}
+ protected abstract static class ShardDocQueue {
+ public abstract boolean push(ShardDoc shardDoc);
+
+ public abstract Map<Object, ShardDoc> resultIds(int offset);
+ }
+ ;
+
+ protected static class ShardDocQueueFactory
+ implements BiFunction<SortField[], Integer, ShardDocQueue> {
+
+ private final SolrIndexSearcher searcher;
+
+ public ShardDocQueueFactory(SolrIndexSearcher searcher) {
+ this.searcher = searcher;
+ }
+
+ @Override
+ public ShardDocQueue apply(SortField[] sortFields, Integer size) {
+ return new ShardDocQueue() {
+
+ // id to shard mapping, to eliminate any accidental dups
+ private final HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ private final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(sortFields, size, searcher);
+
+ @Override
+ public boolean push(ShardDoc shardDoc) {
+ final String prevShard = uniqueDoc.put(shardDoc.id, shardDoc.shard);
+ if (prevShard != null) {
+ // duplicate detected
+
+ // For now, just always use the first encountered since we can't currently
+ // remove the previous one added to the priority queue. If we switched
+ // to the Java5 PriorityQueue, this would be easier.
+ return false;
+ // make which duplicate is used deterministic based on shard
+ // if (prevShard.compareTo(shardDoc.shard) >= 0) {
+ // TODO: remove previous from priority queue
+ // return false;
+ // }
+ }
+
+ queue.insertWithOverflow(shardDoc);
+ return true;
+ }
+
+ @Override
+ public Map<Object, ShardDoc> resultIds(int offset) {
+ final Map<Object, ShardDoc> resultIds = new HashMap<>();
+
+ // The queue now has 0 -> queuesize docs, where queuesize <= start + rows
+ // So we want to pop the last documents off the queue to get
+ // the docs offset -> queuesize
+ int resultSize = queue.size() - offset;
+ resultSize = Math.max(0, resultSize); // there may not be any docs in range
+
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+
+ return resultIds;
+ }
+ };
+ }
+ }
+ ;
+
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ implementMergeIds(rb, sreq, new ShardDocQueueFactory(rb.req.getSearcher()));
+ }
+
+ private void implementMergeIds(
Review Comment:
Err, yes, it would need to be protected so that deriving classes could call it.
```suggestion
implementMergeIds(rb, sreq, new ShardDocQueueFactory(rb.req.getSearcher()));
}
protected void implementMergeIds(
```
---
Or alternatively no `implementMergeIds` but a `newShardDocQueueFactory` method that deriving classes can override:
```
protected ShardDocQueueFactory newShardDocQueueFactory(ResponseBuilder rb) {
return new ShardDocQueueFactory(rb.req.getSearcher()));
}
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
...
}
```
though in that case no need for a factory perhaps:
```
protected ShardDocQueue newShardDocQueue(SolrIndexSearcher searcher, SortField[] sortFields, Integer size)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315324958
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
Review Comment:
Yes, it's only using JSON Query Parser as the `queries` key in the request has to be in a format that can be parsed using a JSON parser. I dont' see any way to parse multiple Solr Queries of different type using different parsers like lucene, dismax etc. Also, it's a long text ideal to be passed as JSON.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "atris (via GitHub)" <gi...@apache.org>.
atris commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2195792945
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
+ * It verifies the functionality of the component by performing various queries and validating the
+ * responses.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class CombinedQueryComponentTest extends SolrTestCaseJ4 {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and adding test documents to the index. This
+ * method prepares the Solr index with a set of documents for subsequent test cases.
+ *
+ * @throws Exception if any error occurs during setup, such as initialization failures or indexing
+ * issues.
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ List<SolrInputDocument> docs = new ArrayList<>();
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
Review Comment:
Please add test for the RRF score calculation explainability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3406714433
Sorry; this PR deserves more attention but I've been prioritizing my limited time to Solr 10. I'll return to this as soon as I can... perhaps tonight. I said that to myself yesterday so it's at least at the top of my discretionary list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2436044532
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
Review Comment:
It seems we have no SolrCloud test introduced here... I was hoping to run in a debugger to see for myself. Hmmmm. I could imagine a dummy ForceDistribSearchComponent test designed only to test that forceDistrib works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2434677644
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
Review Comment:
This would activate for both - SolrCloud and non-SolrCloud as we don't intend to go through the process() method of SearchComponent directly through `shortCircuit` rather we want it to happen through creating a HttpShardHandler which would query all the shards parallely and merge the result at the coordinator (even in case of one shard).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2434417321
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl =
+ String.format(Locale.ROOT, "%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
+ return;
+ }
Review Comment:
Also, the URL core construction here seems like it should be a utility method on HttpSolrCall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2453918348
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
Review Comment:
Yeah, definitely - A SolrCloud test can be introduced by extending `AbstractFullDistribZkTestBase`. Will send it out shortly.
Update: Added a test for SolrCloud.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2296307185
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
okay. But, that (`shortCircuit=false`) needs to be passed along with `distrib=true` as [per](https://github.com/apache/solr/blob/releases/solr/9.9.0/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L324) - can assert through a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2453916641
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
Review Comment:
Sorry, took longer to reply.
The forced distributed would anyway enable the shard call [(isShard=true)](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L565).
> then why wouldn't the logic just pass-through as normal
The `process()` method is directly communicating with the Searcher as we need to process each query at the shard searcher level. If you would look at the method override, it is also creating the combined docSet across queries other than few other metadata, required for merging the docs finally across the shard at the coordinator level.
However, this Searcher call per query can be further parallelised as added in comment.
@dsmiley
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3264406707
>Yet this PR/approach will not be able to comply since unlike most (all?) components, its results are affected substantially by distributed-search. The (unsaid?) vision of sharding / distributed-search was getting the same results as a single shard, and Solr does the work to pull off that trick, with plenty of tests demonstrating it does. In fact I'd say, with great disappointment, that the observed (by a user) results of this component will not be RRF when there's distributed search over shards.
pushed a change to the PR that adds an option for the user to choose which Combiner method to use — Way 1 (pre) or Way 2 (post).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2337697778
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -81,4 +88,33 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /**
+ * rb.distrib {@link ResponseBuilder} must be set for the combined query to work in case of single
+ * core standalone mode. This method set the parameter explicitly with other required solr param
+ * i.e. shards
+ *
+ * @param req the SolrQueryRequest
+ * @return boolean denoting whether the request can be marked as distributed.
+ */
+ @Override
+ protected boolean isDistrib(SolrQueryRequest req) {
+ boolean isDistrib = super.isDistrib(req);
+ if (!isDistrib
+ && !req.getParams().getBool(ShardParams.IS_SHARD, false)
+ && req.getHttpSolrCall() != null) {
+ log.info("Configuring distributed mode to enable Combined Query.");
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl = String.format("%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
Review Comment:
Makes sense. Have changed the logic in such a way that it can be useful for other search components as well where a distributed request can be forced depending on the component requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2332148796
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -81,4 +88,33 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /**
+ * rb.distrib {@link ResponseBuilder} must be set for the combined query to work in case of single
+ * core standalone mode. This method set the parameter explicitly with other required solr param
+ * i.e. shards
+ *
+ * @param req the SolrQueryRequest
+ * @return boolean denoting whether the request can be marked as distributed.
+ */
+ @Override
+ protected boolean isDistrib(SolrQueryRequest req) {
+ boolean isDistrib = super.isDistrib(req);
+ if (!isDistrib
+ && !req.getParams().getBool(ShardParams.IS_SHARD, false)
+ && req.getHttpSolrCall() != null) {
+ log.info("Configuring distributed mode to enable Combined Query.");
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl = String.format("%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
Review Comment:
I followed what you said [here](https://github.com/apache/solr/pull/3418#discussion_r2322678396):
I agree it's a bit sneaky. However, Apologies, but yet I didn't follow the part below:
> for a component or handler to communicate, we need the distributed search algorithm (no so-called short-circuit).
afaik, in standalone distributed search, we need the shards value to be passed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318603576
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * This class implements a query and response combiner that uses the Reciprocal Rank Fusion (RRF)
+ * algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.DEFAULT_COMBINER_RRF_K;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ public int getK() {
+ return k;
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
Review Comment:
Thanks for this insight. I changed the method to accomodate above proposed fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322081128
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
Review Comment:
That's for distributed-search but not single-core search.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2290975025
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
It would be User Managed Standalone Single Core (If someone uses it these days). But, it won’t be a limitation when the `distrib=true` flag is passed explicitly in that case also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286927124
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
Review Comment:
Somehow, I thought the method was non-thread safe. But it has changed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2486972983
##########
solr/core/src/java/org/apache/solr/search/combine/package-info.java:
##########
Review Comment:
Arguably, this package should be under org.apache.solr.handler.component, and then it would consolidate all the functionality added in this PR, not just these 2 classes. I know the combiner thing here is sort of general but it's only used by the new CombineQueryComponent, and I have no reason to believe that it's going to be used by new things that aren't search components.
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQuerySolrCloudTest.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.AbstractFullDistribZkTestBase;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Contains integration tests for the combined query functionality in a SolrCloud environment. This
+ * class extends AbstractFullDistribZkTestBase to leverage the distributed testing framework. The
+ * tests cover various scenarios including single and multiple lexical queries, sorting, pagination,
+ * faceting, and highlighting similar to {@link DistributedCombinedQueryComponentTest}
+ */
+public class CombinedQuerySolrCloudTest extends AbstractFullDistribZkTestBase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
Review Comment:
`initCore` should not be called for these `...Distrib...TestBase` test classes. Cores are managed by the base test infra.
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -21,39 +21,59 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.util.SimpleOrderedMap;
import org.junit.BeforeClass;
import org.junit.Test;
/**
* The CombinedQueryComponentTest class is an integration test suite for the CombinedQueryComponent
- * in Solr. It verifies the functionality of the component by performing few basic queries and
- * validating the responses including limitations and combiner plugin.
+ * in Solr. It verifies the functionality of the component by performing few basic queries in single
+ * sharded mode and validating the responses including limitations and combiner plugin.
*/
-public class CombinedQueryComponentTest extends SolrTestCaseJ4 {
+public class CombinedQueryComponentTest extends BaseDistributedSearchTestCase {
Review Comment:
I don't think extending BaseDistributedSearchTestCase makes sense in this scenario. BaseDistributedSearchTestCase compares a "control" client (single shard) to a distributed setup of a configurable number of shards; this is its fundamental purpose. Calling `fixShardCount(1)` misses the point. Can the single-sharded case can be tested with STCJ4? If not for some weird reason, surely SolrCloudTestCase.
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
Review Comment:
`initCore` should not be called for these abstract distrib test classes. Cores are managed by the base test infra. You can set the config it should use via the static String field `configString` & `schemaString`. So ugly/hacky, I know!
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQuerySolrCloudTest.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.AbstractFullDistribZkTestBase;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Contains integration tests for the combined query functionality in a SolrCloud environment. This
+ * class extends AbstractFullDistribZkTestBase to leverage the distributed testing framework. The
+ * tests cover various scenarios including single and multiple lexical queries, sorting, pagination,
+ * faceting, and highlighting similar to {@link DistributedCombinedQueryComponentTest}
+ */
+public class CombinedQuerySolrCloudTest extends AbstractFullDistribZkTestBase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ @Override
+ protected String getCloudSolrConfig() {
+ return "solrconfig-combined-query.xml";
+ }
+
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
Review Comment:
should be called in `@BeforeClass` place
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3476411171
@dsmiley Was I able to address your concern through above changes and responses? Please let me know if there is more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3574058453
@dsmiley - Just a gentle reminder to have a look at the above comments and provide approval if we are good to merge. Absolutely no rush at all, especially with the holiday season! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2449617288
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
Review Comment:
ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2384510445
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
Review Comment:
I get why doc5 is first, since it's the only one to appear in both lists. But wouldn't the remainder be basically the rest as they scored from both lists? Thus wouldn't doc7 (highest scoring in lexical2) be at position 2 or 3? Here it's at 5. And why would doc3 even be in the top 5? doc2 and doc6 are ahead of it in lexical1.
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
Review Comment:
I get why doc5 is first, since it's the *only* one to appear in both lists. But wouldn't the remainder be basically the rest as they scored from both lists? Thus wouldn't doc7 (highest scoring in lexical2) be at position 2 or 3? And why would doc3 even be in the top 5.
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
+ }
+
+ /**
+ * Test multiple query execution with sort.
+ *
+ * @throws Exception the exception
+ */
+ @Test
+ public void testMultipleQueryWithSort() throws Exception {
Review Comment:
This test's existence surprises me but I'm reminded you are thinking of this feature generically, not assuming RRF. I suppose if we're sorting by a field then the underlying implementation would ideally be merely a BooleanQuery with two SHOULD clauses. Would a user bother with this plugin for such a case when they can do that trivially?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2434634317
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
Review Comment:
With [reference](https://github.com/apache/solr/pull/3418#discussion_r2352867177), In the case of distributed mode, an HTTP request is made with rb.distrib=false through ShardHandlerRequest in one of the phases. The `process `method would be called for that particular shard query request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2596284428
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -86,4 +51,13 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /** Overrides the default list to include {@link CombinedQueryComponent}. */
+ @Override
+ protected List<String> getDefaultComponents() {
+ List<String> names = new ArrayList<>(super.getDefaultComponents());
+ String replaced = names.set(0, CombinedQueryComponent.COMPONENT_NAME);
+ assert replaced.equals(QueryComponent.COMPONENT_NAME);
Review Comment:
Interesting to learn that this is possible, as I thought it would limit the user from setting another SearchComponent as the first one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3301699447
okay @dsmiley . So I have kept only the Way 2 for now, removing Way 1 completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231391584
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
Review Comment:
Wondering if the `super.prepare(rb)` call here is needed and/or if a non-null `params.get(CommonParams.Q)` should throw an exception.
```suggestion
}
else
{
super.prepare(rb);
}
```
Or put another way, if we have (say) `lexical1` and `lexical2` as the keys of the queries to combine then there should be no third (`q`) parameter/key alongside that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2232834600
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
Review Comment:
We still need to populate all other params like Sort, returnFields, fq etc. to the parent `rq - requestBuilder` as this will be used in future processing and the scope of subBuilders are limited to query preparation and processing on searchIndexer level only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2185830584
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
Review Comment:
Am not familiar with RRF on partial results, if that is a concept? But wondering if conceptually it's up to the combiner to decide e.g.
```suggestion
List<Boolean> partialResults = new ArrayList<>(crb.responseBuilders.size());
List<Boolean> segmentTerminatedEarly = new ArrayList<>(crb.responseBuilders.size());
List<QueryResult> queryResults = new ArrayList<>(crb.responseBuilders.size());
```
and then later (pseudo code)
```
combinedPartialResults, combinedSegmentTerminatedEarly, combinedQueryResult = combinerStrategy.combine(partialResults, segmentTerminatedEarly, queryResults);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2185802810
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -1162,14 +1164,29 @@ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
// the docs offset -> queuesize
int resultSize = queue.size() - ss.getOffset();
resultSize = Math.max(0, resultSize); // there may not be any docs in range
-
Map<Object, ShardDoc> resultIds = new HashMap<>();
- for (int i = resultSize - 1; i >= 0; i--) {
- ShardDoc shardDoc = queue.pop();
- shardDoc.positionInResponse = i;
- // Need the toString() for correlation with other lists that must
- // be strings (like keys in highlighting, explain, etc)
- resultIds.put(shardDoc.id.toString(), shardDoc);
+
+ if (rb instanceof CombinedQueryResponseBuilder) {
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams());
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap);
+ maxScore = 0.0f;
+ for (int i = 0; i < resultSize; i++) {
+ ShardDoc shardDoc = combinedShardDocs.get(i);
+ shardDoc.positionInResponse = i;
+ maxScore = Math.max(maxScore, shardDoc.score);
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+ } else {
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
Review Comment:
Wondering if factoring out a protected `QueryComponent` method for this block (and the `resultSize` and `resultIds` above) would allow the `CombinedQueryComponent` to override the method, avoiding the need for `rb instanceof CombinedQueryResponseBuilder` above e.g.
```suggestion
Map<Object, ShardDoc> resultIds = createResultIds(queue, ss.getOffset());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198174747
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
Review Comment:
Highlighting and Faceting are separate components, so not affected, but as far as grouping is concerned, merge logic has to be there. Adding...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2201215313
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
Review Comment:
> ... Only the `CombinedQueryComponent` needs the multiple queries ResponseBuilder for multi-query execution.
From code reading it appears that if one wanted to have highlighting for the results being combined then the highlighting component would also need access.
But then again, perhaps that and various things could be initially deferred as out-of-scope (and documented as such) e.g. no combining with highlighting or faceting or cursor mark functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198422274
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
Review Comment:
We need to enforce some limit on the no of queries to avoid a burst of queries. Adding..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198174747
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
Review Comment:
Highlighting and Faceting are separate components, so not affected, but as far as grouping is concerned, merge logic has to be there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2199346683
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
Review Comment:
This kind of orchestration is already happening in [SearchHandler](https://github.com/ercsonusharma/solr/blob/9b0c76ef88a204829bbe1a2cb1cdec938d1b8586/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L513-L513) - IMO, iterating through each ResponseBuilder is not needed for every SearchComponent. Only the `CombinedQueryComponent` needs the multiple queries ResponseBuilder for multi-query execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2384806103
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
Review Comment:
Ah, thanks for pointing that out. Somehow, I didn't pay attention to th.e overall final score rather focused on getting the same results between single/multiple shards. There was a minor bug which is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3300226161
> IMO, If the data is independently sharded and shard-level scoring doesn't matter in overall query relevance, it may not make sense to first combine shard results based on their original scores and then apply RRF per query (as done in Way 2). In such cases, users may prefer Way 1.
I disagree. Only having per-shard RRF quickly devolves to shard interleaving as the shard count increases, since at the coordination/aggregation, there's no global overall ranking measure left anymore. The results from each shard are ultimately treated as equivalent across the shards. Consequently, if say the _real_ RRF best result pointed clearly to one document (best score of both sub-queries, lets say), then in the per-shard RRF it'd merely be _arbitrarily_ somewhere in the top-20 if say there are 20 shards. Nobody would want that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2352867177
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl =
+ String.format(Locale.ROOT, "%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
+ return;
+ }
Review Comment:
No, shortCircuit=false is not enough as it helps make [the rb.isDistrib=false ](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L449) and force the method to not go through the shardHandler requests, rather stick to [node local indexes](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L493) through SearchComponent.process method.
I couldn't find this code anywhere but created myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315316958
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ if (maxScore != null) responseDocs.setMaxScore(maxScore);
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ // save these results in a private area so we can access them
+ // again when retrieving stored fields.
+ // TODO: use ResponseBuilder (w/ comments) or the request context?
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly)
+ && Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb, Map<String, List<ShardDoc>> shardDocMap, SolrDocumentList responseDocs) {
+ String algorithm =
+ rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap, rb.req.getParams());
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
+ shardDocMap.forEach(
+ (shardKey, shardDocs) ->
+ shardDocs.forEach(shardDoc -> shardDocIdMap.put(shardDoc.id.toString(), shardDoc)));
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
+ float maxScore = 0.0f;
+ Sort sort = rb.getSortSpec().getSort();
+ SortField[] sortFields;
+ if (sort != null) {
+ sortFields = sort.getSort();
+ } else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+ final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(
+ sortFields,
+ rb.getSortSpec().getOffset() + rb.getSortSpec().getCount(),
+ rb.req.getSearcher());
+ combinedShardDocs.forEach(queue::insertWithOverflow);
Review Comment:
To sort the docs basis `sort` parameters if specified along with the new rrf score.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315324958
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
Review Comment:
Yes, it's only using JSON Query Parser as the `queries` key in the request has to be in a format that can be parsed using a JSON parser. I dont' see any way to parse multiple Solr Queries of different type using different parsers like lucene, dismax etc. Also, it's a long text ideal to be passed as JSON. [Here](https://github.com/apache/solr/pull/3418/files#diff-1ca21c13d967412ea6227fe257616a9cb2ac45012b207373a526d3767a46b588R145) the parsed query basis the QueryParser are used directly which is not possible without JSON Query DSL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2312543674
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
Review Comment:
why the name "rbNow"? The "rb" part is obvious but not "Now".
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors. In User-Managed (aka Standalone) Mode, the `shards` parameter must be provided to enable this feature.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
+
+```
+{
+ "queries": {
+ "lexical1": {
+ "lucene": {
+ "query": "title:sales"
+ }
+ },
+ "lexical2": {
+ "lucene": {
+ "query": "title:report"
+ }
+ },
+ "vector": {
+ "knn": {
+ "f": "vector",
+ "topK" :5,
+ "query": "[0.1,-0.34,0.89,0.02]"
+ }
+ }
+ },
+ "limit": 5,
+ "fields": ["id", "score", "title"],
+ "params": {
+ "combiner": true,
+ "combiner.query": ["lexical1", "vector"],
+ "combiner.algorithm": "rrf",
+ "combiner.rrf.k": "15"
+ }
+}
+```
+
+== Search Handler Configuration
+
+Combined Query Feature has a separate handler with class `solr.CombinedQuerySearchHandler` which can be configured as below:
+
+```
+<requestHandler name="/search" class="solr.CombinedQuerySearchHandler">
+<bool name="httpCaching">true</bool>
Review Comment:
why specify httpCaching here? It's not unique to CombinedQuerySearchHandler
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
Review Comment:
it's a shame that this long method is indented 1 level just because of this check (contributes to cyclomatic complexity). Maybe a extract-method refactoring could best resolve that...
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of lexical and vector queries using the combiner component.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
Review Comment:
Only 10 docs... this isn't very interesting. Ideally this distributed test shows the effects of a rows=K (something small) where the query matched more than K, and thus test that we get back the top-K out of more possible matches. This will influence faceting as well since faceting operates on the entire numDocs (DocSet) (not limited by k, AKA rows).
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
Review Comment:
This statement implies that it can only be used with JSON Query DSL. Is this true? If not; don't make any reference to it here (it's still okay to show examples as you have done). If it is, can you please refer me to a line of code somewhere that couples them? I don't see it.
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * This class implements a query and response combiner that uses the Reciprocal Rank Fusion (RRF)
+ * algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.DEFAULT_COMBINER_RRF_K;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ public int getK() {
+ return k;
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
Review Comment:
I see that only the DocList from each input QueryResult is being considered. The DocSet is ignored, and we instead pretend that the final/result DocSet is only those top documents as found in the resulting DocList. Consequently, DocSet consuming components (like faceting) will return a small subset of the total results/counts that they _should_ see. If the query matches very little (as in your test), this won't be noticed.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
Review Comment:
honestly you can skip this useless check since init() *will* be called first. We must rely on Solr to do what it's supposed to do.
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of lexical and vector queries using the combiner component.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("nullfirst", String.valueOf(i % 3));
Review Comment:
that's an odd/confusing choice of fields to use. Maybe use "mod3_idv"
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
Review Comment:
That is quite wrong. process() is local to the current index; it does *not* process responses from shards. We can say that we'll execute all of the queries provided and combine them. I would not speak of "shards" in this description.
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
That was a helpful response; thanks. But I see one *big* (yet easily fixed) issue... exactly what the final DocSet should be:
I see that only the DocList from each input QueryResult is being considered. The DocSet of each QueryResult is ignored, and we instead pretend that the final/result DocSet is only those top documents as found in the resulting DocList. Consequently, DocSet consuming components (like faceting) will return a small subset of the total results/counts that they should see. If the query matches very little (as in your test), this won't be noticed.
A fix would involve changing the method signature of this method to take the QueryResult (not just the DocList) of each query. And then to take the union of the DocSets of each using the utility methods on DocSets to do such. I'm assuming here that RRF doesn't filter out docs, it only ranks them.
##########
solr/core/src/test/org/apache/solr/search/combine/ReciprocalRankFusionTest.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import static org.apache.solr.common.params.CombinerParams.COMBINER_RRF_K;
+import static org.apache.solr.common.params.CombinerParams.RECIPROCAL_RANK_FUSION;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The ReciprocalRankFusionTest class is a unit test suite for the {@link ReciprocalRankFusion}
+ * class. It verifies the correctness of the fusion algorithm and its supporting methods.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class ReciprocalRankFusionTest extends SolrTestCaseJ4 {
Review Comment:
Let's test DocSet combining ;-)
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
Thanks Christine. The amount of code duplication of mergeIds, a massive method, is extreme! Something *must* be done; it'll be a burden to maintain otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315899041
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -22,7 +22,7 @@ It is extending JSON Query DSL ultimately enabling Hybrid Search.
[NOTE]
====
-This feature is currently unsupported for non-distributed query, grouping and Cursors.
+This feature is currently unsupported for grouping and Cursors. In User-Managed (aka Standalone) Mode, the `shards` parameter must be provided to enable this feature.
Review Comment:
okay. Will remove this statement from docs then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318785260
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
My understanding is that
* if (say) `lexical1` is for `title:sales` and there are (say) `numFound=500` results and
* if (say) `lexical2` is for `title:report` and there are (say) `numFound=300` results and
* we combine the two queries and use (say) `rows=10` and
* there are (say) 100 documents in common between the 500 and 300
* then we don't know what is in common i.e. we can only observe the overlap of (say) 3 documents amongst the 10 that each of `lexical1` and `lexical2` returned.
* So then `500 + 300 - 3` would be the calculation whereas `500 + 300 - 100` is the precise number if one were to interpret it as _"number of documents that matched either or both queries"_ or something along those lines.
This is based on code reading/interpretation only so far i.e. haven't yet tried it out locally and/or via a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3165339477
@aruggero Thanks for the review and posting the feedback. Here is the response to it:
> A) 1) The maxCombinerQueries parameter. This is a parameter with no "right" value to set; it would be better just to let the user manage by themselves.
Agreed, that there is no right value that's why it's upto user to choose that value through solrConfig. But, there has to be some default upper value set, right?
>A) 2) Supporting more than one combiner is not necessary by now since only Reciprocal Rank Fusion is currently present.
It's not necessary but good to have feature from beginnning itself so that can be extended later on through plugins.
>A) 3) The cursorMark usage for pagination could be something added a second time with a dedicated pull request. We could start as simply as possible.
Agreed and I didn't implement in a neat way (just propagated the cursorMark to all responseBuilders). Left it for other PR but I missed adding in the doc which I do so.
>B)Could you explain to us the overall general idea of your implementation? There are some points, as also mentioned by Christine, which are intricate and hard to review. Could you help us understand the purpose of each part and how they interact?
Did you mean general idea and purpose about the PR implementation? I can explain in detail if description doesn't help.
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3483724418
I'll handle the forceDistrib matter; it was my idea after all so I'll see to creating a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231134571
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.facet.FacetModule;
+
+/**
+ * The CombinedQuerySearchHandler class extends the SearchHandler and provides custom behavior for
+ * handling combined queries. It overrides methods to create a response builder based on the {@link
+ * CombinerParams#COMBINER} parameter and to define the default components included in the search
+ * configuration.
+ */
+public class CombinedQuerySearchHandler extends SearchHandler {
+
+ /**
+ * Overrides the default response builder creation method. This method checks if the {@link
+ * CombinerParams#COMBINER} parameter is set to true in the request. If it is, it returns an
+ * instance of {@link CombinedQueryResponseBuilder}, otherwise, it returns an instance of {@link
+ * ResponseBuilder}.
+ *
+ * @param req the SolrQueryRequest object
+ * @param rsp the SolrQueryResponse object
+ * @param components the list of SearchComponent objects
+ * @return the appropriate ResponseBuilder instance based on the CombinerParams.COMBINER parameter
+ */
+ @Override
+ protected ResponseBuilder newResponseBuilder(
+ SolrQueryRequest req, SolrQueryResponse rsp, List<SearchComponent> components) {
+ if (req.getParams().getBool(CombinerParams.COMBINER, false)) {
+ return new CombinedQueryResponseBuilder(req, rsp, components);
+ }
+ return new ResponseBuilder(req, rsp, components);
Review Comment:
minor/subjective
```suggestion
return super.newResponseBuilder(req, rsp, components);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231192975
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
Review Comment:
observation: use of `computeIfAbsent` here means that
* any configured combiner silently takes precedence over the in-built one, and
* that if the same combiner key is configured more than once then the second and subsequent ones will be silently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2235495661
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
Yeah, There was a scope of refactoring here. I didn't intend to change anything in QueryComponent, and rather duplicating the code to some extent as it may impact the existing component as recommended by Atri.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2591563892
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
The original intention was to group the *Handler and *Component classes within the same package, as done previously.
It will affect [this](https://github.com/apache/solr/pull/3418/files#diff-c857e7327c8c35700144083caf8d079b87622fa67aabad3de51728e32287f16bR110) where we might deviate from usual convention of having `solr.<ComponentName>` & `solr.<HandlerName>`
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
The original intention was to group the *Handler and *Component classes within the same package, as done previously.
It will affect [this](https://github.com/apache/solr/pull/3418/files#diff-c857e7327c8c35700144083caf8d079b87622fa67aabad3de51728e32287f16bR110) where we might deviate from usual convention of having solr.<ComponentName> & solr.<HandlerName>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2186691717
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
Review Comment:
RRF should just merge multiple doc results irrespective of whether they are partial or complete IMHO. If any of the ResponseBuilder QueryResults contain partial results, the whole merged QueryResults should be marked as `partialResults`. Same should be the case with `segmentTerminatedEarly`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2185784001
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
Review Comment:
Not seen this (newer) java pattern matching with `instanceof` before, nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2187355067
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -1162,14 +1164,29 @@ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
// the docs offset -> queuesize
int resultSize = queue.size() - ss.getOffset();
resultSize = Math.max(0, resultSize); // there may not be any docs in range
-
Map<Object, ShardDoc> resultIds = new HashMap<>();
- for (int i = resultSize - 1; i >= 0; i--) {
- ShardDoc shardDoc = queue.pop();
- shardDoc.positionInResponse = i;
- // Need the toString() for correlation with other lists that must
- // be strings (like keys in highlighting, explain, etc)
- resultIds.put(shardDoc.id.toString(), shardDoc);
+
+ if (rb instanceof CombinedQueryResponseBuilder) {
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams());
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap);
+ maxScore = 0.0f;
+ for (int i = 0; i < resultSize; i++) {
+ ShardDoc shardDoc = combinedShardDocs.get(i);
+ shardDoc.positionInResponse = i;
+ maxScore = Math.max(maxScore, shardDoc.score);
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+ } else {
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
Review Comment:
Thanks for the input. This makes sense to me, and I have refactored out the method to leverage overriding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2232845710
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
Review Comment:
- It's the other way round. The inbuilt combiner with same reserved name `rrf` will take precedence.
- any custom combiner shouldn't have same name as one with reserved name `rrf`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3484388424
> I think my fundamental concerns are addressed. I left some comments.
Thank you, @dsmiley for the extensive review. I have addressed those comments as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2261037169
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
Review Comment:
I thought about this and then checked other components like [FacetModule](https://github.com/ercsonusharma/solr/blob/8041d66aa9536fbd5a5e9c9dc26d0df38ff656b8/solr/core/src/java/org/apache/solr/search/facet/FacetModule.java#L45-L45) where `_` is being used. So, I am confused here as to what the convention is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315991714
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
Review Comment:
refactored
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2312594150
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -22,7 +22,7 @@ It is extending JSON Query DSL ultimately enabling Hybrid Search.
[NOTE]
====
-This feature is currently unsupported for non-distributed query, grouping and Cursors.
+This feature is currently unsupported for grouping and Cursors. In User-Managed (aka Standalone) Mode, the `shards` parameter must be provided to enable this feature.
Review Comment:
Yet all of the tests you added run in this mode (not SolrCloud), and most don't provide "shards". In particular `CombinedQueryComponentTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2296307185
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
okay. But, that (`shortCircuit=false`) needs to be passed along with `distrib=true` as [per](https://github.com/apache/solr/blob/releases/solr/9.9.0/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L324) - can assert through a test.
UPDATE: above method isn't helping as it requires the `shards` parameter too and there is no http server in `SolrTestCaseJ4` class. So, it should be good as long as `shards` parameter is passed in standalone mode as can be seen in the [test cases](https://github.com/apache/solr/pull/3418/files#diff-d18c0ebaeee21519f08c727c3202b4bc8d47a592e886102b07dc99af0a0b48b4R172). The limitation can be resolved by passing the `shards` param explicitly in the handler for standalone (aka User Managed) mode [added in doc].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2341551234
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
Review Comment:
True, but I also wanted to show the results are not the same as what was expected in a single shard, as I described in the jira issue. Asserting only ideal queries (like constant/fixed score) or a type of queries (lexical with ExactStatsCache) may not give a clear picture to the algorithm IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3617328838
BTW I'm busy tweaking some things and preparing a ~final state that's mergeable.
One thing I'm doing is reducing the forcedDistrib API as it's rather internal.
Also merging from main.
Adding changelog.
Updating javadocs; usually to simplify / remove boilerplate likely AI written stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2434411532
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl =
+ String.format(Locale.ROOT, "%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
+ return;
+ }
Review Comment:
Ah. I see so it needs to happen _somewhere_, and you've put it here. It feels out of place in this method, however. Based on what you showed me, it could be in HttpShardHandler where StandaloneReplicaSource.Builder is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322075842
##########
solr/core/src/java/org/apache/solr/handler/component/FacetComponent.java:
##########
@@ -82,6 +82,14 @@ public void prepare(ResponseBuilder rb) throws IOException {
rb.setNeedDocSet(true);
rb.doFacets = true;
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ crb.responseBuilders.forEach(
+ thisRb -> {
+ thisRb.setNeedDocSet(true);
+ thisRb.doFacets = true;
+ });
+ }
+
Review Comment:
well done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2314378342
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ if (maxScore != null) responseDocs.setMaxScore(maxScore);
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ // save these results in a private area so we can access them
+ // again when retrieving stored fields.
+ // TODO: use ResponseBuilder (w/ comments) or the request context?
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly)
+ && Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb, Map<String, List<ShardDoc>> shardDocMap, SolrDocumentList responseDocs) {
+ String algorithm =
+ rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap, rb.req.getParams());
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
+ shardDocMap.forEach(
+ (shardKey, shardDocs) ->
+ shardDocs.forEach(shardDoc -> shardDocIdMap.put(shardDoc.id.toString(), shardDoc)));
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
+ float maxScore = 0.0f;
+ Sort sort = rb.getSortSpec().getSort();
+ SortField[] sortFields;
+ if (sort != null) {
+ sortFields = sort.getSort();
+ } else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+ final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(
+ sortFields,
+ rb.getSortSpec().getOffset() + rb.getSortSpec().getCount(),
+ rb.req.getSearcher());
+ combinedShardDocs.forEach(queue::insertWithOverflow);
Review Comment:
Could you help me understand why the combined shard docs need to go through the sorting queue here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315036138
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
Review Comment:
There can be multiple ResponseBuilder in the `crb` or `rb` - "Now" refers to the current ResponseBuilder inside the `crb.responseBuilders` which is being referenced in the for loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286964590
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
> Solr DocSet is used to hold all doc IDs that match a query, ignoring rows or start params
Absolutely Right!
> It's only for a non-distrib request (distrib=false)
It's for distributed as well as distributed request, both have to go through this phase through ShardRequest. Only difference being that it sets the start=0 and row+=start. Then, final merge happens after collecting the results from all ShardRequests. Am I right?
This method is invoked right after the SolrSearchIndexer call for all the sub-queries. It's not creating DocSet rather modifying the existing one, returned from SolrSearchIndexer. The complete flow is like:
1. Prepare all the sub-queries in terms of parsing basis the QueryParser.
2. Process all the sub-queries by sending it across the SolrSearchIndexer which will return the `QueryResult` having `DocSet`.
3. The `ReciprocalRankFusion` then rescores them and creates a modified `DocSet`.
Let me know if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286964590
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
> Solr DocSet is used to hold all doc IDs that match a query, ignoring rows or start params
Absolutely Right!
> It's only for a non-distrib request (distrib=false)
It's for distributed as well as distributed request, both have to go through this phase through ShardHandlerRequest. Only difference being that it sets the start=0 and row+=start. Then, final merge happens after collecting the results from all ShardHandlers. Am I right?
This method is invoked right after the SolrSearchIndexer call for all the sub-queries. It's not creating DocSet rather modifying the existing one, returned from SolrSearchIndexer. The complete flow is like:
1. Prepare all the sub-queries in terms of parsing basis the QueryParser.
2. Process all the sub-queries by sending it across the SolrSearchIndexer which will return the `QueryResult` having `DocSet`.
3. The `ReciprocalRankFusion` then rescores them and creates a modified `DocSet`.
Let me know if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286964590
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
> Solr DocSet is used to hold all doc IDs that match a query, ignoring rows or start params
Absolutely Right!
> It's only for a non-distrib request (distrib=false)
It's for distributed as well as distributed request, both have to go through this phase through ShardRequest. Only difference being that it sets the start=0 and row+=start. Then, final merge happens after collecting the results from all ShardRequests. Is that correct?
This method is invoked right after the SolrSearchIndexer call for all the sub-queries. It's not creating DocSet rather modifying the existing one, returned from SolrSearchIndexer. The complete flow is like:
1. Prepare all the sub-queries in terms of parsing basis the QueryParser.
2. Process all the sub-queries by sending it across the SolrSearchIndexer which will return the `QueryResult` having `DocSet`.
3. The `ReciprocalRankFusion` then rescores them and creates a modified `DocSet`.
Let me know if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2288168710
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
Users will be as confused as i was reading this. **If** you mean "standalone mode" then just say that, plus say "AKA User managed") in parenthesis after, as this is the IMO unfortunate documented term for it. BUT, I doubt you actually mean this. I didn't notice SolrCloud specific logic in your PR; is it there? Your response didn't indicate if this would work for a SolrCloud single-shard collection. Will it? Your test subclassed BaseDistributedSearchTestCase which compares equivalency in responses between a single shard and multi-shard. Doesn't use SolrCloud.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2294115027
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
> ... For a single-sharded collection (or single core standalone mode), distrib will be false. ...
From what I recall, if a user wanted to use the `distrib=true` code path in a single-sharded collection (e.g. because the component in use does not or not-yet support the `distrib=false` code path) then they could pass in `shortCircuit=false` based on this: https://github.com/apache/solr/blob/releases/solr/9.9.0/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2296307185
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
okay. But, that (`shortCircuit=false`) needs to be passed along with `distrib=true` as [per](https://github.com/apache/solr/blob/releases/solr/9.9.0/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L324) - can assert through a test.
UPDATE: above method isn't helping as it requires the `shards` parameter too and there is no http server in `SolrTestCaseJ4` class. So, it should be good as long as `shards` parameter is passed in standalone mode as can be seen in the [test cases](https://github.com/apache/solr/pull/3418/files#diff-d18c0ebaeee21519f08c727c3202b4bc8d47a592e886102b07dc99af0a0b48b4R172). The limitation can be resolved by passing the `shards` param explicitly in the handler for standalone (aka User Managed) mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3053424774
@alessandrobenedetti @dsmiley, please help review it whenever you can. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2591563892
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
The original intention was to group the *Handler and *Component classes within the same package, as done previously. However, there is no issue with placing them in the combine package instead. Please let me know if you would like me to move them there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2591607613
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
The original intention was to group the *Handler and *Component classes within the same package, as done previously.
It will affect [this](https://github.com/apache/solr/pull/3418/files#diff-c857e7327c8c35700144083caf8d079b87622fa67aabad3de51728e32287f16bR110) where we might deviate from usual convention of having `solr.<ComponentName>` & `solr.<HandlerName>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3649615238
@dsmiley - Thanks for the PR! It looks good on my end. What do you think the next step should be, considering the other PR has been open for quite a few days now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2296307185
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
okay. But, that (`shortCircuit=false`) needs to be passed along with `distrib=true` as [per](https://github.com/apache/solr/blob/releases/solr/9.9.0/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L324)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2353470681
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
Review Comment:
I concur that in a realistic setting, sharding will yield score variances or ties that could change the final ordering. But why test under that circumstance? It only makes testing harder and for a reader of the tests to understand that the assertions are valid. _We won't have less test coverage by fixing the scores somehow._ I think the vast majority of tests here should operate in this circumstance, which makes them clear to understand / validate. This should save you time (less tests to maintain).
If the way RRF tie brakes matters (I doubt it), I could understand adding such a test to ensure we don't introduce a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2282280820
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
Review Comment:
@ercsonusharma could you explain to me what totalMatches represents and why it is computed in this way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286964590
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
> Solr DocSet is used to hold all doc IDs that match a query, ignoring rows or start params
Absolutely Right!
> It's only for a non-distrib request (distrib=false)
It's for distributed as well as distributed request have also to go through this phase through ShardHandlerRequest. Only difference being that it sets the start=0 and row+=start. Then, final merge happens after collecting the results from all ShardHandlers. Am I right?
This method is invoked right after the SolrSearchIndexer call for all the sub-queries. It's not creating DocSet rather modifying the existing one, returned from SolrSearchIndexer. The complete flow is like:
1. Prepare all the sub-queries in terms of parsing basis the QueryParser.
2. Process all the sub-queries by sending it across the SolrSearchIndexer which will return the `QueryResult` having `DocSet`.
3. The `ReciprocalRankFusion` then rescores them and creates a modified `DocSet`.
Let me know if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286964590
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
> Solr DocSet is used to hold all doc IDs that match a query, ignoring rows or start params
Absolutely Right!
> It's only for a non-distrib request (distrib=false)
It's for distributed as well as distributed request, both have to go through this phase, distributed goes through ShardRequest. Only difference being that it sets the start=0 and row+=start. Then, final merge happens after collecting the results from all ShardRequests. Is that correct?
This method is invoked right after the SolrSearchIndexer call for all the sub-queries. It's not creating DocSet rather modifying the existing one, returned from SolrSearchIndexer. The complete flow is like:
1. Prepare all the sub-queries in terms of parsing basis the QueryParser.
2. Process all the sub-queries by sending it across the SolrSearchIndexer which will return the `QueryResult` having `DocSet`.
3. The `ReciprocalRankFusion` then rescores them and creates a modified `DocSet`.
Let me know if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2284370141
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed and grouping query. Using Cursors is not advisable, as it may produce inconsistent results.
Review Comment:
Added cursors as non-supported functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198432554
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
Review Comment:
Then, only docs.size() number of documents are ranked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2201498073
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
Review Comment:
Multiple Queries are executed in `CombinedQueryComponent` (just queries) which is set inside the `queries` field in JSON query DSL. After that, all the other components like Faceting and highlighting happens from `(CombinedQuery)ResponseBuilder` builder.
> From code reading it appears that if one wanted to have highlighting for the results being combined then the highlighting component would also need access.
Not exactly, the highlighting components works by highlighting on the [rb.getResults()](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/HighlightComponent.java#L162) set which is already create in the CombinedQueryComponent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2203163245
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
Review Comment:
ShardDoc is local to the mergeIds method and not shared in any other object available to other component. Also, `SolrDocumentList` is created later using ShardDoc. Please help me understand if it's being shared anywhere else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2294774649
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
Ah yes; thanks Christine. In fact at my last job, we configured that to be so all the time for certain reasons. It'd be reasonable for the documentation for this thing to prescribe setting that in the request handler definition. But I'd like to see a test of this component that would fail without this being set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2310667088
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
#3565 explores factoring something out in QueryComponent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3239019175
Since everything looks good, may I request approval so we can proceed with the merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2332148796
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -81,4 +88,33 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /**
+ * rb.distrib {@link ResponseBuilder} must be set for the combined query to work in case of single
+ * core standalone mode. This method set the parameter explicitly with other required solr param
+ * i.e. shards
+ *
+ * @param req the SolrQueryRequest
+ * @return boolean denoting whether the request can be marked as distributed.
+ */
+ @Override
+ protected boolean isDistrib(SolrQueryRequest req) {
+ boolean isDistrib = super.isDistrib(req);
+ if (!isDistrib
+ && !req.getParams().getBool(ShardParams.IS_SHARD, false)
+ && req.getHttpSolrCall() != null) {
+ log.info("Configuring distributed mode to enable Combined Query.");
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl = String.format("%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
Review Comment:
I followed what you said [here](https://github.com/apache/solr/pull/3418#discussion_r2322678396):
I agree it's a bit sneaky. However, Apologies, but yet I didn't follow the part below:
> for a component or handler to communicate, we need the distributed search algorithm (no so-called short-circuit).
afaik, in standalone distributed search, we need the shards value to be passed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3254540627
> The beauty/wisdom of BaseDistributedSearchTestCase is that it tests consistency between single shard and multi-shard. I think it's brilliant; that is the point of this base class.
Yet this PR/approach will not be able to comply since unlike most (all?) components, its results are affected substantially by distributed-search. The (unsaid?) vision of sharding / distributed-search was getting the same results as a single shard, and Solr does the work to pull off that trick, with plenty of tests demonstrating it does. In fact I'd say, with great disappointment, that the observed (by a user) results of this component will _not_ be RRF when there's distributed search over shards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315324958
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
Review Comment:
Yes, it's only using JSON Query Parser as the `queries` key in the request has to be in a format that can be parsed using a JSON parser. I dont' see any way to parse multiple Solr Queries of different type using different parasers like lucene, dismax etc. Also, it's a long text ideal to be passed as JSON.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2316189319
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
Would love to also see a sample response for illustrative purposes.
open question (for me at least) - do we need to document something about the meaning of `numFound` in the response and how it is computed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3245354980
Can you please "resolve" any conversation you think were addressed? This is a long PR with many conversations, making it hard to catch up with the current state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2316184710
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
+
+```
+{
+ "queries": {
+ "lexical1": {
+ "lucene": {
+ "query": "title:sales"
+ }
+ },
+ "lexical2": {
+ "lucene": {
+ "query": "title:report"
+ }
+ },
+ "vector": {
+ "knn": {
+ "f": "vector",
+ "topK" :5,
+ "query": "[0.1,-0.34,0.89,0.02]"
+ }
+ }
+ },
+ "limit": 5,
+ "fields": ["id", "score", "title"],
+ "params": {
+ "combiner": true,
+ "combiner.query": ["lexical1", "vector"],
Review Comment:
Was this meant to be `lexical2` here and/or can `lexical2` above be omitted if `vector` is the second query?
```suggestion
"combiner.query": ["lexical1", "lexical2"],
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2314366325
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ if (maxScore != null) responseDocs.setMaxScore(maxScore);
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ // save these results in a private area so we can access them
+ // again when retrieving stored fields.
+ // TODO: use ResponseBuilder (w/ comments) or the request context?
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly)
+ && Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb, Map<String, List<ShardDoc>> shardDocMap, SolrDocumentList responseDocs) {
+ String algorithm =
+ rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap, rb.req.getParams());
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
+ shardDocMap.forEach(
+ (shardKey, shardDocs) ->
+ shardDocs.forEach(shardDoc -> shardDocIdMap.put(shardDoc.id.toString(), shardDoc)));
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
+ float maxScore = 0.0f;
+ Sort sort = rb.getSortSpec().getSort();
+ SortField[] sortFields;
+ if (sort != null) {
+ sortFields = sort.getSort();
+ } else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+ final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(
+ sortFields,
+ rb.getSortSpec().getOffset() + rb.getSortSpec().getCount(),
+ rb.req.getSearcher());
+ combinedShardDocs.forEach(queue::insertWithOverflow);
+ int resultSize = queue.size() - rb.getSortSpec().getOffset();
+ resultSize = Math.max(0, resultSize);
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ maxScore = Math.max(maxScore, shardDoc.score);
+ if (Float.isNaN(shardDocIdMap.get(shardDoc.id.toString()).score)) {
+ shardDoc.score = Float.NaN;
+ }
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+ responseDocs.setMaxScore(maxScore);
Review Comment:
Might we just use the `maxScore` already set at line 474 above?
I note that in the existing query component the `maxScore` is computed prior to consideration of the offset and that for combined queries there probably isn't really a meaningful `maxScore` meaning since different queries' scores' ranges could vary quite a bit.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2319177772
##########
solr/core/src/java/org/apache/solr/handler/component/FacetComponent.java:
##########
@@ -82,6 +82,14 @@ public void prepare(ResponseBuilder rb) throws IOException {
rb.setNeedDocSet(true);
rb.doFacets = true;
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ crb.responseBuilders.forEach(
+ thisRb -> {
+ thisRb.setNeedDocSet(true);
+ thisRb.doFacets = true;
+ });
+ }
+
Review Comment:
Please see a method created `propagate` to take care of this.
- Please see a test for highlighter where we need to provide a query in hl.
- We are combining the docSet using union which would help coombining the facets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "atris (via GitHub)" <gi...@apache.org>.
atris commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2195755106
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * responses. It provides a framework for different algorithms to be implemented for merging ranked
+ * lists and shard documents.
+ */
+public abstract class QueryAndResponseCombiner {
+
+ protected int upTo;
+
+ /**
+ * Constructs a QueryAndResponseCombiner instance.
+ *
+ * @param requestParams the SolrParams containing the request parameters.
+ */
+ protected QueryAndResponseCombiner(SolrParams requestParams) {
+ this.upTo =
+ requestParams.getInt(CombinerParams.COMBINER_UP_TO, CombinerParams.COMBINER_UP_TO_DEFAULT);
+ }
+
+ /**
+ * Combines multiple ranked lists into a single QueryResult.
+ *
+ * @param rankedLists a list of ranked lists to be combined
+ * @return a new QueryResult containing the combined results
+ * @throws IllegalArgumentException if the input list is empty
+ */
+ public abstract QueryResult combine(List<QueryResult> rankedLists);
+
+ /**
+ * Combines shard documents based on the provided map.
+ *
+ * @param shardDocMap a map where keys represent shard IDs and values are lists of ShardDocs for
+ * each shard
+ * @return a combined list of ShardDocs from all shards
+ */
+ public abstract List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap);
+
+ /**
+ * Retrieves a list of explanations for the given queries and results.
+ *
+ * @param queryKeys the keys associated with the queries
+ * @param queries the list of queries for which explanations are requested
+ * @param resultsPerQuery the list of document lists corresponding to each query
+ * @param searcher the SolrIndexSearcher used to perform the search
+ * @param schema the IndexSchema used to interpret the search results
+ * @return a list of explanations for the given queries and results
+ * @throws IOException if an I/O error occurs during the explanation retrieval process
+ */
+ public abstract NamedList<Explanation> getExplanations(
+ String[] queryKeys,
+ List<Query> queries,
+ List<DocList> resultsPerQuery,
+ SolrIndexSearcher searcher,
+ IndexSchema schema)
+ throws IOException;
+
+ /**
+ * Retrieves an implementation of the QueryAndResponseCombiner based on the specified algorithm.
+ *
+ * @param requestParams the SolrParams containing the request parameters, including the combiner
+ * algorithm.
+ * @return an instance of QueryAndResponseCombiner corresponding to the specified algorithm.
+ * @throws SolrException if an unknown combiner algorithm is specified.
+ */
+ public static QueryAndResponseCombiner getImplementation(SolrParams requestParams) {
+ String algorithm =
+ requestParams.get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.RECIPROCAL_RANK_FUSION);
+ if (algorithm.equals(CombinerParams.RECIPROCAL_RANK_FUSION)) {
Review Comment:
This is hardcoded - why not have a Plugin interface here, allowing dynamic plugin loaded here?
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * responses. It provides a framework for different algorithms to be implemented for merging ranked
+ * lists and shard documents.
+ */
+public abstract class QueryAndResponseCombiner {
+
+ protected int upTo;
+
+ /**
+ * Constructs a QueryAndResponseCombiner instance.
+ *
+ * @param requestParams the SolrParams containing the request parameters.
+ */
+ protected QueryAndResponseCombiner(SolrParams requestParams) {
+ this.upTo =
+ requestParams.getInt(CombinerParams.COMBINER_UP_TO, CombinerParams.COMBINER_UP_TO_DEFAULT);
+ }
+
+ /**
+ * Combines multiple ranked lists into a single QueryResult.
+ *
+ * @param rankedLists a list of ranked lists to be combined
+ * @return a new QueryResult containing the combined results
+ * @throws IllegalArgumentException if the input list is empty
+ */
+ public abstract QueryResult combine(List<QueryResult> rankedLists);
+
+ /**
+ * Combines shard documents based on the provided map.
+ *
+ * @param shardDocMap a map where keys represent shard IDs and values are lists of ShardDocs for
+ * each shard
+ * @return a combined list of ShardDocs from all shards
+ */
+ public abstract List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap);
+
+ /**
+ * Retrieves a list of explanations for the given queries and results.
+ *
+ * @param queryKeys the keys associated with the queries
+ * @param queries the list of queries for which explanations are requested
+ * @param resultsPerQuery the list of document lists corresponding to each query
+ * @param searcher the SolrIndexSearcher used to perform the search
+ * @param schema the IndexSchema used to interpret the search results
+ * @return a list of explanations for the given queries and results
+ * @throws IOException if an I/O error occurs during the explanation retrieval process
+ */
+ public abstract NamedList<Explanation> getExplanations(
Review Comment:
Please implement support for debug as well
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
Review Comment:
I wonder if we can abstract the subquery propagated execution in a separate class:
```
public class SubQueryExecutor {
private final SolrQueryRequest sharedReq;
private final List<SearchComponent> components;
public SubQueryExecutor(SolrQueryRequest req, List<SearchComponent> components) {
this.sharedReq = req;
this.components = components;
}
public void execute(List<ResponseBuilder> builders) throws IOException {
for (ResponseBuilder rb : builders) {
for (SearchComponent c : components) {
c.prepare(rb); // or distributedPrepare
}
}
for (ResponseBuilder rb : builders) {
for (SearchComponent c : components) {
c.process(rb); // or distributedProcess
}
}
}
}
```
This will avoid nesting like ` super.prepare(rbNew); `
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
Review Comment:
How does this work with grouping, highlighting and faceting? Those methods from QueryComponent are not overridden here, so updated ResponseBuilders are not propagated there.
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
Review Comment:
This is dangerous - this is mutating the original ShardDoc object. It might be referred to by another component, and is a bad idea to modify in place.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= SolrQueryResponse.isPartialResults(rbNow.rsp.getResponseHeader());
+ rbNow.setCursorMark(rbNow.getCursorMark());
+ if (rbNow.rsp.getResponseHeader() != null) {
+ segmentTerminatedEarly |=
+ (boolean)
+ rbNow
+ .rsp
+ .getResponseHeader()
+ .getOrDefault(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, false);
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams());
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults);
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT, crb.getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb,
+ int resultSize,
+ ShardFieldSortedHitQueue queue,
+ Map<String, List<ShardDoc>> shardDocMap,
+ SolrDocumentList responseDocs) {
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams());
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap);
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
+ shardDocMap.forEach(
+ (shardKey, shardDocs) ->
+ shardDocs.forEach(shardDoc -> shardDocIdMap.put(shardDoc.id.toString(), shardDoc)));
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
+ float maxScore = 0.0f;
+ for (int i = 0; i < resultSize; i++) {
+ ShardDoc shardDoc = combinedShardDocs.get(i);
+ shardDoc.positionInResponse = i;
+ maxScore = Math.max(maxScore, shardDoc.score);
Review Comment:
How does this normalise across different query types (KNN, BM25,, filters)?
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
Review Comment:
QueryComponent is specifically designed for Solr's distributed search processing. We override prepare method, but then invoke super.prepare with the sub response. This could quickly get uncontrolled for a query with large number of clauses.
I would suggest overriding SearchComponent and defining explicit subBuilder.process and subBuilder.prepare methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2331961665
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -81,4 +88,33 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /**
+ * rb.distrib {@link ResponseBuilder} must be set for the combined query to work in case of single
+ * core standalone mode. This method set the parameter explicitly with other required solr param
+ * i.e. shards
+ *
+ * @param req the SolrQueryRequest
+ * @return boolean denoting whether the request can be marked as distributed.
+ */
+ @Override
+ protected boolean isDistrib(SolrQueryRequest req) {
+ boolean isDistrib = super.isDistrib(req);
+ if (!isDistrib
+ && !req.getParams().getBool(ShardParams.IS_SHARD, false)
+ && req.getHttpSolrCall() != null) {
+ log.info("Configuring distributed mode to enable Combined Query.");
Review Comment:
not worth logging; or maybe trace
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -81,4 +88,33 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /**
+ * rb.distrib {@link ResponseBuilder} must be set for the combined query to work in case of single
+ * core standalone mode. This method set the parameter explicitly with other required solr param
+ * i.e. shards
+ *
+ * @param req the SolrQueryRequest
+ * @return boolean denoting whether the request can be marked as distributed.
+ */
+ @Override
+ protected boolean isDistrib(SolrQueryRequest req) {
+ boolean isDistrib = super.isDistrib(req);
+ if (!isDistrib
+ && !req.getParams().getBool(ShardParams.IS_SHARD, false)
+ && req.getHttpSolrCall() != null) {
+ log.info("Configuring distributed mode to enable Combined Query.");
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl = String.format("%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
Review Comment:
I think it's a bit sneaky that a predicate looking method has a side-effect. This is a hack to work around a need for something proper -- for a component or handler to communicate we need the distributed search algorithm (no so-called short-circuit).
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
Review Comment:
Please update this test to not call `queryServer` since it doesn't compare against the control client. I wish that method came with a big fat disclaimer and was named differently that didn't look so normal. There are probably more callers of it than there should be. I did https://github.com/apache/solr/pull/3639 just now.
In order for this test to compare both single shard & multi-shard with score relevancy, you could do a couple different things. One is to use the ExactStatsCache for distributed-IDF. *Or*, write queries that set a constant/fixed score. That would actually be most clear.
With such a test, you don't need to test non-distributed since the test infrastructure here allows you to do both as one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2282705396
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
Review Comment:
ResponseBuilder consists of just`SolrQueryRequest` and `SolrQueryResponse` in this case which will any way be needed for standard queries. All the built-in method uses the parameters for querying using `ResponseBuilder` which makes convenient to re-use them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2282674479
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.facet.FacetModule;
+
+/**
+ * The CombinedQuerySearchHandler class extends the SearchHandler and provides custom behavior for
+ * handling combined queries. It overrides methods to create a response builder based on the {@link
+ * CombinerParams#COMBINER} parameter and to define the default components included in the search
+ * configuration.
+ */
+public class CombinedQuerySearchHandler extends SearchHandler {
+
+ /**
+ * Overrides the default response builder creation method. This method checks if the {@link
+ * CombinerParams#COMBINER} parameter is set to true in the request. If it is, it returns an
+ * instance of {@link CombinedQueryResponseBuilder}, otherwise, it returns an instance of {@link
+ * ResponseBuilder}.
+ *
+ * @param req the SolrQueryRequest object
+ * @param rsp the SolrQueryResponse object
+ * @param components the list of SearchComponent objects
+ * @return the appropriate ResponseBuilder instance based on the CombinerParams.COMBINER parameter
+ */
+ @Override
+ protected ResponseBuilder newResponseBuilder(
+ SolrQueryRequest req, SolrQueryResponse rsp, List<SearchComponent> components) {
+ if (req.getParams().getBool(CombinerParams.COMBINER, false)) {
+ return new CombinedQueryResponseBuilder(req, rsp, components);
+ }
+ return super.newResponseBuilder(req, rsp, components);
+ }
+
+ /**
+ * Overrides the default components and returns a list of component names that are included in the
+ * default configuration.
+ *
+ * @return a list of component names
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ protected List<String> getDefaultComponents() {
+ List<String> names = new ArrayList<>(9);
+ names.add(CombinedQueryComponent.COMPONENT_NAME);
Review Comment:
I don't think so. In the case of SearchHandler, we are using `QueryComponent` as the first component.
##########
solr/core/src/test/org/apache/solr/search/combine/ReciprocalRankFusionTest.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import static org.apache.solr.common.params.CombinerParams.COMBINER_RRF_K;
+import static org.apache.solr.common.params.CombinerParams.RECIPROCAL_RANK_FUSION;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The ReciprocalRankFusionTest class is a unit test suite for the {@link ReciprocalRankFusion}
+ * class. It verifies the correctness of the fusion algorithm and its supporting methods.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class ReciprocalRankFusionTest extends SolrTestCaseJ4 {
+
+ public static ReciprocalRankFusion reciprocalRankFusion;
+
+ /**
+ * Initializes the test environment by setting up the {@link ReciprocalRankFusion} instance with
+ * specific parameters.
+ */
+ @BeforeClass
+ public static void beforeClass() {
+ NamedList<?> args = new NamedList<>(Map.of("k", "20"));
+ reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(args);
+ }
+
+ /** Tests the functionality of combining using RRF across local search indices. */
+ @Test
+ public void testSearcherCombine() {
+ List<QueryResult> rankedList = getQueryResults();
+ SolrParams solrParams = params(COMBINER_RRF_K, "10");
+ QueryResult result = reciprocalRankFusion.combine(rankedList, solrParams);
+ assertEquals(20, reciprocalRankFusion.getK());
+ assertEquals(3, result.getDocList().size());
+ }
+
+ private static List<QueryResult> getQueryResults() {
+ QueryResult r1 = new QueryResult();
+ r1.setDocList(
+ new DocSlice(
+ 0,
+ 2,
+ new int[] {1, 2},
+ new float[] {0.67f, 0, 62f},
Review Comment:
typo. corrected.
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed and grouping query. Using Cursors is not advisable, as it may produce inconsistent results.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner.enable` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
+
+```
+{
+ "queries": {
+ "lexical1": {
+ "lucene": {
+ "query": "title:sales"
+ }
+ },
+ "lexical2": {
+ "lucene": {
+ "query": "title:report"
+ }
+ },
+ "vector": {
+ "knn": {
+ "f": "vector",
+ "topK" :5,
+ "query": "[0.1,-0.34,0.89,0.02]"
+ }
+ }
+ },
+ "limit": 5,
+ "fields": ["id", "score", "title"],
+ "params": {
+ "combiner": true,
Review Comment:
typo. corrected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231263967
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
Review Comment:
subjective/clarity: the params being passed to both `getImplementation` and to `combine` jumps out.
Would it be sufficient to pass the algorithm parameter?
```suggestion
QueryAndResponseCombiner combinerStrategy =
QueryAndResponseCombiner.getImplementation(rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER), combiners);
QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2232834600
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
Review Comment:
We still need to populate all other params like Sort, returnFields etc. to the parent rq - requestBuilder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2256571786
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
Review Comment:
Perhaps it would be better to align the nomenclature with that of the other components, which do not use underscores.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
Following Christine's comment, this method is very hard to follow, and it needs a refactor.
Could you give us a general overview of the logic within this code? What should it do? Which elements should it use and how, etc?
Thanks
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
Review Comment:
In general, it would be better to use an "expanded" version of variable names to make them more readable in the code...
srsp, nl, rhste, rhath, etc..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286898422
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
non-distributed (means Standalone non-SolrCloud mode, distrib=false and when *paginated*) we need some work in the upcoming PR, where similar logic (as in the case of distrib=true) of fetching all docs right from start and selecting the r rows based on `ShardFieldSortedHitQueue` needs to be implemented. As SortSpec applies on all queries in the set globally and not individually.
I should have mentioned non-distributed with pagination?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2453916641
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
Review Comment:
Sorry, took longer to reply.
The forced distributed would anyway enable the shard call [(isShard=true)](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L565).
> then why wouldn't the logic just pass-through as normal
The `process()` method is directly communicating with the Searcher as we need to process each query at the shard searcher level. If you would look at the method override, it is also creating the combined docSet across queries other than few other metadata, required for merging the docs finally across the shard at the coordinator level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2489065508
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
+ }
+
+ /**
+ * Test multiple query execution with sort.
+ *
+ * @throws Exception the exception
+ */
+ @Test
+ public void testMultipleQueryWithSort() throws Exception {
Review Comment:
> "Consistency of the query structure"
I meant user creating the query for a use case. They don't have to use two query types - Boolean specifically for sorting and Combined for others.
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
+ }
+
+ /**
+ * Test multiple query execution with sort.
+ *
+ * @throws Exception the exception
+ */
+ @Test
+ public void testMultipleQueryWithSort() throws Exception {
Review Comment:
> "Consistency of the query structure"
I meant user creating the query for a use case. They don't have to use two query types - Boolean specifically for sorting and Combined for others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3668594281
Workflow failures seem likely to be infrastructure-related and a flaky SolrCloud test case. Re-triggering the workflow may help it pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3675296776
@dsmiley - I see it failed again after re-triggering the same workflows. Luckily, I found a tiny cleanup to commit. Can you help approve the workflow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley merged PR #3418:
URL: https://github.com/apache/solr/pull/3418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2596378463
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -86,4 +51,13 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /** Overrides the default list to include {@link CombinedQueryComponent}. */
+ @Override
+ protected List<String> getDefaultComponents() {
+ List<String> names = new ArrayList<>(super.getDefaultComponents());
+ String replaced = names.set(0, CombinedQueryComponent.COMPONENT_NAME);
+ assert replaced.equals(QueryComponent.COMPONENT_NAME);
Review Comment:
This is merely the *default* component list. It doesn't stop someone from configuring a list in solrconfig.xml that is different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2591563892
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
The original intention was to group the *Handler and *Component classes within the same package, as done previously. However, there is no issue with placing them in the combine package instead.
It will affect [this](https://github.com/apache/solr/pull/3418/files#diff-c857e7327c8c35700144083caf8d079b87622fa67aabad3de51728e32287f16bR110) where we might deviate from usual convention of having `solr.<ComponentName>` & `solr.<HandlerName>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2319049507
##########
solr/core/src/java/org/apache/solr/handler/component/FacetComponent.java:
##########
@@ -82,6 +82,14 @@ public void prepare(ResponseBuilder rb) throws IOException {
rb.setNeedDocSet(true);
rb.doFacets = true;
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ crb.responseBuilders.forEach(
+ thisRb -> {
+ thisRb.setNeedDocSet(true);
+ thisRb.doFacets = true;
+ });
+ }
+
Review Comment:
> ... Maybe CombinedQueryComponent could at some point after the other components to their preparation, it could propagate settings from parent to children using new code to do so on CombinedQueryResponseBuilder. ...
I had a similar thought.
Thinking specifically about the faceting component though, I'm curious how that interacts with combined queries:
* in the case of the highlighting component, if we have (say) `title:sales` and `title:report` as the two lexical queries being combined then we wish for the matching term to be highlighted in each document.
* in the case of the faceting component, faceting by (say) some language id field that the document has, how would the facet counts from the two queries be combined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2319184435
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
> and we can adjust the response to indicate the inexactness if more were underlying results were present.
True, it's already being marked as part of this [code](https://github.com/apache/solr/pull/3418/files#diff-9ee2d3ca53340b37b597cf2666844ce73ea534416ead953dddc15c5399458282R173) `numExactCount` would be false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2288385230
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
Agreed. I think it's confusing and I don't find a right term yet. But, here is the [code line](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L493) which is getting set by method [isDistrib(SolrQueryRequest req)](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L307). Would you be able to help here with right term, may be `Standalone (AKA user managed)` as suggested?
I am using `shards` param to query in my test case which uses `BaseDistributedSearchTestCase` (or I could have used distrib=true).
> Your response didn't indicate if this would work for a SolrCloud single-shard collection. Will it?
Yes, it would work for single shard or multi shard both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322185934
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
Review Comment:
Yeah, that's for SolrCloud and/or distributed search. But, for User Managed mode where we don't pass `shards` param, the pagination would not work for now and that's the limitation I was trying to establish [here](https://github.com/apache/solr/pull/3418#discussion_r2312593325).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2322678396
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
Review Comment:
I think user-managed/standalone vs SolrCloud is orthogonal. This is about a single shard working correctly (in whatever Solr mode). IMO it's not optional for basic paging parameters to work correctly with one shard.
I could imagine we'd prefer a mechanism for a SearchComponent to force the "shortCircuit"=false thereby ensuring there's always a distributed phase. Maybe that could be done by re-ordering SearchHandler's call to `getAndPrepShardHandler` to be _after_ `prepareComponents` (swap adjacent lines)? Then the prepare method of this component could force distrib and add the shortCircuit=false or something like that. And/or maybe a component should have a more elegant callback to communicate that it forces distributed search (even when there's one shard/core). This would overall simplify this component, no longer needing to handle paging in process(); instead do for distributed-search only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2321606729
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_START;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
+ * It verifies the functionality of the component by performing various queries and validating the
+ * responses.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
Review Comment:
Somehow tests were failing and I see this as a fix referring other tests. But, have removed now and see if test fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2352867177
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl =
+ String.format(Locale.ROOT, "%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
+ return;
+ }
Review Comment:
No, shortCircuit=false is not enough as ShardHandler also [needs the `shards` parameter](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L483) which comes directly from SolrParams. While, making shortCircuit=true makes [the rb.isDistrib=false ](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L449) and force the method to not go through the shardHandler requests, rather stick to [node local indexes](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L493) through SearchComponent.process method.
I couldn't find this code anywhere but created myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231523759
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
A little more context for this thought above ---^
The `QueryComponent.mergeIds` function is quite intricate and am curious about ways of factoring something out to minimise code duplication. By replacing `srsp.getShards()` with `shardDoc.shard` here it is more apparent that conceptually `shardDocMap` is a shard doc consumer and perhaps a shard doc consumer supplier style approach could work.
pseudo-code for `QueryComponent`
```
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ implementMergeIds(rb, sreq, ...) {
+ }
+
+ protected void implementMergeIds(ResponseBuilder rb, ShardRequest sreq,
+ Supplier< Consumer<ShardDoc> > shardDocConsumerSupplier) {
...
- final ShardFieldSortedHitQueue queue =
- new ShardFieldSortedHitQueue(
- sortFields, ss.getOffset() + ss.getCount(), rb.req.getSearcher());
+ final Consumer<ShardDoc> shardDocConsumer = shardDocConsumerSupplier.get();
...
- queue.insertWithOverflow(shardDoc);
+ shardDocConsumer.accept(shardDoc);
```
pseudo-code for `CombinedQueryComponent`
```
// supplier returns shardDocMap
// Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
- shardDocMap.computeIfAbsent(shardDoc.shard, list -> new ArrayList<>()).add(shardDoc);
+ shardDocConsumer.accept(shardDoc);
```
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2232845106
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
Review Comment:
can't use this due to type `NamedList<?> args` doesn't allow call `getOrDefault(...)` without an unchecked cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2196062773
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -1234,6 +1230,34 @@ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
}
}
+ /**
+ * Creates a map of shard results based on the provided parameters.
+ *
+ * @param rb ResponseBuilder to retrieve any request info
+ * @param resultSize the desired size of the result map
+ * @param queue the queue containing sorted hits from different shards
+ * @param shardDocMap a map of shard documents indexed by shard ID
+ * @param responseDocs the final SolrDocumentList to be returned or set in the responseBuilder
+ * @return a map of shard documents, where the keys are the shard IDs as strings, and the values
+ * are the corresponding ShardDoc objects
+ */
+ protected Map<Object, ShardDoc> createShardResult(
Review Comment:
Shouldn't the key be a String and not an Object?
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -1234,6 +1230,34 @@ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
}
}
+ /**
+ * Creates a map of shard results based on the provided parameters.
+ *
+ * @param rb ResponseBuilder to retrieve any request info
+ * @param resultSize the desired size of the result map
+ * @param queue the queue containing sorted hits from different shards
+ * @param shardDocMap a map of shard documents indexed by shard ID
+ * @param responseDocs the final SolrDocumentList to be returned or set in the responseBuilder
+ * @return a map of shard documents, where the keys are the shard IDs as strings, and the values
+ * are the corresponding ShardDoc objects
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb,
+ int resultSize,
+ ShardFieldSortedHitQueue queue,
+ Map<String, List<ShardDoc>> shardDocMap,
+ SolrDocumentList responseDocs) {
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
Review Comment:
See `org.apache.solr.common.util.CollectionUtil#newHashMap` and pre-size
##########
solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java:
##########
@@ -240,7 +241,7 @@ public void changed(SolrPackageLoader.SolrPackage pkg, Ctx ctx) {
}
@SuppressWarnings({"unchecked"})
- private void initComponents() {
+ private void initComponents(boolean isCombinedQuery) {
Review Comment:
Strongly agree with the smell!
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * responses. It provides a framework for different algorithms to be implemented for merging ranked
Review Comment:
results & responses -- seem synonymous to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198420584
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
Review Comment:
As usual, User-specified top N is being enforced at the shard level [here](https://github.com/ercsonusharma/solr/blob/9b0c76ef88a204829bbe1a2cb1cdec938d1b8586/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L1163-L1163)
and search index level at SolrIndexSearcher by setting the nums and offset in SortSpec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198427144
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
Review Comment:
Going to add user input for ordering the docs across multiple queries in case of tie.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286898422
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
non-distributed (means Standalone non-SolrCloud mode, distrib=false and when *paginated*) we need some work in the upcoming PR, where similar logic (as in the case of distrib=true) of fetching all docs right from start and selecting the r rows based on `ShardFieldSortedHitQueue` needs to be implemented.
I should have mentioned non-distributed with pagination?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "atris (via GitHub)" <gi...@apache.org>.
atris commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2292769134
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
No, I am fine with whatever approach guarantees no breakage in QueryComponent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2319184435
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
> and we can adjust the response to indicate the inexactness if more were underlying results were present.
True, it's already being marked as part of this [code](https://github.com/apache/solr/pull/3418/files#diff-9ee2d3ca53340b37b597cf2666844ce73ea534416ead953dddc15c5399458282R173) `numFoundExact ` would be false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318929806
##########
solr/core/src/java/org/apache/solr/handler/component/FacetComponent.java:
##########
@@ -82,6 +82,14 @@ public void prepare(ResponseBuilder rb) throws IOException {
rb.setNeedDocSet(true);
rb.doFacets = true;
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ crb.responseBuilders.forEach(
+ thisRb -> {
+ thisRb.setNeedDocSet(true);
+ thisRb.doFacets = true;
+ });
+ }
+
Review Comment:
This is unfortunate.
Could `CombinedQueryResponseBuilder.setNeedDocSet` be overridden to propagate to children? That leaves `doFacets`; maybe a setter cold be added similarly.
OR,
Maybe CombinedQueryComponent could at some point after the other components to their preparation, it could propagate settings from parent to children using new code to do so on CombinedQueryResponseBuilder.
I recognize ResponseBuilder is a sloppy mess of entanglement. I don't think it has ever been cleaned up and re-thought out.
##########
solr/core/src/java/org/apache/solr/handler/component/FacetComponent.java:
##########
@@ -82,6 +82,14 @@ public void prepare(ResponseBuilder rb) throws IOException {
rb.setNeedDocSet(true);
rb.doFacets = true;
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ crb.responseBuilders.forEach(
+ thisRb -> {
+ thisRb.setNeedDocSet(true);
+ thisRb.doFacets = true;
+ });
+ }
+
Review Comment:
I could imagine ResponseBuilder having a method like `copyTo(ResponseBuilder other)`. Albeit certain fields relate to the query indirectly, and that's what varies across the queries you are combining. The CombinedQueryResponseBuilder could arrange for saving those, and reinstating them after copying them.
##########
solr/core/src/java/org/apache/solr/handler/component/HighlightComponent.java:
##########
@@ -97,6 +97,13 @@ public void prepare(ResponseBuilder rb) throws IOException {
rb.doHighlights = solrConfigHighlighter.isHighlightingEnabled(params);
if (rb.doHighlights) {
rb.setNeedDocList(true);
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
Review Comment:
same feedback as for facets
##########
solr/core/src/java/org/apache/solr/handler/component/HighlightComponent.java:
##########
@@ -97,6 +97,13 @@ public void prepare(ResponseBuilder rb) throws IOException {
rb.doHighlights = solrConfigHighlighter.isHighlightingEnabled(params);
if (rb.doHighlights) {
rb.setNeedDocList(true);
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
Review Comment:
And there are other components; for some reason these two alone have your attention but a number of them use the ResponseBuilder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2317761872
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ if (maxScore != null) responseDocs.setMaxScore(maxScore);
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ // save these results in a private area so we can access them
+ // again when retrieving stored fields.
+ // TODO: use ResponseBuilder (w/ comments) or the request context?
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly)
+ && Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb, Map<String, List<ShardDoc>> shardDocMap, SolrDocumentList responseDocs) {
+ String algorithm =
+ rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap, rb.req.getParams());
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
+ shardDocMap.forEach(
+ (shardKey, shardDocs) ->
+ shardDocs.forEach(shardDoc -> shardDocIdMap.put(shardDoc.id.toString(), shardDoc)));
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
+ float maxScore = 0.0f;
+ Sort sort = rb.getSortSpec().getSort();
+ SortField[] sortFields;
+ if (sort != null) {
+ sortFields = sort.getSort();
+ } else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+ final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(
+ sortFields,
+ rb.getSortSpec().getOffset() + rb.getSortSpec().getCount(),
+ rb.req.getSearcher());
+ combinedShardDocs.forEach(queue::insertWithOverflow);
+ int resultSize = queue.size() - rb.getSortSpec().getOffset();
+ resultSize = Math.max(0, resultSize);
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ maxScore = Math.max(maxScore, shardDoc.score);
+ if (Float.isNaN(shardDocIdMap.get(shardDoc.id.toString()).score)) {
+ shardDoc.score = Float.NaN;
+ }
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+ responseDocs.setMaxScore(maxScore);
Review Comment:
`maxScore` is the score after rrf compuation across queries which will be normalised score of all the docs across queries and shards and it has to be added at later step after merging the shardDocs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2320851066
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * This class implements a query and response combiner that uses the Reciprocal Rank Fusion (RRF)
+ * algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.DEFAULT_COMBINER_RRF_K;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ public int getK() {
+ return k;
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocListAndSet> docListAndSet = getDocListsAndSetFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docListAndSet, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocListAndSet> getDocListsAndSetFromQueryResults(
Review Comment:
minor: IMO this method isn't worthwhile. Instead, just pass the `List<QueryResult>` to combineResults. That method need not be separated either; I think just inline it.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries in the SearchIndexer, combines them using the
Review Comment:
What is the "SearchIndexer"? I think you could drop this part of the sentence.
BTW it's best to `{@link classname}` so make class names hyperlinked
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_START;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
Review Comment:
this is not a unit test
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries in the SearchIndexer, combines them using the
+ * specified QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in
+ * the CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder thisRb : crb.responseBuilders) {
Review Comment:
TODO (future) parallelize
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
Review Comment:
wouldn't we want to manipulate the sort spec so that we get all docs up to offset (AKA "start" param) + rows since RRF/combiner is going to want to see all docs/rankings up to offset+rows? Otherwise our combiner is blind to the "offset" docs. Assuming you agree, then we need to basically apply paging at this layer (our component) instead of letting the subquery do it.
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of lexical and vector queries using the combiner component.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp;
+ rsp =
+ queryServer(
+ createParams(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^=10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ "shards",
+ getShardsString(),
+ CommonParams.QT,
+ "/search"));
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the Solr server.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp;
+ rsp =
+ queryServer(
+ createParams(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"title:title test for doc 1\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"text:test text for doc 2\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}",
+ "shards",
+ getShardsString(),
+ CommonParams.QT,
+ "/search"));
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2", "1", "4", "5", "8");
+ }
+
+ /**
+ * Test multiple query execution with sort.
+ *
+ * @throws Exception the exception
+ */
+ @Test
+ public void testMultipleQueryWithSort() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp;
+ rsp =
+ queryServer(
+ createParams(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"title:title test for doc 1\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"text:test text for doc 2\"}}},"
+ + "\"limit\":5,\"sort\":\"mod3_idv desc\""
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}",
+ "shards",
+ getShardsString(),
+ CommonParams.QT,
+ "/search"));
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2", "8", "5", "4", "1");
+ }
+
+ /**
+ * Tests the hybrid query functionality of the system.
+ *
+ * @throws Exception if any unexpected error occurs during the test execution.
+ */
+ @Test
+ public void testHybridQueryWithPagination() throws Exception {
Review Comment:
This test (or another test?) should be more comprehensive so that it's easier to validate that pagination is in fact working. I have doubts based on the code, and the test itself doesn't help at all.
For example, first do the query without pagination (return ~5 docs). Then (same test!), do again (same query) with pagination using an offset & limit that slice into the first request. And maybe one more time; different page. It'll be straight-forward to reason about valid pagination results given the first set of results.
As a reviewer I look at the current test as-is and I have no idea if it's working.
I don't care if the test is "hybrid" or not, so long as there are 2 sub-queries thus exercising typical logic.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries in the SearchIndexer, combines them using the
+ * specified QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in
+ * the CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder thisRb : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ thisRb.setCursorMark(crb.getCursorMark());
+ super.process(thisRb);
+ DocListAndSet docListAndSet = thisRb.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ prepareCombinedResponseBuilder(
+ rb, crb, queryResults, partialResults, segmentTerminatedEarly, setMaxHitsTerminatedEarly);
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ private void prepareCombinedResponseBuilder(
+ ResponseBuilder rb,
+ CombinedQueryResponseBuilder crb,
+ List<QueryResult> queryResults,
+ boolean partialResults,
+ boolean segmentTerminatedEarly,
+ boolean setMaxHitsTerminatedEarly)
+ throws IOException {
+ String algorithm =
+ rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
Review Comment:
I think you should specifically look at `isDebugResults()`
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * This class implements a query and response combiner that uses the Reciprocal Rank Fusion (RRF)
+ * algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.DEFAULT_COMBINER_RRF_K;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ public int getK() {
+ return k;
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocListAndSet> docListAndSet = getDocListsAndSetFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docListAndSet, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocListAndSet> getDocListsAndSetFromQueryResults(
+ List<QueryResult> rankedLists) {
+ List<DocListAndSet> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocListAndSet());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
Review Comment:
could use some javadocs to explain what we're doing here. AFAIKT, this is changing the score of the docs to be the RRF score. The order appears to be interleaved since the score gets reset for each list, and because doc IDs are unique per shard. This is very non-obvious reading the code.
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.plugin.NamedListInitializedPlugin;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * shard documents. It provides a framework for different algorithms to be implemented for merging
+ * ranked lists and shard documents.
+ */
+public abstract class QueryAndResponseCombiner implements NamedListInitializedPlugin {
+ /**
+ * Combines multiple ranked lists into a single QueryResult.
+ *
+ * @param rankedLists a list of ranked lists to be combined
+ * @param solrParams params to be used when provided at query time
+ * @return a new QueryResult containing the combined results
+ * @throws IllegalArgumentException if the input list is empty
+ */
+ public abstract QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams);
+
+ /**
+ * Combines shard documents based on the provided map.
+ *
+ * @param shardDocMap a map where keys represent shard IDs and values are lists of ShardDocs for
+ * each shard
+ * @param solrParams params to be used when provided at query time
+ * @return a combined list of ShardDocs from all shards
+ */
+ public abstract List<ShardDoc> combine(
+ Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams);
+
+ /**
+ * Retrieves a list of explanations for the given queries and results.
+ *
+ * @param queryKeys the keys associated with the queries
+ * @param queries the list of queries for which explanations are requested
+ * @param queryResult the list of QueryResult corresponding to each query
+ * @param searcher the SolrIndexSearcher used to perform the search
+ * @param schema the IndexSchema used to interpret the search results
+ * @param solrParams params to be used when provided at query time
+ * @return a list of explanations for the given queries and results
+ * @throws IOException if an I/O error occurs during the explanation retrieval process
+ */
+ public abstract NamedList<Explanation> getExplanations(
Review Comment:
Should be a SimpleOrderedMap if the keys are unique. Presumably "queryKeys" are unique, albeit your docs don't say if its so
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_START;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
+ * It verifies the functionality of the component by performing various queries and validating the
+ * responses.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
Review Comment:
why this threadLeakScope? Javadoc says it's highly discouraged, which I believe.
It's also on the other test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2321569807
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * This class implements a query and response combiner that uses the Reciprocal Rank Fusion (RRF)
+ * algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.DEFAULT_COMBINER_RRF_K;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ public int getK() {
+ return k;
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocListAndSet> docListAndSet = getDocListsAndSetFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docListAndSet, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocListAndSet> getDocListsAndSetFromQueryResults(
+ List<QueryResult> rankedLists) {
+ List<DocListAndSet> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocListAndSet());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
Review Comment:
Yes, it would be interleaved in this case as the each shard has unique docs.
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.plugin.NamedListInitializedPlugin;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * shard documents. It provides a framework for different algorithms to be implemented for merging
+ * ranked lists and shard documents.
+ */
+public abstract class QueryAndResponseCombiner implements NamedListInitializedPlugin {
+ /**
+ * Combines multiple ranked lists into a single QueryResult.
+ *
+ * @param rankedLists a list of ranked lists to be combined
+ * @param solrParams params to be used when provided at query time
+ * @return a new QueryResult containing the combined results
+ * @throws IllegalArgumentException if the input list is empty
+ */
+ public abstract QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams);
+
+ /**
+ * Combines shard documents based on the provided map.
+ *
+ * @param shardDocMap a map where keys represent shard IDs and values are lists of ShardDocs for
+ * each shard
+ * @param solrParams params to be used when provided at query time
+ * @return a combined list of ShardDocs from all shards
+ */
+ public abstract List<ShardDoc> combine(
+ Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams);
+
+ /**
+ * Retrieves a list of explanations for the given queries and results.
+ *
+ * @param queryKeys the keys associated with the queries
+ * @param queries the list of queries for which explanations are requested
+ * @param queryResult the list of QueryResult corresponding to each query
+ * @param searcher the SolrIndexSearcher used to perform the search
+ * @param schema the IndexSchema used to interpret the search results
+ * @param solrParams params to be used when provided at query time
+ * @return a list of explanations for the given queries and results
+ * @throws IOException if an I/O error occurs during the explanation retrieval process
+ */
+ public abstract NamedList<Explanation> getExplanations(
Review Comment:
It's [SimpleOrderedMap](https://github.com/apache/solr/pull/3418/files#diff-9ee2d3ca53340b37b597cf2666844ce73ea534416ead953dddc15c5399458282R207) only
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,598 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
Review Comment:
It anyways happening [here](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L842)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318895113
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
I think this feature should work with the [minExactCount](https://solr.apache.org/guide/solr/latest/query-guide/common-query-parameters.html#minexactcount-parameter) param in this regard. If it's fully exact, then docSets need to be requested and we simply get the cardinality (size) of the final docSet. When docSets are not enabled, then we're only capable of accurately counting up to "rows", and we can adjust the response to indicate the inexactness if more were underlying results were present.
> This is also true for a single query which are spread over multiple shards.
No; a given doc (identified by its ID) may not be in more than one shard. If a user using implicit routing were to arrange for that to happen; it's _unsupported_ (the user _should not_ do that). There's some trivial dedupe logic you saw but there's no real support for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318978519
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
> My next intended step is to look further into the `resultIds` and `responseDocs` interaction and to better understand the `CombinedQueryComponent.createShardResult` implementation.
https://github.com/apache/solr/pull/3418/commits/195f3f1c96d609456a3b844ee878755fd1a40ed3 factors out a protected `QueryComponent.setResultIdsAndResponseDocs` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2312593325
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
AFAICT there's no issue here. If you see an issue, then can you suggest a local hack change to your test and I see for myself the problem hands-on? Or maybe a new test would need to be developed to show the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318605879
##########
solr/core/src/test/org/apache/solr/search/combine/ReciprocalRankFusionTest.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import static org.apache.solr.common.params.CombinerParams.COMBINER_RRF_K;
+import static org.apache.solr.common.params.CombinerParams.RECIPROCAL_RANK_FUSION;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The ReciprocalRankFusionTest class is a unit test suite for the {@link ReciprocalRankFusion}
+ * class. It verifies the correctness of the fusion algorithm and its supporting methods.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class ReciprocalRankFusionTest extends SolrTestCaseJ4 {
Review Comment:
Isn't it being tested as part of [this](https://github.com/apache/solr/pull/3418/files#diff-9f3a36d6ae6c7df2f303bdd1ec89f6fdc2d7187ee215ce08072ef277048b311cR64). Do you want me to test the DocSet combining explicitly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318601809
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of lexical and vector queries using the combiner component.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
Review Comment:
I added a [testcase](https://github.com/apache/solr/pull/3418/files#diff-d18c0ebaeee21519f08c727c3202b4bc8d47a592e886102b07dc99af0a0b48b4R243) to assert this behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2352636387
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -21,39 +21,59 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.util.SimpleOrderedMap;
import org.junit.BeforeClass;
import org.junit.Test;
/**
* The CombinedQueryComponentTest class is an integration test suite for the CombinedQueryComponent
- * in Solr. It verifies the functionality of the component by performing few basic queries and
- * validating the responses including limitations and combiner plugin.
+ * in Solr. It verifies the functionality of the component by performing few basic queries in single
+ * sharded mode and validating the responses including limitations and combiner plugin.
*/
-public class CombinedQueryComponentTest extends SolrTestCaseJ4 {
+public class CombinedQueryComponentTest extends BaseDistributedSearchTestCase {
Review Comment:
I'm now confused on how to differentiate the testing approach between this class and DistributedCombinedQueryComponentTest. _That_ one is named appropriately (consistently with other distributed tests) and extends BaseDistributedSearchTestCase. I'm not sure *this* test has a role/purpose if that one can be comprehensive.
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl =
+ String.format(Locale.ROOT, "%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
+ return;
+ }
Review Comment:
This part looks suspicious to me; so shortCircuit=false isn't enough? Did you construct this based on seeing similar code elsewhere (where?) to create a shards URL?
##########
solr/core/src/java/org/apache/solr/handler/component/ResponseBuilder.java:
##########
@@ -141,6 +141,15 @@ public ResponseBuilder(
public List<ShardRequest> outgoing; // requests to be sent
public List<ShardRequest> finished; // requests that have received responses from all shards
public String shortCircuitedURL;
+ private boolean forcedDistrib = false;
Review Comment:
I thought of this as well, yet it feels weird to put a request aspect into the response data holder. It doesn't affect the response. Any way... it's really minor. This is pragmatic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2354613676
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
Review Comment:
Alright, I have removed the Non-distributed test case separately using `queryServer` and now, using `query` directly with constant score queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2384826680
##########
solr/core/src/test/org/apache/solr/handler/component/DistributedCombinedQueryComponentTest.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The DistributedCombinedQueryComponentTest class is a JUnit test suite that evaluates the
+ * functionality of the CombinedQueryComponent in a Solr distributed search environment. It focuses
+ * on testing the integration of combiner queries with different configurations.
+ */
+public class DistributedCombinedQueryComponentTest extends BaseDistributedSearchTestCase {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and setting system properties. This method is
+ * executed before all test methods in the class.
+ *
+ * @throws Exception if any exception occurs during initialization
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ System.setProperty("validateAfterInactivity", "200");
+ System.setProperty("solr.httpclient.retries", "0");
+ System.setProperty("distribUpdateSoTimeout", "5000");
+ }
+
+ /**
+ * Prepares Solr input documents for indexing, including adding sample data and vector fields.
+ * This method populates the Solr index with test data, including text, title, and vector fields.
+ * The vector fields are used to calculate cosine distance for testing purposes.
+ *
+ * @throws Exception if any error occurs during the indexing process.
+ */
+ private synchronized void prepareIndexDocs() throws Exception {
+ List<SolrInputDocument> docs = new ArrayList<>();
+ fixShardCount(2);
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ doc.addField("mod3_idv", (i % 3));
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
+ // cosine distance vector1= 0.554
+ docs.get(8).addField(vectorField, Arrays.asList(200f, 50f, 100f, 25f));
+ // cosine distance vector1= 0.997
+ docs.get(9).addField(vectorField, Arrays.asList(1.8f, 2.5f, 3.7f, 4.9f));
+ del("*:*");
+ clients.sort(
+ (client1, client2) -> {
+ try {
+ if (client2 instanceof HttpSolrClient httpClient2
+ && client1 instanceof HttpSolrClient httpClient1)
+ return new URI(httpClient1.getBaseURL()).getPort()
+ - new URI(httpClient2.getBaseURL()).getPort();
+ } catch (URISyntaxException e) {
+ throw new RuntimeException("Unable to get URI from SolrClient", e);
+ }
+ return 0;
+ });
+ for (SolrInputDocument doc : docs) {
+ indexDoc(doc);
+ }
+ commit();
+ }
+
+ /**
+ * Tests a single lexical query against the Solr server using both combiner methods.
+ *
+ * @throws Exception if any exception occurs during the test execution
+ */
+ @Test
+ public void testSingleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ QueryResponse rsp =
+ query(
+ CommonParams.JSON,
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:2^10\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\"]}}",
+ CommonParams.QT,
+ "/search");
+ assertEquals(1, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "2");
+ }
+
+ @Override
+ protected String getShardsString() {
+ if (deadServers == null) return shards;
+ Arrays.sort(shardsArr);
+ StringBuilder sb = new StringBuilder();
+ for (String shard : shardsArr) {
+ if (!sb.isEmpty()) sb.append(',');
+ sb.append(shard);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Tests multiple lexical queries using the distributed solr client.
+ *
+ * @throws Exception if any error occurs during the test execution
+ */
+ @Test
+ public void testMultipleLexicalQuery() throws Exception {
+ prepareIndexDocs();
+ String jsonQuery =
+ "{\"queries\":"
+ + "{\"lexical1\":{\"lucene\":{\"query\":\"id:(2^2 OR 3^1 OR 6^2 OR 5^1)\"}},"
+ + "\"lexical2\":{\"lucene\":{\"query\":\"id:(4^2 OR 5^1 OR 7^3 OR 10^2)\"}}},"
+ + "\"limit\":5,"
+ + "\"fields\":[\"id\",\"score\",\"title\"],"
+ + "\"params\":{\"combiner\":true,\"combiner.query\":[\"lexical1\",\"lexical2\"]}}";
+ QueryResponse rsp = query(CommonParams.JSON, jsonQuery, CommonParams.QT, "/search");
+ assertEquals(5, rsp.getResults().size());
+ assertFieldValues(rsp.getResults(), id, "5", "4", "2", "3", "7");
+ }
+
+ /**
+ * Test multiple query execution with sort.
+ *
+ * @throws Exception the exception
+ */
+ @Test
+ public void testMultipleQueryWithSort() throws Exception {
Review Comment:
Sorting can also be applied to the results after processing RRF; otherwise, there is no built-in support for joining two queries. Sorting should be supported in case of combined query to maintain consistency with the query structure. The final RRF score would act as a tie breaker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2434634317
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
Review Comment:
In the case of distributed mode, an HTTP request is made with rb.distrib=false through ShardHandlerRequest in one of the phases. The `process `method would be called for that particular shard query request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2591563892
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
The original intention was to group the *Handler and *Component classes within the same package, as done previously.
It will affect [this](https://github.com/apache/solr/pull/3418/files#diff-c857e7327c8c35700144083caf8d079b87622fa67aabad3de51728e32287f16bR110) where we might deviate from usual convention of having `solr.<ComponentName>` & `solr.<HandlerName>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3232953304
Hi @ercsonusharma,
We reviewed everything again with @alessandrobenedetti.
In general, the current status of the contribution is ok for us; we agree with the observations of the other reviewers.
About previously highlighted points:
- About the maxQueries parameter, the discussion is complicated, so ok. Let's keep it as it is.
- About supporting more than one combiner, since this is not adding too much complexity to the contribution, let's keep it as it is.
Is it worth adding some additional tests to check the behaviour of this new query component with highlighting/faceting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "atris (via GitHub)" <gi...@apache.org>.
atris commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2195790595
##########
solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java:
##########
@@ -240,7 +241,7 @@ public void changed(SolrPackageLoader.SolrPackage pkg, Ctx ctx) {
}
@SuppressWarnings({"unchecked"})
- private void initComponents() {
+ private void initComponents(boolean isCombinedQuery) {
Review Comment:
This is a code smell - initComponents should not be changing behaviour based on a flag specific to a component.
This would be solved if we inherited from SearchHandler or dynamically injected CombinedQueryComponent using a factory pattern
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
Review Comment:
This is assuming that each query returns upTo number of documents - what happens when a query returns lesser number of documents?
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
Review Comment:
How about early termination for non competitive iterators?
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
Review Comment:
upTo limit is per query, not a global top N. In fusion part, we return all unique docs across all subqueries. Where are we enforcing user specified top N limit?
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
Review Comment:
We should add tests for queries returning no results and score ties
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private final int k;
+
+ /**
+ * Constructs a ReciprocalRankFusion instance.
+ *
+ * @param requestParams the SolrParams containing the configuration parameters for this combiner.
+ */
+ public ReciprocalRankFusion(SolrParams requestParams) {
+ super(requestParams);
+ this.k =
+ requestParams.getInt(CombinerParams.COMBINER_RRF_K, CombinerParams.COMBINER_RRF_K_DEFAULT);
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false);
+ return combinedResult;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap) {
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size() && ranking <= upTo) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ // TODO: Add the remaining items out of upTo limit to the docIdToScore
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext() && ranking <= upTo) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (k + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
Review Comment:
This is essentially number of queries * upto. Have we scale tested this?
##########
solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.junit.BeforeClass;
+
+/**
+ * The CombinedQueryComponentTest class is a unit test suite for the CombinedQueryComponent in Solr.
+ * It verifies the functionality of the component by performing various queries and validating the
+ * responses.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class CombinedQueryComponentTest extends SolrTestCaseJ4 {
+
+ private static final int NUM_DOCS = 10;
+ private static final String vectorField = "vector";
+
+ /**
+ * Sets up the test class by initializing the core and adding test documents to the index. This
+ * method prepares the Solr index with a set of documents for subsequent test cases.
+ *
+ * @throws Exception if any error occurs during setup, such as initialization failures or indexing
+ * issues.
+ */
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ initCore("solrconfig-combined-query.xml", "schema-vector-catchall.xml");
+ List<SolrInputDocument> docs = new ArrayList<>();
+ for (int i = 1; i <= NUM_DOCS; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", Integer.toString(i));
+ doc.addField("text", "test text for doc " + i);
+ doc.addField("title", "title test for doc " + i);
+ docs.add(doc);
+ }
+ // cosine distance vector1= 1.0
+ docs.get(0).addField(vectorField, Arrays.asList(1f, 2f, 3f, 4f));
+ // cosine distance vector1= 0.998
+ docs.get(1).addField(vectorField, Arrays.asList(1.5f, 2.5f, 3.5f, 4.5f));
+ // cosine distance vector1= 0.992
+ docs.get(2).addField(vectorField, Arrays.asList(7.5f, 15.5f, 17.5f, 22.5f));
+ // cosine distance vector1= 0.999
+ docs.get(3).addField(vectorField, Arrays.asList(1.4f, 2.4f, 3.4f, 4.4f));
+ // cosine distance vector1= 0.862
+ docs.get(4).addField(vectorField, Arrays.asList(30f, 22f, 35f, 20f));
+ // cosine distance vector1= 0.756
+ docs.get(5).addField(vectorField, Arrays.asList(40f, 1f, 1f, 200f));
+ // cosine distance vector1= 0.970
+ docs.get(6).addField(vectorField, Arrays.asList(5f, 10f, 20f, 40f));
+ // cosine distance vector1= 0.515
+ docs.get(7).addField(vectorField, Arrays.asList(120f, 60f, 30f, 15f));
Review Comment:
Please add test for the REF score calculation explainability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2198428280
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= SolrQueryResponse.isPartialResults(rbNow.rsp.getResponseHeader());
+ rbNow.setCursorMark(rbNow.getCursorMark());
+ if (rbNow.rsp.getResponseHeader() != null) {
+ segmentTerminatedEarly |=
+ (boolean)
+ rbNow
+ .rsp
+ .getResponseHeader()
+ .getOrDefault(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, false);
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams());
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults);
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT, crb.getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb,
+ int resultSize,
+ ShardFieldSortedHitQueue queue,
+ Map<String, List<ShardDoc>> shardDocMap,
+ SolrDocumentList responseDocs) {
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams());
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap);
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
+ shardDocMap.forEach(
+ (shardKey, shardDocs) ->
+ shardDocs.forEach(shardDoc -> shardDocIdMap.put(shardDoc.id.toString(), shardDoc)));
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
+ float maxScore = 0.0f;
+ for (int i = 0; i < resultSize; i++) {
+ ShardDoc shardDoc = combinedShardDocs.get(i);
+ shardDoc.positionInResponse = i;
+ maxScore = Math.max(maxScore, shardDoc.score);
Review Comment:
Normalisation is not applicable for RRF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231289023
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
Review Comment:
perhaps
```suggestion
this.k = args.getOrDefault("k", this.k);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2232849089
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
Review Comment:
- Shouldn't that be expectation (?) otherwise a different name can be used.
- Added an exception for this case, but user should make sure it's not duplicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2314372917
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ if (maxScore != null) responseDocs.setMaxScore(maxScore);
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ // save these results in a private area so we can access them
+ // again when retrieving stored fields.
+ // TODO: use ResponseBuilder (w/ comments) or the request context?
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs);
Review Comment:
If as per https://github.com/apache/solr/pull/3418/files#r2314366325 the `maxScore` setting were to be removed then I think here we could simplify like this ...
```suggestion
rb.resultIds = createShardResult(rb, shardDocMap);
for (int i = 0; i < rb.resultIds.size(); i++) responseDocs.add(null);
```
... and then somehow (still thinking about that) something`createShardResult-like` could be factored out in the `QueryComponent` base class and overridden here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2317761758
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,112 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for grouping and Cursors.
+====
+
+== Query DSL Structure
+The query structure is similar to JSON Query DSL except for how multiple queries are defined along with their parameters.
+
+* Multiple queries can be defined under the `queries` key by providing their name with the same syntax as a single query is defined with the key `query`.
+* In addition to the other supported parameters, there are several parameters which can be defined under `params` key as below:
+`combiner` | Default: `false`::
+ Enables the combined query mode when set to `true`.
+`combiner.query`::
+ The list of queries to be executed as defined in the `queries` key. Example: `["query1", "query2"]`
+`combiner.algorithm` | Default: `rrf`::
+ The algorithm to be used for combining the results. Reciprocal Rank Fusion (RRF) is the in-built fusion algorithm.
+ Any other algorithm can be configured using xref:json-combined-query-dsl.adoc#combiner-algorithm-plugin[plugin].
+`combiner.rrf.k` | Default: `60`::
+ The k parameter in the RRF algorithm.
+
+=== Example
+
+Below is a sample JSON query payload:
Review Comment:
Response is standard and same as any other query. Don't think we need to illustrate any response.
Is there any change in how `numFound` calculated earlier? If yes, it makes sense.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ if (maxScore != null) responseDocs.setMaxScore(maxScore);
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ // save these results in a private area so we can access them
+ // again when retrieving stored fields.
+ // TODO: use ResponseBuilder (w/ comments) or the request context?
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly)
+ && Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb, Map<String, List<ShardDoc>> shardDocMap, SolrDocumentList responseDocs) {
+ String algorithm =
+ rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap, rb.req.getParams());
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
+ shardDocMap.forEach(
+ (shardKey, shardDocs) ->
+ shardDocs.forEach(shardDoc -> shardDocIdMap.put(shardDoc.id.toString(), shardDoc)));
+ Map<Object, ShardDoc> resultIds = new HashMap<>();
+ float maxScore = 0.0f;
+ Sort sort = rb.getSortSpec().getSort();
+ SortField[] sortFields;
+ if (sort != null) {
+ sortFields = sort.getSort();
+ } else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+ final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(
+ sortFields,
+ rb.getSortSpec().getOffset() + rb.getSortSpec().getCount(),
+ rb.req.getSearcher());
+ combinedShardDocs.forEach(queue::insertWithOverflow);
+ int resultSize = queue.size() - rb.getSortSpec().getOffset();
+ resultSize = Math.max(0, resultSize);
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ maxScore = Math.max(maxScore, shardDoc.score);
+ if (Float.isNaN(shardDocIdMap.get(shardDoc.id.toString()).score)) {
+ shardDoc.score = Float.NaN;
+ }
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+ responseDocs.setMaxScore(maxScore);
Review Comment:
`maxScore` is the score after rrf compuation across queries which will be normalised score of all the docs across queries and shards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315946946
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ if (maxScore != null) responseDocs.setMaxScore(maxScore);
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ // save these results in a private area so we can access them
+ // again when retrieving stored fields.
+ // TODO: use ResponseBuilder (w/ comments) or the request context?
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs);
Review Comment:
No, it cannot be removed. maxScore is shown as the part of SolrDocument result and it has to be updated with latest `maxScore` after rrf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315936441
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
Review Comment:
observations: the `QueryComponent` equivalent to this block of code is https://github.com/apache/solr/blob/releases/solr/9.9.0/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L1122-L1156 but there are differences:
* in `QueryComponent` duplicates are unusual and will be omitted (and the `numFound` counter decremented)
* in `CombinedQueryComponent` duplicates are possible and must not be omitted (but the `numFound` counter will be decremented)
Added https://github.com/apache/solr/pull/3418/commits/4dcbb576f2ce25b5e07a86aa05ffc6ad06985c6f dev increment with this in mind i.e. supportive of both scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2315942209
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
+ if (prevShard != null) {
+ // duplicate detected
+ numFound--;
+ }
Review Comment:
Yes, it's not exactly duplicated all the code of the method, rather half of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2313937099
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
> #3565 explores factoring something out in QueryComponent.
So my current thinking here is that once #3565 looks reasonable then it would _not_ be merged to `origin/main` but I'd add it to this pull request's branch as a development increment.
And then next I'm hoping to explore factoring something else out in QueryComponent w.r.t. the `resultIds` computation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2316063996
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
> ... And then next I'm hoping to explore factoring something else out in QueryComponent w.r.t. the `resultIds` computation.
https://github.com/apache/solr/pull/3418/commits/006b8c231fd17655f1d6e7e4d02ba607e66af702 and https://github.com/apache/solr/pull/3418/commits/771089ba7625d7a1d48661638eeabdc7d04786ea and https://github.com/apache/solr/pull/3418/commits/460e8cd6fce2aa439417f8dc960a3c753a7604d0 sequence is for that -- @dsmiley if you get a chance I'd appreciate a direction check on that.
My next intended step is to look further into the `resultIds` and `responseDocs` interaction and to better understand the `CombinedQueryComponent.createShardResult` implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2317770682
##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -905,7 +906,84 @@ protected boolean addFL(StringBuilder fl, String field, boolean additionalAdded)
return true;
}
+ protected abstract static class ShardDocQueue {
+ public abstract boolean push(ShardDoc shardDoc);
+
+ public abstract Map<Object, ShardDoc> resultIds(int offset);
+ }
+ ;
+
+ protected static class ShardDocQueueFactory
+ implements BiFunction<SortField[], Integer, ShardDocQueue> {
+
+ private final SolrIndexSearcher searcher;
+
+ public ShardDocQueueFactory(SolrIndexSearcher searcher) {
+ this.searcher = searcher;
+ }
+
+ @Override
+ public ShardDocQueue apply(SortField[] sortFields, Integer size) {
+ return new ShardDocQueue() {
+
+ // id to shard mapping, to eliminate any accidental dups
+ private final HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ private final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(sortFields, size, searcher);
+
+ @Override
+ public boolean push(ShardDoc shardDoc) {
+ final String prevShard = uniqueDoc.put(shardDoc.id, shardDoc.shard);
+ if (prevShard != null) {
+ // duplicate detected
+
+ // For now, just always use the first encountered since we can't currently
+ // remove the previous one added to the priority queue. If we switched
+ // to the Java5 PriorityQueue, this would be easier.
+ return false;
+ // make which duplicate is used deterministic based on shard
+ // if (prevShard.compareTo(shardDoc.shard) >= 0) {
+ // TODO: remove previous from priority queue
+ // return false;
+ // }
+ }
+
+ queue.insertWithOverflow(shardDoc);
+ return true;
+ }
+
+ @Override
+ public Map<Object, ShardDoc> resultIds(int offset) {
+ final Map<Object, ShardDoc> resultIds = new HashMap<>();
+
+ // The queue now has 0 -> queuesize docs, where queuesize <= start + rows
+ // So we want to pop the last documents off the queue to get
+ // the docs offset -> queuesize
+ int resultSize = queue.size() - offset;
+ resultSize = Math.max(0, resultSize); // there may not be any docs in range
+
+ for (int i = resultSize - 1; i >= 0; i--) {
+ ShardDoc shardDoc = queue.pop();
+ shardDoc.positionInResponse = i;
+ // Need the toString() for correlation with other lists that must
+ // be strings (like keys in highlighting, explain, etc)
+ resultIds.put(shardDoc.id.toString(), shardDoc);
+ }
+
+ return resultIds;
+ }
+ };
+ }
+ }
+ ;
+
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ implementMergeIds(rb, sreq, new ShardDocQueueFactory(rb.req.getSearcher()));
+ }
+
+ private void implementMergeIds(
Review Comment:
> though in that case no need for a factory perhaps:
Simple!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2321569807
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+
+/**
+ * This class implements a query and response combiner that uses the Reciprocal Rank Fusion (RRF)
+ * algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.DEFAULT_COMBINER_RRF_K;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ public int getK() {
+ return k;
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocListAndSet> docListAndSet = getDocListsAndSetFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docListAndSet, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocListAndSet> getDocListsAndSetFromQueryResults(
+ List<QueryResult> rankedLists) {
+ List<DocListAndSet> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocListAndSet());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
Review Comment:
added java-docs. Yes, it would be interleaved in this case as the each shard has unique docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2318914312
##########
solr/core/src/test/org/apache/solr/search/combine/ReciprocalRankFusionTest.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import static org.apache.solr.common.params.CombinerParams.COMBINER_RRF_K;
+import static org.apache.solr.common.params.CombinerParams.RECIPROCAL_RANK_FUSION;
+
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * The ReciprocalRankFusionTest class is a unit test suite for the {@link ReciprocalRankFusion}
+ * class. It verifies the correctness of the fusion algorithm and its supporting methods.
+ */
+@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
+public class ReciprocalRankFusionTest extends SolrTestCaseJ4 {
Review Comment:
I'm looking at that and I only see the DocList being examined.
ReciprocalRankFusionTest, being a unit test, ought to also test what it did with DocSets since it combines the QueryResult, which is the combo of both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2434418768
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
Review Comment:
It appears this would activate for SolrCloud... yet we wouldn't want that.
As I suggest below, by moving this logic to the one spot where we need it, then this whole portion of forceDistributed is removed.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
Review Comment:
As this component forces distributed mode, why would process() be called?
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ Boolean setMaxHitsTerminatedEarly = null;
+ List<QueryResult> queryResults = new ArrayList<>();
+ int rbIndex = 0;
+ boolean shouldReturn = false;
+ // TODO: to be parallelized
+ for (ResponseBuilder thisRb : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ thisRb.setCursorMark(crb.getCursorMark());
+ super.process(thisRb);
+ int purpose =
+ thisRb
+ .req
+ .getParams()
+ .getInt(ShardParams.SHARDS_PURPOSE, ShardRequest.PURPOSE_GET_TOP_IDS);
+ if ((purpose & ShardRequest.PURPOSE_GET_TERM_STATS) != 0) {
+ shouldReturn = true;
+ continue;
+ }
+ DocListAndSet docListAndSet = thisRb.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ if (setMaxHitsTerminatedEarly == null) {
+ setMaxHitsTerminatedEarly = queryResult.getMaxHitsTerminatedEarly();
+ }
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ doFieldSortValues(thisRb, crb.req.getSearcher());
+ NamedList<Object[]> sortValues =
+ (NamedList<Object[]>) thisRb.rsp.getValues().get("sort_values");
+ crb.rsp.add(String.format(Locale.ROOT, "sort_values_%s", rbIndex), sortValues);
+ ResultContext ctx = new BasicResultContext(thisRb);
+ if (crb.rsp.getValues().get(RESPONSE_PER_QUERY_KEY) == null) {
+ crb.rsp.add(RESPONSE_PER_QUERY_KEY, new ArrayList<>(List.of(ctx)));
+ } else {
+ ((List<ResultContext>) crb.rsp.getValues().get(RESPONSE_PER_QUERY_KEY)).add(ctx);
+ }
+ rbIndex++;
+ }
+ if (shouldReturn) {
+ return;
+ }
+ prepareCombinedResponseBuilder(
+ crb, queryResults, partialResults, segmentTerminatedEarly, setMaxHitsTerminatedEarly);
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ private void prepareCombinedResponseBuilder(
+ CombinedQueryResponseBuilder crb,
+ List<QueryResult> queryResults,
+ boolean partialResults,
+ boolean segmentTerminatedEarly,
+ Boolean setMaxHitsTerminatedEarly) {
+ QueryResult combinedQueryResult = QueryAndResponseCombiner.simpleCombine(queryResults);
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ SortSpec ss = rb.getSortSpec();
+ Sort sort = ss.getSort();
+
+ SortField[] sortFields;
+ if (sort != null) sortFields = sort.getSort();
+ else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ String[] queriesToCombineKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ // TODO: to be parallelized outer loop
+ for (int queryIndex = 0; queryIndex < queriesToCombineKeys.length; queryIndex++) {
+ int failedShardCount = 0;
+ long queryNumFound = 0;
+ long queryApproximateTotalHits = 0;
+ final ShardDocQueue queuePerQuery =
+ newShardDocQueue(rb.req.getSearcher(), sortFields, ss.getOffset() + ss.getCount());
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader;
+
+ if (SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, RESPONSE_PER_QUERY_KEY, false)
+ instanceof List<?> docList
+ && docList.get(queryIndex) instanceof SolrDocumentList solrDocumentList) {
+ docs = Objects.requireNonNull(solrDocumentList);
+ queryNumFound += docs.getNumFound();
+ hitCountIsExact = hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact());
+ }
+ failedShardCount +=
+ addShardInfo(
+ shardInfo, failedShardCount, srsp, rb, queriesToCombineKeys[queryIndex], docs);
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(
+ SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (ste instanceof Boolean steFlag) {
+ segmentTerminatedEarly = steFlag;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly
+ && Boolean.TRUE.equals(
+ responseHeader.get(
+ SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+
+ Object ath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ queryApproximateTotalHits += queryNumFound;
+ } else {
+ queryApproximateTotalHits += ((Number) ath).longValue();
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, String.format(Locale.ROOT, "sort_values_%s", queryIndex), true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling
+ ? unmarshalSortValues(ss, sortFieldValues, schema)
+ : new NamedList<>();
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the uniqueDoc to dedup
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.orderInShard = i;
+ shardDoc.shard = srsp.getShard();
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String scoreStr) {
+ shardDoc.score = Float.parseFloat(scoreStr);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ if (!queuePerQuery.push(shardDoc)) {
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+ List<ShardDoc> shardDocsPerQuery = new ArrayList<>(queuePerQuery.resultIds(0).values());
+ shardDocsPerQuery.sort(Comparator.comparingInt(a -> a.positionInResponse));
+ shardDocMap.put(queriesToCombineKeys[queryIndex], shardDocsPerQuery);
+ numFound = max(numFound, queryNumFound);
+ approximateTotalHits = max(approximateTotalHits, queryApproximateTotalHits);
+ }
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs, sortFields);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ postMergeIds(
+ rb,
+ thereArePartialResults,
+ segmentTerminatedEarly,
+ maxHitsTerminatedEarly,
+ approximateTotalHits);
+ }
+
+ private static void postMergeIds(
+ ResponseBuilder rb,
+ boolean thereArePartialResults,
+ Boolean segmentTerminatedEarly,
+ boolean maxHitsTerminatedEarly,
+ long approximateTotalHits) {
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly) && segmentTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, true);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ private int addShardInfo(
+ NamedList<Object> shardInfo,
+ int failedShardCount,
+ ShardResponse srsp,
+ ResponseBuilder rb,
+ String queryKey,
+ SolrDocumentList docs) {
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+ NamedList<?> responseHeader;
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ return failedShardCount;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ if (docs == null) {
+ return failedShardCount;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard() + "_" + queryKey;
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + queryKey + "_" + failedShardCount;
+ }
+ nl.add("queryKey", queryKey);
+ shardInfo.add(shard, nl);
+ }
+ return failedShardCount;
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @param sortFields An array of field for sorting to be applied.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb,
+ Map<String, List<ShardDoc>> shardDocMap,
+ SolrDocumentList responseDocs,
+ SortField[] sortFields) {
+ String algorithm =
+ rb.req.getParams().get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ List<ShardDoc> combinedShardDocs = combinerStrategy.combine(shardDocMap, rb.req.getParams());
+ if (rb.isDebugResults()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys, shardDocMap, combinedShardDocs, rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ Map<String, ShardDoc> shardDocIdMap = new HashMap<>();
Review Comment:
the method here is big; I think some occassional blank lines maybe with some comments to separate phases is helpful. About here is a juncture.
##########
solr/core/src/java/org/apache/solr/handler/component/SearchComponent.java:
##########
@@ -93,6 +93,16 @@ public String getName() {
@Override
public abstract String getDescription();
+ /**
+ * A component can force solr to run in distributed mode to prevent extra development cost of an
Review Comment:
s/solr/SearchHandler/
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ Boolean setMaxHitsTerminatedEarly = null;
+ List<QueryResult> queryResults = new ArrayList<>();
+ int rbIndex = 0;
+ boolean shouldReturn = false;
+ // TODO: to be parallelized
+ for (ResponseBuilder thisRb : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ thisRb.setCursorMark(crb.getCursorMark());
+ super.process(thisRb);
+ int purpose =
+ thisRb
+ .req
+ .getParams()
+ .getInt(ShardParams.SHARDS_PURPOSE, ShardRequest.PURPOSE_GET_TOP_IDS);
+ if ((purpose & ShardRequest.PURPOSE_GET_TERM_STATS) != 0) {
+ shouldReturn = true;
+ continue;
+ }
+ DocListAndSet docListAndSet = thisRb.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ if (setMaxHitsTerminatedEarly == null) {
+ setMaxHitsTerminatedEarly = queryResult.getMaxHitsTerminatedEarly();
+ }
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ doFieldSortValues(thisRb, crb.req.getSearcher());
+ NamedList<Object[]> sortValues =
+ (NamedList<Object[]>) thisRb.rsp.getValues().get("sort_values");
+ crb.rsp.add(String.format(Locale.ROOT, "sort_values_%s", rbIndex), sortValues);
+ ResultContext ctx = new BasicResultContext(thisRb);
+ if (crb.rsp.getValues().get(RESPONSE_PER_QUERY_KEY) == null) {
+ crb.rsp.add(RESPONSE_PER_QUERY_KEY, new ArrayList<>(List.of(ctx)));
+ } else {
+ ((List<ResultContext>) crb.rsp.getValues().get(RESPONSE_PER_QUERY_KEY)).add(ctx);
+ }
+ rbIndex++;
+ }
+ if (shouldReturn) {
+ return;
+ }
+ prepareCombinedResponseBuilder(
+ crb, queryResults, partialResults, segmentTerminatedEarly, setMaxHitsTerminatedEarly);
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ private void prepareCombinedResponseBuilder(
+ CombinedQueryResponseBuilder crb,
+ List<QueryResult> queryResults,
+ boolean partialResults,
+ boolean segmentTerminatedEarly,
+ Boolean setMaxHitsTerminatedEarly) {
+ QueryResult combinedQueryResult = QueryAndResponseCombiner.simpleCombine(queryResults);
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ SortSpec ss = rb.getSortSpec();
+ Sort sort = ss.getSort();
+
+ SortField[] sortFields;
+ if (sort != null) sortFields = sort.getSort();
+ else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ String[] queriesToCombineKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ // TODO: to be parallelized outer loop
+ for (int queryIndex = 0; queryIndex < queriesToCombineKeys.length; queryIndex++) {
+ int failedShardCount = 0;
+ long queryNumFound = 0;
+ long queryApproximateTotalHits = 0;
+ final ShardDocQueue queuePerQuery =
+ newShardDocQueue(rb.req.getSearcher(), sortFields, ss.getOffset() + ss.getCount());
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader;
+
+ if (SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, RESPONSE_PER_QUERY_KEY, false)
+ instanceof List<?> docList
+ && docList.get(queryIndex) instanceof SolrDocumentList solrDocumentList) {
+ docs = Objects.requireNonNull(solrDocumentList);
+ queryNumFound += docs.getNumFound();
+ hitCountIsExact = hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact());
+ }
+ failedShardCount +=
+ addShardInfo(
+ shardInfo, failedShardCount, srsp, rb, queriesToCombineKeys[queryIndex], docs);
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(
+ SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (ste instanceof Boolean steFlag) {
+ segmentTerminatedEarly = steFlag;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly
+ && Boolean.TRUE.equals(
+ responseHeader.get(
+ SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+
+ Object ath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ queryApproximateTotalHits += queryNumFound;
+ } else {
+ queryApproximateTotalHits += ((Number) ath).longValue();
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, String.format(Locale.ROOT, "sort_values_%s", queryIndex), true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling
+ ? unmarshalSortValues(ss, sortFieldValues, schema)
+ : new NamedList<>();
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the uniqueDoc to dedup
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.orderInShard = i;
+ shardDoc.shard = srsp.getShard();
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String scoreStr) {
+ shardDoc.score = Float.parseFloat(scoreStr);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ if (!queuePerQuery.push(shardDoc)) {
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+ List<ShardDoc> shardDocsPerQuery = new ArrayList<>(queuePerQuery.resultIds(0).values());
+ shardDocsPerQuery.sort(Comparator.comparingInt(a -> a.positionInResponse));
+ shardDocMap.put(queriesToCombineKeys[queryIndex], shardDocsPerQuery);
+ numFound = max(numFound, queryNumFound);
+ approximateTotalHits = max(approximateTotalHits, queryApproximateTotalHits);
+ }
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs, sortFields);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ postMergeIds(
+ rb,
+ thereArePartialResults,
+ segmentTerminatedEarly,
+ maxHitsTerminatedEarly,
+ approximateTotalHits);
+ }
+
+ private static void postMergeIds(
+ ResponseBuilder rb,
+ boolean thereArePartialResults,
+ Boolean segmentTerminatedEarly,
+ boolean maxHitsTerminatedEarly,
+ long approximateTotalHits) {
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly) && segmentTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, true);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ private int addShardInfo(
+ NamedList<Object> shardInfo,
+ int failedShardCount,
+ ShardResponse srsp,
+ ResponseBuilder rb,
+ String queryKey,
+ SolrDocumentList docs) {
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+ NamedList<?> responseHeader;
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ return failedShardCount;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ if (docs == null) {
+ return failedShardCount;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard() + "_" + queryKey;
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + queryKey + "_" + failedShardCount;
+ }
+ nl.add("queryKey", queryKey);
+ shardInfo.add(shard, nl);
+ }
+ return failedShardCount;
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @param sortFields An array of field for sorting to be applied.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
+ ResponseBuilder rb,
+ Map<String, List<ShardDoc>> shardDocMap,
+ SolrDocumentList responseDocs,
Review Comment:
it's unfortunate to have a method that both returns something and also manipulates an argument. At least you documented it.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import static java.lang.Math.max;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private final Map<String, QueryAndResponseCombiner> combiners = new HashMap<>();
+ private int maxCombinerQueries;
+ private static final String RESPONSE_PER_QUERY_KEY = "response_per_query";
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ for (Map.Entry<String, ?> initEntry : initParams) {
+ if ("combiners".equals(initEntry.getKey())
+ && initEntry.getValue() instanceof NamedList<?> all) {
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ }
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ @Override
+ protected boolean isForceDistributed() {
+ return true;
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.setFilters(rbNew.getFilters());
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple queries, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ @SuppressWarnings("unchecked")
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ Boolean setMaxHitsTerminatedEarly = null;
+ List<QueryResult> queryResults = new ArrayList<>();
+ int rbIndex = 0;
+ boolean shouldReturn = false;
+ // TODO: to be parallelized
+ for (ResponseBuilder thisRb : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ thisRb.setCursorMark(crb.getCursorMark());
+ super.process(thisRb);
+ int purpose =
+ thisRb
+ .req
+ .getParams()
+ .getInt(ShardParams.SHARDS_PURPOSE, ShardRequest.PURPOSE_GET_TOP_IDS);
+ if ((purpose & ShardRequest.PURPOSE_GET_TERM_STATS) != 0) {
+ shouldReturn = true;
+ continue;
+ }
+ DocListAndSet docListAndSet = thisRb.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ if (setMaxHitsTerminatedEarly == null) {
+ setMaxHitsTerminatedEarly = queryResult.getMaxHitsTerminatedEarly();
+ }
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ doFieldSortValues(thisRb, crb.req.getSearcher());
+ NamedList<Object[]> sortValues =
+ (NamedList<Object[]>) thisRb.rsp.getValues().get("sort_values");
+ crb.rsp.add(String.format(Locale.ROOT, "sort_values_%s", rbIndex), sortValues);
+ ResultContext ctx = new BasicResultContext(thisRb);
+ if (crb.rsp.getValues().get(RESPONSE_PER_QUERY_KEY) == null) {
+ crb.rsp.add(RESPONSE_PER_QUERY_KEY, new ArrayList<>(List.of(ctx)));
+ } else {
+ ((List<ResultContext>) crb.rsp.getValues().get(RESPONSE_PER_QUERY_KEY)).add(ctx);
+ }
+ rbIndex++;
+ }
+ if (shouldReturn) {
+ return;
+ }
+ prepareCombinedResponseBuilder(
+ crb, queryResults, partialResults, segmentTerminatedEarly, setMaxHitsTerminatedEarly);
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ private void prepareCombinedResponseBuilder(
+ CombinedQueryResponseBuilder crb,
+ List<QueryResult> queryResults,
+ boolean partialResults,
+ boolean segmentTerminatedEarly,
+ Boolean setMaxHitsTerminatedEarly) {
+ QueryResult combinedQueryResult = QueryAndResponseCombiner.simpleCombine(queryResults);
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp.addToLog(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request and future cursor improvement
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ SortSpec ss = rb.getSortSpec();
+ Sort sort = ss.getSort();
+
+ SortField[] sortFields;
+ if (sort != null) sortFields = sort.getSort();
+ else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ String[] queriesToCombineKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ // TODO: to be parallelized outer loop
+ for (int queryIndex = 0; queryIndex < queriesToCombineKeys.length; queryIndex++) {
+ int failedShardCount = 0;
+ long queryNumFound = 0;
+ long queryApproximateTotalHits = 0;
+ final ShardDocQueue queuePerQuery =
+ newShardDocQueue(rb.req.getSearcher(), sortFields, ss.getOffset() + ss.getCount());
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader;
+
+ if (SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, RESPONSE_PER_QUERY_KEY, false)
+ instanceof List<?> docList
+ && docList.get(queryIndex) instanceof SolrDocumentList solrDocumentList) {
+ docs = Objects.requireNonNull(solrDocumentList);
+ queryNumFound += docs.getNumFound();
+ hitCountIsExact = hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact());
+ }
+ failedShardCount +=
+ addShardInfo(
+ shardInfo, failedShardCount, srsp, rb, queriesToCombineKeys[queryIndex], docs);
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(
+ SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (ste instanceof Boolean steFlag) {
+ segmentTerminatedEarly = steFlag;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly
+ && Boolean.TRUE.equals(
+ responseHeader.get(
+ SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+
+ Object ath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ queryApproximateTotalHits += queryNumFound;
+ } else {
+ queryApproximateTotalHits += ((Number) ath).longValue();
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, String.format(Locale.ROOT, "sort_values_%s", queryIndex), true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling
+ ? unmarshalSortValues(ss, sortFieldValues, schema)
+ : new NamedList<>();
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the uniqueDoc to dedup
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.orderInShard = i;
+ shardDoc.shard = srsp.getShard();
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String scoreStr) {
+ shardDoc.score = Float.parseFloat(scoreStr);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ if (!queuePerQuery.push(shardDoc)) {
+ numFound--;
+ }
+ } // end for-each-doc-in-response
+ } // end for-each-response
+ List<ShardDoc> shardDocsPerQuery = new ArrayList<>(queuePerQuery.resultIds(0).values());
+ shardDocsPerQuery.sort(Comparator.comparingInt(a -> a.positionInResponse));
+ shardDocMap.put(queriesToCombineKeys[queryIndex], shardDocsPerQuery);
+ numFound = max(numFound, queryNumFound);
+ approximateTotalHits = max(approximateTotalHits, queryApproximateTotalHits);
+ }
+
+ SolrDocumentList responseDocs = new SolrDocumentList();
+ rb.rsp.addToLog("hits", numFound);
+
+ responseDocs.setNumFound(numFound);
+ responseDocs.setNumFoundExact(hitCountIsExact);
+ responseDocs.setStart(ss.getOffset());
+
+ rb.resultIds = createShardResult(rb, shardDocMap, responseDocs, sortFields);
+ rb.setResponseDocs(responseDocs);
+
+ populateNextCursorMarkFromMergedShards(rb);
+
+ postMergeIds(
+ rb,
+ thereArePartialResults,
+ segmentTerminatedEarly,
+ maxHitsTerminatedEarly,
+ approximateTotalHits);
+ }
+
+ private static void postMergeIds(
+ ResponseBuilder rb,
+ boolean thereArePartialResults,
+ Boolean segmentTerminatedEarly,
+ boolean maxHitsTerminatedEarly,
+ long approximateTotalHits) {
+ if (thereArePartialResults) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ }
+ if (segmentTerminatedEarly != null) {
+ final Object existingSegmentTerminatedEarly =
+ rb.rsp
+ .getResponseHeader()
+ .get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (existingSegmentTerminatedEarly == null) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY,
+ segmentTerminatedEarly);
+ } else if (!Boolean.TRUE.equals(existingSegmentTerminatedEarly) && segmentTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .remove(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, true);
+ }
+ }
+ if (maxHitsTerminatedEarly) {
+ rb.rsp
+ .getResponseHeader()
+ .add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, Boolean.TRUE);
+ if (approximateTotalHits > 0) {
+ rb.rsp
+ .getResponseHeader()
+ .add(
+ SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, approximateTotalHits);
+ }
+ }
+ }
+
+ private int addShardInfo(
+ NamedList<Object> shardInfo,
+ int failedShardCount,
+ ShardResponse srsp,
+ ResponseBuilder rb,
+ String queryKey,
+ SolrDocumentList docs) {
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+ NamedList<?> responseHeader;
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ return failedShardCount;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ if (docs == null) {
+ return failedShardCount;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard() + "_" + queryKey;
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + queryKey + "_" + failedShardCount;
+ }
+ nl.add("queryKey", queryKey);
+ shardInfo.add(shard, nl);
+ }
+ return failedShardCount;
+ }
+
+ /**
+ * Combines and sorts documents from multiple shards to create the final result set. This method
+ * uses a combiner strategy to merge shard responses, then sorts the resulting documents using a
+ * priority queue based on the request's sort specification. It handles pagination (offset and
+ * count) and calculates the maximum score for the response.
+ *
+ * @param rb The ResponseBuilder containing the request and context, such as sort specifications.
+ * @param shardDocMap A map from shard addresses to the list of documents returned by each shard.
+ * @param responseDocs The final response document list, which will be populated with null
+ * placeholders and have its max score set.
+ * @param sortFields An array of field for sorting to be applied.
+ * @return A map from document IDs to the corresponding ShardDoc objects for the documents in the
+ * final sorted page of results.
+ */
+ protected Map<Object, ShardDoc> createShardResult(
Review Comment:
the name of this isn't helpful IMO. Maybe computeResultIdsWithCombiner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2306958149
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
Review Comment:
I have no exact answer for this, but it seems to me that the majority of the components do not have the `_` in the name. Also, those in facet are like "FACET_KEY", not the component name..
The others I could see do not have it like: `responseLog`, `querycancellation`, `activetaskslist` ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3235767193
Thanks @aruggero for your time and feedback and glad to hear the current state looks good!
> Is it worth adding some additional tests to check the behaviour of this new query component with highlighting/faceting?
Sure, Have added it in recent commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3666228103
Sounds good, Thank you @dsmiley - done with update from main. I would appreciate it if you could trigger the workflows in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2232845710
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
Review Comment:
- It's the other way round. The inbuilt combiner with same reserved name `rrf` will take precedence.
- any custom combiner shouldn't have same name as one with reserved name `rrf`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2231466261
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,572 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.computeIfAbsent(name, combinerName -> combiner);
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // propagating from global ResponseBuilder, so that if in case cursor is needed for
+ // retrieving the next batch of documents
+ // which might have duplicate results from the previous cursorMark as we are dealing with
+ // multiple queries
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(rb.req.getParams(), combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
+ .add(
+ "hits",
+ crb.getResults() == null || crb.getResults().docList == null
+ ? 0
+ : crb.getResults().docList.matches());
+ if (!crb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
+ // for non-distributed request
+ if (null != crb.getNextCursorMark()) {
+ crb.rsp.add(
+ CursorMarkParams.CURSOR_MARK_NEXT,
+ crb.responseBuilders.getFirst().getNextCursorMark().getSerializedTotem());
+ }
+ }
+
+ if (crb.mergeFieldHandler != null) {
+ crb.mergeFieldHandler.handleMergeFields(crb, crb.req.getSearcher());
+ } else {
+ doFieldSortValues(rb, crb.req.getSearcher());
+ }
+ doPrefetch(crb);
+ } else {
+ super.process(rb);
+ }
+ }
+
+ @Override
+ protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
+ }
+ }
+
+ if (idsMerged) {
+ return; // ids were merged above so return.
+ }
+ }
+
+ SortSpec ss = rb.getSortSpec();
+
+ // If the shard request was also used to get fields (along with the scores), there is no reason
+ // to copy over the score dependent fields, since those will already exist in the document with
+ // the return fields
+ Set<String> scoreDependentFields;
+ if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) == 0) {
+ scoreDependentFields =
+ rb.rsp.getReturnFields().getScoreDependentReturnFields().keySet().stream()
+ .filter(field -> !field.equals(SolrReturnFields.SCORE))
+ .collect(Collectors.toSet());
+ } else {
+ scoreDependentFields = Collections.emptySet();
+ }
+
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
+
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
+
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ boolean maxHitsTerminatedEarly = false;
+ long approximateTotalHits = 0;
+ int failedShardCount = 0;
+ Map<String, List<ShardDoc>> shardDocMap = new HashMap<>();
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException && t.getCause() != null) {
+ t = t.getCause();
+ }
+ nl.add("error", t.toString());
+ if (!rb.req.getCore().getCoreContainer().hideStackTrace()) {
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ }
+ if (!StrUtils.isNullOrEmpty(srsp.getShardAddress())) {
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ } else {
+ responseHeader =
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false);
+ if (responseHeader == null) {
+ continue;
+ }
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
+ }
+ final Object rhmhte =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY);
+ if (rhmhte != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY, rhmhte);
+ }
+ final Object rhath =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (rhath != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY, rhath);
+ }
+ docs =
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false);
+ if (docs == null) {
+ continue;
+ }
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
+ }
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ }
+ // This ought to be better, but at least this ensures no duplicate keys in JSON result
+ String shard = srsp.getShard();
+ if (StrUtils.isNullOrEmpty(shard)) {
+ failedShardCount += 1;
+ shard = "unknown_shard_" + failedShardCount;
+ }
+ shardInfo.add(shard, nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
+
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs =
+ Objects.requireNonNull(
+ (SolrDocumentList)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "response", false));
+ }
+
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader =
+ Objects.requireNonNull(
+ (NamedList<?>)
+ SolrResponseUtil.getSubsectionFromShardResponse(
+ rb, srsp, "responseHeader", false));
+ }
+
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
+ }
+ }
+
+ if (!maxHitsTerminatedEarly) {
+ if (Boolean.TRUE.equals(
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_MAX_HITS_TERMINATED_EARLY_KEY))) {
+ maxHitsTerminatedEarly = true;
+ }
+ }
+ Object ath = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_APPROXIMATE_TOTAL_HITS_KEY);
+ if (ath == null) {
+ approximateTotalHits += numFound;
+ } else {
+ approximateTotalHits += ((Number) ath).longValue();
+ }
+
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
+
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
+
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>)
+ SolrResponseUtil.getSubsectionFromShardResponse(rb, srsp, "sort_values", true);
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
+ }
+
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are being returned from the shard, then skip merging the results for the
+ // shard. This avoids an exception below. if the shard returned partial results but we don't
+ // need to unmarshal (a normal scoring query), then merge what we got.
+ if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
+ continue;
+ }
+
+ // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
+ NamedList<List<Object>> unmarshalledSortFieldValues =
+ needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
+
+ // go through every doc in this response, construct a ShardDoc, and
+ // put it in the priority queue so it can be ordered.
+ for (int i = 0; i < docs.size(); i++) {
+ SolrDocument doc = docs.get(i);
+ Object id = doc.getFieldValue(uniqueKeyField.getName());
+ ShardDoc shardDoc = new ShardDoc();
+ shardDoc.id = id;
+ shardDoc.shard = srsp.getShard();
+ shardDoc.orderInShard = i;
+ Object scoreObj = doc.getFieldValue(SolrReturnFields.SCORE);
+ if (scoreObj != null) {
+ if (scoreObj instanceof String) {
+ shardDoc.score = Float.parseFloat((String) scoreObj);
+ } else {
+ shardDoc.score = ((Number) scoreObj).floatValue();
+ }
+ }
+ if (!scoreDependentFields.isEmpty()) {
+ shardDoc.scoreDependentFields = doc.getSubsetOfFields(scoreDependentFields);
+ }
+
+ shardDoc.sortFieldValues = unmarshalledSortFieldValues;
+ shardDocMap.computeIfAbsent(srsp.getShard(), list -> new ArrayList<>()).add(shardDoc);
+ String prevShard = uniqueDoc.put(id, srsp.getShard());
Review Comment:
```suggestion
shardDocMap.computeIfAbsent(shardDoc.shard, list -> new ArrayList<>()).add(shardDoc);
String prevShard = uniqueDoc.put(id, shardDoc.shard);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on PR #3418:
URL: https://github.com/apache/solr/pull/3418#issuecomment-3157437020
> Hi @ercsonusharma , thanks for resurrecting this, didn't have time to dedicate to the feature in the last few months, good to see some movement!
In the next couple of weeks, I should be able to give it a go and review it!
@alessandrobenedetti I have addressed the review comments here. Wanted to check if you’ve had a chance to review it. Looking forward to your feedback. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286779679
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
Review Comment:
please avoid indexed access to a NamedList unless you really need it. Use forEach.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
Review Comment:
why concurrent?
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
I'm surprised here; perhaps I'm missing the bigger context. Solr DocSet is used to hold *all* doc IDs that match a query, ignoring rows or start params. It's only for a non-distrib request (distrib=false), as the coordinator isn't going to get *all* doc IDs. My limited understanding of CombinedQueryComponent is that it runs at the coordinator. In such a place, it's not reasonable for it to create a DocSet.
##########
solr/solr-ref-guide/modules/query-guide/pages/json-combined-query-dsl.adoc:
##########
@@ -0,0 +1,113 @@
+= JSON Combined Query DSL
+:tabs-sync-option:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Combined Query feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion).
+It is extending JSON Query DSL ultimately enabling Hybrid Search.
+
+[NOTE]
+====
+This feature is currently unsupported for non-distributed query, grouping and Cursors.
Review Comment:
Doesn't support non-distributed?! (single-core)? Really?
##########
solr/solrj/src/java/org/apache/solr/common/params/CombinerParams.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.params;
+
+/**
+ * The CombinerParams class provides constants for configuration parameters related to the combiner.
+ * It defines keys for various properties used in the combiner configuration.
+ */
+public class CombinerParams {
+
+ private CombinerParams() {}
+
+ public static final String COMBINER = "combiner";
+ public static final String COMBINER_ALGORITHM = COMBINER + ".algorithm";
+ public static final String COMBINER_QUERY = COMBINER + ".query";
+ public static final String RECIPROCAL_RANK_FUSION = "rrf";
+ public static final String COMBINER_RRF_K = COMBINER + "." + RECIPROCAL_RANK_FUSION + ".k";
+ public static final String DEFAULT_COMBINER = RECIPROCAL_RANK_FUSION;
+ public static final int COMBINER_RRF_K_DEFAULT = 60;
Review Comment:
This one ends with DEFAULT but two others start with DEFAULT. Let's be consistent.
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
Review Comment:
nit: javadoc style guide prescribes you skip these initial words here as it's boilerplate noise. Remove "The ReciprocalRankFusion class implements "
##########
solr/solrj/src/java/org/apache/solr/common/params/CombinerParams.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.params;
+
+/**
+ * The CombinerParams class provides constants for configuration parameters related to the combiner.
Review Comment:
nit: again, skip leading boilerplate words (repetition of class name) and get straight to the point
##########
solr/core/src/test-files/solr/collection1/conf/solrconfig-combined-query.xml:
##########
@@ -0,0 +1,555 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- This is a "kitchen sink" config file that tests can use.
+ When writting a new test, feel free to add *new* items (plugins,
+ config options, etc...) as long as they don't break any existing
+ tests. if you need to test something esoteric please add a new
+ "solrconfig-your-esoteric-purpose.xml" config file.
+
+ Note in particular that this test is used by MinimalSchemaTest so
+ Anything added to this file needs to work correctly even if there
+ is now uniqueKey or defaultSearch Field.
+ -->
+
+<config>
+
+ <!-- Used to specify an alternate directory to hold all index data.
+ It defaults to "index" if not present, and should probably
+ not be changed if replication is in use. -->
+ <dataDir>${solr.data.dir:}</dataDir>
+
+ <!-- The DirectoryFactory to use for indexes.
+ solr.StandardDirectoryFactory, the default, is filesystem based.
+ solr.RAMDirectoryFactory is memory based and not persistent. -->
+ <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.MockDirectoryFactory}">
+ <double name="maxWriteMBPerSecDefault">1000000</double>
+ <double name="maxWriteMBPerSecFlush">2000000</double>
+ <double name="maxWriteMBPerSecMerge">3000000</double>
+ <double name="maxWriteMBPerSecRead">4000000</double>
+ </directoryFactory>
+
+ <schemaFactory class="ClassicIndexSchemaFactory"/>
+
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+
+ <statsCache class="${solr.statsCache:}"/>
+
+ <xi:include href="solrconfig.snippet.randomindexconfig.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
+
+ <updateHandler class="${solr.updateHandler:solr.DirectUpdateHandler2}">
+
+ <autoCommit>
+ <maxTime>${solr.autoCommit.maxTime:-1}</maxTime>
+ </autoCommit>
+
+ <!-- autocommit pending docs if certain criteria are met
+ <autoCommit>
+ <maxDocs>10000</maxDocs>
+ <maxTime>3600000</maxTime>
+ </autoCommit>
+ -->
+
+ <updateLog enable="${enable.update.log:true}">
+ <str name="dir">${solr.ulog.dir:}</str>
+ </updateLog>
+
+ <commitWithin>
+ <softCommit>${solr.commitwithin.softcommit:true}</softCommit>
+ </commitWithin>
+
+ </updateHandler>
+
+ <query>
+ <!-- Maximum number of clauses in a boolean query... can affect
+ range or wildcard queries that expand to big boolean
+ queries. An exception is thrown if exceeded.
+ -->
+ <maxBooleanClauses>${solr.max.booleanClauses:1024}</maxBooleanClauses>
+
+ <!-- Minimum acceptable prefix-size for prefix-based queries.
+
+ Prefix-based queries consume memory in proportion to the number of terms in the index
+ that start with that prefix. Short prefixes tend to match many many more indexed-terms
+ and consume more memory as a result, sometimes causing stability issues on the node.
+
+ This setting allows administrators to require that prefixes meet or exceed a specified
+ minimum length requirement. Prefix queries that don't meet this requirement return an
+ error to users. The limit may be overridden on a per-query basis by specifying a
+ 'minPrefixQueryTermLength' local-param value.
+
+ The flag value of '-1' can be used to disable enforcement of this limit.
+ -->
+ <minPrefixQueryTermLength>${solr.query.minPrefixLength:-1}</minPrefixQueryTermLength>
+
+ <!-- Cache specification for Filters or DocSets - unordered set of *all* documents
+ that match a particular query.
+ -->
+ <filterCache
+ size="512"
+ initialSize="512"
+ autowarmCount="2"
+ async="${solr.filterCache.async:false}"/>
+
+ <queryResultCache
+ size="512"
+ initialSize="512"
+ autowarmCount="2"/>
+
+ <documentCache
+ size="512"
+ initialSize="512"
+ autowarmCount="0"/>
+
+ <cache name="perSegFilter"
+ class="solr.CaffeineCache"
+ size="10"
+ initialSize="0"
+ autowarmCount="10" />
+
+ <!-- If true, stored fields that are not requested will be loaded lazily.
+ -->
+ <enableLazyFieldLoading>true</enableLazyFieldLoading>
+
+ <!--
+
+ <cache name="myUserCache"
+ size="4096"
+ initialSize="1024"
+ autowarmCount="1024"
+ regenerator="MyRegenerator"
+ />
+ -->
+
+ <!--
+ <useFilterForSortedQuery>true</useFilterForSortedQuery>
+ -->
+
+ <queryResultWindowSize>10</queryResultWindowSize>
+
+ <!-- boolToFilterOptimizer converts boolean clauses with zero boost
+ into cached filters if the number of docs selected by the clause exceeds
+ the threshold (represented as a fraction of the total index)
+ -->
+ <boolTofilterOptimizer enabled="false" cacheSize="32" threshold=".05"/>
Review Comment:
that thing is ancient
##########
solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.plugin.NamedListInitializedPlugin;
+
+/**
+ * The QueryAndResponseCombiner class is an abstract base class for combining query results and
+ * shard documents. It provides a framework for different algorithms to be implemented for merging
+ * ranked lists and shard documents.
+ */
+public abstract class QueryAndResponseCombiner implements NamedListInitializedPlugin {
+ /**
+ * Combines multiple ranked lists into a single QueryResult.
+ *
+ * @param rankedLists a list of ranked lists to be combined
+ * @param solrParams params to be used when provided at query time
+ * @return a new QueryResult containing the combined results
+ * @throws IllegalArgumentException if the input list is empty
+ */
+ public abstract QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams);
+
+ /**
+ * Combines shard documents based on the provided map.
+ *
+ * @param shardDocMap a map where keys represent shard IDs and values are lists of ShardDocs for
+ * each shard
+ * @param solrParams params to be used when provided at query time
+ * @return a combined list of ShardDocs from all shards
+ */
+ public abstract List<ShardDoc> combine(
+ Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams);
+
+ /**
+ * Retrieves a list of explanations for the given queries and results.
+ *
+ * @param queryKeys the keys associated with the queries
+ * @param queries the list of queries for which explanations are requested
+ * @param queryResult the list of QueryResult corresponding to each query
+ * @param searcher the SolrIndexSearcher used to perform the search
+ * @param schema the IndexSchema used to interpret the search results
+ * @param solrParams params to be used when provided at query time
+ * @return a list of explanations for the given queries and results
+ * @throws IOException if an I/O error occurs during the explanation retrieval process
+ */
+ public abstract NamedList<Explanation> getExplanations(
+ String[] queryKeys,
+ List<Query> queries,
+ List<QueryResult> queryResult,
+ SolrIndexSearcher searcher,
+ IndexSchema schema,
+ SolrParams solrParams)
+ throws IOException;
+
+ /**
+ * Retrieves an implementation of the QueryAndResponseCombiner based on the specified algorithm.
+ *
+ * @param algorithm the combiner algorithm
+ * @param combiners The already initialised map of QueryAndResponseCombiner
+ * @return an instance of QueryAndResponseCombiner corresponding to the specified algorithm.
+ * @throws SolrException if an unknown combiner algorithm is specified.
+ */
+ public static QueryAndResponseCombiner getImplementation(
+ String algorithm, Map<String, QueryAndResponseCombiner> combiners) {
+ if (combiners.containsKey(algorithm)) {
Review Comment:
nit: we prefer to avoid double-lookup. Just call "get" and check if null.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
Review Comment:
nit: always statics above non-statics
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
Review Comment:
very weird to see methods above the constructor. Please don't do that.
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
+ NamedList<?> all = (NamedList<?>) initParams.get("combiners");
+ for (int i = 0; i < all.size(); i++) {
+ String name = all.getName(i);
+ NamedList<?> combinerConfig = (NamedList<?>) all.getVal(i);
+ String className = (String) combinerConfig.get("class");
+ QueryAndResponseCombiner combiner =
+ core.getResourceLoader().newInstance(className, QueryAndResponseCombiner.class);
+ combiner.init(combinerConfig);
+ combiners.compute(
+ name,
+ (k, existingCombiner) -> {
+ if (existingCombiner == null) {
+ return combiner;
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Found more than one combiner with same name");
+ });
+ }
+ Object maxQueries = initParams.get("maxCombinerQueries");
+ if (maxQueries != null) {
+ this.maxCombinerQueries = Integer.parseInt(maxQueries.toString());
+ }
+ }
+ combiners.computeIfAbsent(
+ CombinerParams.RECIPROCAL_RANK_FUSION,
+ key -> {
+ ReciprocalRankFusion reciprocalRankFusion = new ReciprocalRankFusion();
+ reciprocalRankFusion.init(initParams);
+ return reciprocalRankFusion;
+ });
+ }
+
+ /**
+ * Overrides the prepare method to handle combined queries.
+ *
+ * @param rb the ResponseBuilder to prepare
+ * @throws IOException if an I/O error occurs during preparation
+ */
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ SolrParams params = crb.req.getParams();
+ if (params.get(CursorMarkParams.CURSOR_MARK_PARAM) != null
+ || params.getBool(GroupParams.GROUP, false)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unsupported functionality for Combined Queries.");
+ }
+ String[] queriesToCombineKeys = params.getParams(CombinerParams.COMBINER_QUERY);
+ if (queriesToCombineKeys.length > maxCombinerQueries) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Too many queries to combine: limit is " + maxCombinerQueries);
+ }
+ for (String queryKey : queriesToCombineKeys) {
+ final var unparsedQuery = params.get(queryKey);
+ ResponseBuilder rbNew = new ResponseBuilder(rb.req, new SolrQueryResponse(), rb.components);
+ rbNew.setQueryString(unparsedQuery);
+ super.prepare(rbNew);
+ crb.responseBuilders.add(rbNew);
+ }
+ }
+ super.prepare(rb);
+ }
+
+ /**
+ * Overrides the process method to handle CombinedQueryResponseBuilder instances. This method
+ * processes the responses from multiple shards, combines them using the specified
+ * QueryAndResponseCombiner strategy, and sets the appropriate results and metadata in the
+ * CombinedQueryResponseBuilder.
+ *
+ * @param rb the ResponseBuilder object to process
+ * @throws IOException if an I/O error occurs during processing
+ */
+ @Override
+ public void process(ResponseBuilder rb) throws IOException {
+ if (rb instanceof CombinedQueryResponseBuilder crb) {
+ boolean partialResults = false;
+ boolean segmentTerminatedEarly = false;
+ boolean setMaxHitsTerminatedEarly = false;
+ List<QueryResult> queryResults = new ArrayList<>();
+ for (ResponseBuilder rbNow : crb.responseBuilders) {
+ // Just a placeholder for future implementation for Cursors
+ rbNow.setCursorMark(crb.getCursorMark());
+ super.process(rbNow);
+ DocListAndSet docListAndSet = rbNow.getResults();
+ QueryResult queryResult = new QueryResult();
+ queryResult.setDocListAndSet(docListAndSet);
+ queryResults.add(queryResult);
+ partialResults |= queryResult.isPartialResults();
+ if (queryResult.getSegmentTerminatedEarly() != null) {
+ segmentTerminatedEarly |= queryResult.getSegmentTerminatedEarly();
+ }
+ if (queryResult.getMaxHitsTerminatedEarly() != null) {
+ setMaxHitsTerminatedEarly |= queryResult.getMaxHitsTerminatedEarly();
+ }
+ }
+ String algorithm =
+ rb.req
+ .getParams()
+ .get(CombinerParams.COMBINER_ALGORITHM, CombinerParams.DEFAULT_COMBINER);
+ QueryAndResponseCombiner combinerStrategy =
+ QueryAndResponseCombiner.getImplementation(algorithm, combiners);
+ QueryResult combinedQueryResult = combinerStrategy.combine(queryResults, rb.req.getParams());
+ combinedQueryResult.setPartialResults(partialResults);
+ combinedQueryResult.setSegmentTerminatedEarly(segmentTerminatedEarly);
+ combinedQueryResult.setMaxHitsTerminatedEarly(setMaxHitsTerminatedEarly);
+ crb.setResult(combinedQueryResult);
+ if (rb.isDebug()) {
+ String[] queryKeys = rb.req.getParams().getParams(CombinerParams.COMBINER_QUERY);
+ List<Query> queries = crb.responseBuilders.stream().map(ResponseBuilder::getQuery).toList();
+ NamedList<Explanation> explanations =
+ combinerStrategy.getExplanations(
+ queryKeys,
+ queries,
+ queryResults,
+ rb.req.getSearcher(),
+ rb.req.getSchema(),
+ rb.req.getParams());
+ rb.addDebugInfo("combinerExplanations", explanations);
+ }
+ ResultContext ctx = new BasicResultContext(crb);
+ crb.rsp.addResponse(ctx);
+ crb.rsp
+ .getToLog()
Review Comment:
nit: please use the convenience method addToLog(...)
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.component;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.CursorMarkParams;
+import org.apache.solr.common.params.GroupParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.response.BasicResultContext;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocListAndSet;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrReturnFields;
+import org.apache.solr.search.SortSpec;
+import org.apache.solr.search.combine.QueryAndResponseCombiner;
+import org.apache.solr.search.combine.ReciprocalRankFusion;
+import org.apache.solr.util.SolrResponseUtil;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The CombinedQueryComponent class extends QueryComponent and provides support for executing
+ * multiple queries and combining their results.
+ */
+public class CombinedQueryComponent extends QueryComponent implements SolrCoreAware {
+
+ public static final String COMPONENT_NAME = "combined_query";
+ protected NamedList<?> initParams;
+ private Map<String, QueryAndResponseCombiner> combiners = new ConcurrentHashMap<>();
+ private int maxCombinerQueries;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ this.initParams = args;
+ this.maxCombinerQueries = CombinerParams.DEFAULT_MAX_COMBINER_QUERIES;
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ if (initParams != null && initParams.size() > 0) {
+ log.info("Initializing CombinedQueryComponent");
Review Comment:
It's common to work on a new component and log at info but that's pretty noisy. Imagine if every search component did that!
##########
solr/core/src/test-files/solr/collection1/conf/solrconfig-combined-query.xml:
##########
@@ -0,0 +1,555 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- This is a "kitchen sink" config file that tests can use.
+ When writting a new test, feel free to add *new* items (plugins,
+ config options, etc...) as long as they don't break any existing
+ tests. if you need to test something esoteric please add a new
+ "solrconfig-your-esoteric-purpose.xml" config file.
+
+ Note in particular that this test is used by MinimalSchemaTest so
+ Anything added to this file needs to work correctly even if there
+ is now uniqueKey or defaultSearch Field.
+ -->
+
+<config>
Review Comment:
we may need a test config file but these things are a burden for the Solr project to maintain over time, especially when they look kitchen-sink like this one looks like. It's better to start with a "minimal" config and only add what you need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2286964590
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
> Solr DocSet is used to hold all doc IDs that match a query, ignoring rows or start params
Absolutely Right!
> It's only for a non-distrib request (distrib=false)
It's for distributed as well as distributed request have also to go through this phase through ShardHandlerRequest. Only difference being that it sets the start=0 and row+=start. Then, final merge happens after collecting the results from all ShardHandlers. Am I right?
This method is invoked right after the SolrSearchIndexer call for all the sub-queries. It's not creating DocSet rather modifying the existing one, returned from SolrSearchIndexer. The complete flow is like:
1. Prepare all the sub-queries in terms of parsing basis the QueryParser.
2. Process all the sub-queries by sending it across the SolrSearchIndexer which will return the `QueryResult` having `DocSet`.
3. The `ReciprocalRankFusion` then rescores them and creates a modified `DocSet`.
Let me know if that makes sense.
##########
solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.combine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TotalHits;
+import org.apache.solr.common.params.CombinerParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.handler.component.ShardDoc;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QueryResult;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SortedIntDocSet;
+
+/**
+ * The ReciprocalRankFusion class implements a query and response combiner that uses the Reciprocal
+ * Rank Fusion (RRF) algorithm to combine multiple ranked lists into a single ranked list.
+ */
+public class ReciprocalRankFusion extends QueryAndResponseCombiner {
+
+ private int k;
+
+ public int getK() {
+ return k;
+ }
+
+ public ReciprocalRankFusion() {
+ this.k = CombinerParams.COMBINER_RRF_K_DEFAULT;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ Object kParam = args.get("k");
+ if (kParam != null) {
+ this.k = Integer.parseInt(kParam.toString());
+ }
+ }
+
+ @Override
+ public QueryResult combine(List<QueryResult> rankedLists, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ List<DocList> docLists = getDocListsFromQueryResults(rankedLists);
+ QueryResult combinedResult = new QueryResult();
+ combineResults(combinedResult, docLists, false, kVal);
+ return combinedResult;
+ }
+
+ private static List<DocList> getDocListsFromQueryResults(List<QueryResult> rankedLists) {
+ List<DocList> docLists = new ArrayList<>(rankedLists.size());
+ for (QueryResult rankedList : rankedLists) {
+ docLists.add(rankedList.getDocList());
+ }
+ return docLists;
+ }
+
+ @Override
+ public List<ShardDoc> combine(Map<String, List<ShardDoc>> shardDocMap, SolrParams solrParams) {
+ int kVal = solrParams.getInt(CombinerParams.COMBINER_RRF_K, this.k);
+ HashMap<String, Float> docIdToScore = new HashMap<>();
+ Map<String, ShardDoc> docIdToShardDoc = new HashMap<>();
+ List<ShardDoc> finalShardDocList = new ArrayList<>();
+ for (Map.Entry<String, List<ShardDoc>> shardDocEntry : shardDocMap.entrySet()) {
+ List<ShardDoc> shardDocList = shardDocEntry.getValue();
+ int ranking = 1;
+ while (ranking <= shardDocList.size()) {
+ String docId = shardDocList.get(ranking - 1).id.toString();
+ docIdToShardDoc.put(docId, shardDocList.get(ranking - 1));
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<String, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+ for (Map.Entry<String, Float> scoredDoc : sortedByScoreDescending) {
+ String docId = scoredDoc.getKey();
+ Float score = scoredDoc.getValue();
+ ShardDoc shardDoc = docIdToShardDoc.get(docId);
+ shardDoc.score = score;
+ finalShardDocList.add(shardDoc);
+ }
+ return finalShardDocList;
+ }
+
+ private Map<Integer, Integer[]> combineResults(
+ QueryResult combinedRankedList,
+ List<DocList> rankedLists,
+ boolean saveRankPositionsForExplain,
+ int kVal) {
+ Map<Integer, Integer[]> docIdToRanks = null;
+ HashMap<Integer, Float> docIdToScore = new HashMap<>();
+ long totalMatches = 0;
+ for (DocList rankedList : rankedLists) {
+ DocIterator docs = rankedList.iterator();
+ totalMatches = Math.max(totalMatches, rankedList.matches());
+ int ranking = 1;
+ while (docs.hasNext()) {
+ int docId = docs.nextDoc();
+ float rrfScore = 1f / (kVal + ranking);
+ docIdToScore.compute(docId, (id, score) -> (score == null) ? rrfScore : score + rrfScore);
+ ranking++;
+ }
+ }
+ List<Map.Entry<Integer, Float>> sortedByScoreDescending =
+ docIdToScore.entrySet().stream()
+ .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
+ .toList();
+
+ int combinedResultsLength = docIdToScore.size();
+ int[] combinedResultsDocIds = new int[combinedResultsLength];
+ float[] combinedResultScores = new float[combinedResultsLength];
+
+ int i = 0;
+ for (Map.Entry<Integer, Float> scoredDoc : sortedByScoreDescending) {
+ combinedResultsDocIds[i] = scoredDoc.getKey();
+ combinedResultScores[i] = scoredDoc.getValue();
+ i++;
+ }
+
+ if (saveRankPositionsForExplain) {
+ docIdToRanks = getRanks(rankedLists, combinedResultsDocIds);
+ }
+
+ DocSlice combinedResultSlice =
+ new DocSlice(
+ 0,
+ combinedResultsLength,
+ combinedResultsDocIds,
+ combinedResultScores,
+ Math.max(combinedResultsLength, totalMatches),
+ combinedResultScores.length > 0 ? combinedResultScores[0] : 0,
+ TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO);
+ combinedRankedList.setDocList(combinedResultSlice);
+ SortedIntDocSet docSet = new SortedIntDocSet(combinedResultsDocIds, combinedResultsLength);
Review Comment:
> Solr DocSet is used to hold all doc IDs that match a query, ignoring rows or start params
Absolutely Right!
> It's only for a non-distrib request (distrib=false)
It's for distributed as well as distributed request have also to go through this phase through ShardHandlerRequest. Only difference being that it sets the start=0 and row+=start. Then, final merge happens after collecting the results from all ShardHandlers. Am I right?
This method is invoked right after the SolrSearchIndexer call for all the sub-queries. It's not creating DocSet rather modifying the existing one, returned from SolrSearchIndexer. The complete flow is like:
1. Prepare all the sub-queries in terms of parsing basis the QueryParser.
2. Process all the sub-queries by sending it across the SolrSearchIndexer which will return the `QueryResult` having `DocSet`.
3. The `ReciprocalRankFusion` then rescores them and creates a modified `DocSet`.
Let me know if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2334303365
##########
solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java:
##########
@@ -81,4 +88,33 @@ protected void postPrepareComponents(ResponseBuilder rb) {
crb.propagate();
}
}
+
+ /**
+ * rb.distrib {@link ResponseBuilder} must be set for the combined query to work in case of single
+ * core standalone mode. This method set the parameter explicitly with other required solr param
+ * i.e. shards
+ *
+ * @param req the SolrQueryRequest
+ * @return boolean denoting whether the request can be marked as distributed.
+ */
+ @Override
+ protected boolean isDistrib(SolrQueryRequest req) {
+ boolean isDistrib = super.isDistrib(req);
+ if (!isDistrib
+ && !req.getParams().getBool(ShardParams.IS_SHARD, false)
+ && req.getHttpSolrCall() != null) {
+ log.info("Configuring distributed mode to enable Combined Query.");
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ String scheme = req.getHttpSolrCall().getReq().getScheme();
+ String host = req.getHttpSolrCall().getReq().getServerName();
+ int port = req.getHttpSolrCall().getReq().getServerPort();
+ String context = req.getHttpSolrCall().getReq().getContextPath();
+ String core = req.getCore().getName();
+ String localShardUrl = String.format("%s://%s:%d%s/%s", scheme, host, port, context, core);
+ solrParams.set(ShardParams.SHARDS, localShardUrl);
+ req.setParams(solrParams);
Review Comment:
It's not just standalone; a single shard collection will "short-circuit". I was going to link to my same comment. I understand the need/desire. Interestingly, this is the first component to want to prevent the short circuit, but I could see it being useful for any search component author who doesn't want the extra development cost of an optimized single-shard algorithm.
Perhaps if SearchHandler called SearchComponent.prepare before it initialized ShardHandler (that latter part needs to know if short-circuit), then a component could add the short-circuit param.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2591563892
##########
solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;
Review Comment:
The original intention was to group the *Handler and *Component classes within the same package, as done previously. However, there is no issue with placing them in the combine package instead.
It will affect [this](https://github.com/apache/solr/pull/3418/files#diff-c857e7327c8c35700144083caf8d079b87622fa67aabad3de51728e32287f16bR110) where we might deviate from usual convention of having `solr.<ComponentName>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17319 : Combined Query Feature for Multi Query Execution [solr]
Posted by "ercsonusharma (via GitHub)" <gi...@apache.org>.
ercsonusharma commented on code in PR #3418:
URL: https://github.com/apache/solr/pull/3418#discussion_r2453918348
##########
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
##########
@@ -500,6 +505,28 @@ public void prepDistributed(ResponseBuilder rb) {
}
}
+ private static void forceDistributed(ResponseBuilder rb) {
+ SolrQueryRequest req = rb.req;
+ ModifiableSolrParams solrParams = new ModifiableSolrParams(req.getParams());
+ solrParams.set("shortCircuit", false);
+ req.setParams(solrParams);
+ if (req.getHttpSolrCall() != null
+ && StringUtils.isEmpty(req.getParams().get(ShardParams.SHARDS))) {
Review Comment:
Yeah, definitely - A SolrCloud test can be introduced by extending `AbstractFullDistribZkTestBase`. Will send it out shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org