You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "gaobinlong (via GitHub)" <gi...@apache.org> on 2026/01/23 07:52:20 UTC

[PR] Introduce TopGroupsCollectorManager [lucene]

gaobinlong opened a new pull request, #15603:
URL: https://github.com/apache/lucene/pull/15603

   ### Description
   
   This pr introduces TopGroupsCollectorManager and switches BaseGroupSelectorTestCase to use search concurrency and move away from the deprecated search(Query, Collector) method.
   
   Relates to https://github.com/apache/lucene/issues/12892.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Introduce TopGroupsCollectorManager [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15603:
URL: https://github.com/apache/lucene/pull/15603#issuecomment-4282422794

   It was targeted against 11. If you think it should apply to 10x, please backport or file a separate PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Introduce TopGroupsCollectorManager [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #15603:
URL: https://github.com/apache/lucene/pull/15603#discussion_r3011736487


##########
lucene/grouping/src/java/org/apache/lucene/search/grouping/TopGroupsCollectorManager.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search.grouping;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Supplier;
+import org.apache.lucene.search.CollectorManager;
+import org.apache.lucene.search.Sort;
+
+/** A CollectorManager implementation for TopGroupsCollector. */
+public class TopGroupsCollectorManager<T>
+    implements CollectorManager<TopGroupsCollector<T>, TopGroups<T>> {
+
+  private final Supplier<GroupSelector<T>> groupSelectorFactory;
+  private final Collection<SearchGroup<T>> searchGroups;
+  private final Sort groupSort;
+  private final Sort sortWithinGroup;
+  private final int maxDocsPerGroup;
+  private final boolean getMaxScores;
+  private final TopGroups.ScoreMergeMode scoreMergeMode;
+  private final List<TopGroupsCollector<T>> collectors;
+
+  /**
+   * Creates a new TopGroupsCollectorManager.
+   *
+   * @param groupSelectorFactory factory to create group selectors for each collector
+   * @param searchGroups the search groups from the first pass
+   * @param groupSort the sort to use for groups
+   * @param sortWithinGroup the sort to use within each group
+   * @param maxDocsPerGroup the maximum number of documents per group
+   * @param getMaxScores whether to compute max scores
+   */
+  public TopGroupsCollectorManager(
+      Supplier<GroupSelector<T>> groupSelectorFactory,
+      Collection<SearchGroup<T>> searchGroups,
+      Sort groupSort,
+      Sort sortWithinGroup,
+      int maxDocsPerGroup,
+      boolean getMaxScores) {
+    this(
+        groupSelectorFactory,
+        searchGroups,
+        groupSort,
+        sortWithinGroup,
+        maxDocsPerGroup,
+        getMaxScores,
+        TopGroups.ScoreMergeMode.None);
+  }
+
+  /**
+   * Creates a new TopGroupsCollectorManager.
+   *
+   * @param groupSelectorFactory factory to create group selectors for each collector
+   * @param searchGroups the search groups from the first pass
+   * @param groupSort the sort to use for groups
+   * @param sortWithinGroup the sort to use within each group
+   * @param maxDocsPerGroup the maximum number of documents per group
+   * @param getMaxScores whether to compute max scores
+   * @param scoreMergeMode the mode for merging scores across shards
+   */
+  public TopGroupsCollectorManager(
+      Supplier<GroupSelector<T>> groupSelectorFactory,
+      Collection<SearchGroup<T>> searchGroups,
+      Sort groupSort,
+      Sort sortWithinGroup,
+      int maxDocsPerGroup,
+      boolean getMaxScores,
+      TopGroups.ScoreMergeMode scoreMergeMode) {
+    this.groupSelectorFactory = groupSelectorFactory;
+    this.searchGroups = searchGroups;
+    this.groupSort = groupSort;
+    this.sortWithinGroup = sortWithinGroup;
+    this.maxDocsPerGroup = maxDocsPerGroup;
+    this.getMaxScores = getMaxScores;
+    this.scoreMergeMode = scoreMergeMode;
+    this.collectors = new ArrayList<>();
+  }
+
+  @Override
+  public TopGroupsCollector<T> newCollector() throws IOException {
+    TopGroupsCollector<T> collector =
+        new TopGroupsCollector<>(
+            groupSelectorFactory.get(),
+            searchGroups,
+            groupSort,
+            sortWithinGroup,
+            maxDocsPerGroup,
+            getMaxScores);
+    collectors.add(collector);
+    return collector;
+  }
+
+  @Override
+  public TopGroups<T> reduce(Collection<TopGroupsCollector<T>> collectors) throws IOException {
+    if (collectors.isEmpty()) {
+      return null;
+    }
+
+    if (collectors.size() == 1) {
+      return collectors.iterator().next().getTopGroups(0);
+    }
+
+    // Merge results from multiple collectors
+    List<TopGroups<T>> shardGroupsList = new ArrayList<>();
+    for (TopGroupsCollector<T> collector : collectors) {
+      TopGroups<T> groups = collector.getTopGroups(0);
+      if (groups != null) {
+        shardGroupsList.add(groups);
+      }
+    }
+
+    if (shardGroupsList.isEmpty()) {
+      return null;
+    }
+
+    @SuppressWarnings({"unchecked", "rawtypes"})
+    TopGroups<T>[] shardGroups = (TopGroups<T>[]) shardGroupsList.toArray(new TopGroups[0]);

Review Comment:
   ```suggestion
       TopGroups<T>[] shardGroups = (TopGroups<T>[]) shardGroupsList.toArray(TopGroups[]::new);
   ```



##########
lucene/grouping/src/java/org/apache/lucene/search/grouping/TopGroupsCollectorManager.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search.grouping;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Supplier;
+import org.apache.lucene.search.CollectorManager;
+import org.apache.lucene.search.Sort;
+
+/** A CollectorManager implementation for TopGroupsCollector. */
+public class TopGroupsCollectorManager<T>
+    implements CollectorManager<TopGroupsCollector<T>, TopGroups<T>> {
+
+  private final Supplier<GroupSelector<T>> groupSelectorFactory;
+  private final Collection<SearchGroup<T>> searchGroups;
+  private final Sort groupSort;
+  private final Sort sortWithinGroup;
+  private final int maxDocsPerGroup;
+  private final boolean getMaxScores;
+  private final TopGroups.ScoreMergeMode scoreMergeMode;
+  private final List<TopGroupsCollector<T>> collectors;
+
+  /**
+   * Creates a new TopGroupsCollectorManager.
+   *
+   * @param groupSelectorFactory factory to create group selectors for each collector
+   * @param searchGroups the search groups from the first pass
+   * @param groupSort the sort to use for groups
+   * @param sortWithinGroup the sort to use within each group
+   * @param maxDocsPerGroup the maximum number of documents per group
+   * @param getMaxScores whether to compute max scores
+   */
+  public TopGroupsCollectorManager(
+      Supplier<GroupSelector<T>> groupSelectorFactory,
+      Collection<SearchGroup<T>> searchGroups,
+      Sort groupSort,
+      Sort sortWithinGroup,
+      int maxDocsPerGroup,
+      boolean getMaxScores) {
+    this(
+        groupSelectorFactory,
+        searchGroups,
+        groupSort,
+        sortWithinGroup,
+        maxDocsPerGroup,
+        getMaxScores,
+        TopGroups.ScoreMergeMode.None);
+  }
+
+  /**
+   * Creates a new TopGroupsCollectorManager.
+   *
+   * @param groupSelectorFactory factory to create group selectors for each collector
+   * @param searchGroups the search groups from the first pass
+   * @param groupSort the sort to use for groups
+   * @param sortWithinGroup the sort to use within each group
+   * @param maxDocsPerGroup the maximum number of documents per group
+   * @param getMaxScores whether to compute max scores
+   * @param scoreMergeMode the mode for merging scores across shards
+   */
+  public TopGroupsCollectorManager(
+      Supplier<GroupSelector<T>> groupSelectorFactory,
+      Collection<SearchGroup<T>> searchGroups,
+      Sort groupSort,
+      Sort sortWithinGroup,
+      int maxDocsPerGroup,
+      boolean getMaxScores,
+      TopGroups.ScoreMergeMode scoreMergeMode) {
+    this.groupSelectorFactory = groupSelectorFactory;
+    this.searchGroups = searchGroups;
+    this.groupSort = groupSort;
+    this.sortWithinGroup = sortWithinGroup;
+    this.maxDocsPerGroup = maxDocsPerGroup;
+    this.getMaxScores = getMaxScores;
+    this.scoreMergeMode = scoreMergeMode;
+    this.collectors = new ArrayList<>();
+  }
+
+  @Override
+  public TopGroupsCollector<T> newCollector() throws IOException {
+    TopGroupsCollector<T> collector =
+        new TopGroupsCollector<>(
+            groupSelectorFactory.get(),
+            searchGroups,
+            groupSort,
+            sortWithinGroup,
+            maxDocsPerGroup,
+            getMaxScores);
+    collectors.add(collector);
+    return collector;
+  }
+
+  @Override
+  public TopGroups<T> reduce(Collection<TopGroupsCollector<T>> collectors) throws IOException {
+    if (collectors.isEmpty()) {
+      return null;
+    }
+
+    if (collectors.size() == 1) {
+      return collectors.iterator().next().getTopGroups(0);
+    }
+
+    // Merge results from multiple collectors
+    List<TopGroups<T>> shardGroupsList = new ArrayList<>();
+    for (TopGroupsCollector<T> collector : collectors) {
+      TopGroups<T> groups = collector.getTopGroups(0);
+      if (groups != null) {
+        shardGroupsList.add(groups);
+      }
+    }
+
+    if (shardGroupsList.isEmpty()) {
+      return null;
+    }
+
+    @SuppressWarnings({"unchecked", "rawtypes"})
+    TopGroups<T>[] shardGroups = (TopGroups<T>[]) shardGroupsList.toArray(new TopGroups[0]);

Review Comment:
   I wonder if we could just alter TopGroups.merge to accept  list... then we wouldn't need those suppressions. Can be another patch though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Introduce TopGroupsCollectorManager [lucene]

Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #15603:
URL: https://github.com/apache/lucene/pull/15603#issuecomment-4280619980

   It appears that this change was not backported, any reason why it shouldn't be?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Introduce TopGroupsCollectorManager [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15603:
URL: https://github.com/apache/lucene/pull/15603#issuecomment-3863145236

   This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Introduce TopGroupsCollectorManager [lucene]

Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #15603:
URL: https://github.com/apache/lucene/pull/15603#issuecomment-4311722885

   I just backported it to 10.x and updated changes on main to reflect that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Introduce TopGroupsCollectorManager [lucene]

Posted by "gaobinlong (via GitHub)" <gi...@apache.org>.
gaobinlong commented on PR #15603:
URL: https://github.com/apache/lucene/pull/15603#issuecomment-4152852318

   @jpountz @romseygeek @dweiss could you help to review this PR, thank you~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Introduce TopGroupsCollectorManager [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss merged PR #15603:
URL: https://github.com/apache/lucene/pull/15603


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org