You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "gaobinlong (via GitHub)" <gi...@apache.org> on 2025/09/09 08:49:43 UTC
[PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
gaobinlong opened a new pull request, #15167:
URL: https://github.com/apache/lucene/pull/15167
### Description
Resolve a TODO in FirstPassGroupingCollector, introduce a new parameter `ignoreDocsWithoutGroupField`, if it's true, ignore documents that don't have the group field instead of putting them into a null group.
This change can also improve the field collapsing performance, I did some benchmark test against OpenSearch, here are the result:
```
Query DSL:
GET big5/_search?size=10
{
"query": {
"match": {
"message": "50-136-239-27"
}
},
"collapse": {
"field": "count"
}
}
Before:
| 50th percentile service time | collapsing | 217.497 | ms |
| 90th percentile service time | collapsing | 221.677 | ms |
| 99th percentile service time | collapsing | 236.481 | ms |
| 99.9th percentile service time | collapsing | 266.285 | ms |
| 100th percentile service time | collapsing | 271.294 | ms |
After:
| 50th percentile service time | collapsing | 180.974 | ms |
| 90th percentile service time | collapsing | 185.945 | ms |
| 99th percentile service time | collapsing | 205.012 | ms |
| 99.9th percentile service time | collapsing | 222.983 | ms |
| 100th percentile service time | collapsing | 259.105 | ms |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "gaobinlong (via GitHub)" <gi...@apache.org>.
gaobinlong commented on code in PR #15167:
URL: https://github.com/apache/lucene/pull/15167#discussion_r2333869308
##########
lucene/grouping/src/java/org/apache/lucene/search/grouping/FirstPassGroupingCollector.java:
##########
@@ -363,4 +388,15 @@ protected void doSetNextReader(LeafReaderContext readerContext) throws IOExcepti
public GroupSelector<T> getGroupSelector() {
return groupSelector;
}
+
+ private boolean isNullGroupValue(T groupValue) {
+ if (groupValue == null) {
+ return true;
+ }
+ // For ValueSourceGroupSelector, check if MutableValue exists
+ if (groupValue instanceof MutableValue) {
+ return !((MutableValue) groupValue).exists();
+ }
Review Comment:
Thanks, changed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "alexmm-amzn (via GitHub)" <gi...@apache.org>.
alexmm-amzn commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3273992386
That's the current behavior of `ValueSourceGroupSelector`. Is it the expected one though? What's the purpose of this return value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "gaobinlong (via GitHub)" <gi...@apache.org>.
gaobinlong commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3273405955
> Should this use the return value of `GroupSelector#advanceTo(int)` (returns an enum with `SKIP` or `ACCEPT`) instead of checking for a `null` value? The collector currently ignores it, but it looks like the intended way of skipping a hit.
That doesn't work, ValueSourceGroupSelector.advanceTo() always return ACCEPT:
https://github.com/apache/lucene/blob/main/lucene/grouping/src/java/org/apache/lucene/search/grouping/ValueSourceGroupSelector.java#L69
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "alexmm-amzn (via GitHub)" <gi...@apache.org>.
alexmm-amzn commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3271122626
Should this use the return value of `GroupSelector#advanceTo(int)` (return an enum with `SKIP` and `ACCEPT`) instead of checking for a `null` value? The collector currently ignores it, but it looks like the intended way of skipping a hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3301851113
I agree with @alexmm-amzn, I think this needs to use the return value of `GroupSelector#advanceTo(int)` to distinguish between actual `null` values and missing values, and that `ValueSourceGroupSelector` should be updated to behave more like the other implementations.
This also reminds me that at some point I should get round to deprecating `ValueSource`...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3271089938
I am not 100% familiar with this part of the code base, but it seems like a pretty safe change, and tests prove it out.
I will leave this PR open for a little bit for others to comment if they find issue with it. I will likely merge & backport this next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "gaobinlong (via GitHub)" <gi...@apache.org>.
gaobinlong commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3279193603
> That's the current behavior of `ValueSourceGroupSelector`. Is it the expected one though? What's the purpose of this return value?
By checking the usage, the return value is designed to be used in second pass grouping selector, not for the FirstPassGroupingCollector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on code in PR #15167:
URL: https://github.com/apache/lucene/pull/15167#discussion_r2362275566
##########
lucene/grouping/src/java/org/apache/lucene/search/grouping/ValueSourceGroupSelector.java:
##########
@@ -83,7 +88,11 @@ public MutableValue copyValue() {
public void setGroups(Collection<SearchGroup<MutableValue>> searchGroups) {
secondPassGroups = new HashSet<>();
for (SearchGroup<MutableValue> group : searchGroups) {
- secondPassGroups.add(group.groupValue);
+ if (!group.groupValue.exists()) {
Review Comment:
```suggestion
if (group.groupValue.exists() == false) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on code in PR #15167:
URL: https://github.com/apache/lucene/pull/15167#discussion_r2333292145
##########
lucene/grouping/src/java/org/apache/lucene/search/grouping/FirstPassGroupingCollector.java:
##########
@@ -363,4 +388,15 @@ protected void doSetNextReader(LeafReaderContext readerContext) throws IOExcepti
public GroupSelector<T> getGroupSelector() {
return groupSelector;
}
+
+ private boolean isNullGroupValue(T groupValue) {
+ if (groupValue == null) {
+ return true;
+ }
+ // For ValueSourceGroupSelector, check if MutableValue exists
+ if (groupValue instanceof MutableValue) {
+ return !((MutableValue) groupValue).exists();
+ }
Review Comment:
```suggestion
if (groupValue instanceof MutableValue mutable) {
return mutable.exists() == false;
}
```
##########
lucene/CHANGES.txt:
##########
@@ -315,6 +315,8 @@ Optimizations
and uses an optimized branchless approach. Any subclasses that have implemented the optimized method
need to remove it as it will disappear in Lucene 11. (Uwe Schindler)
+* GITHUB#15167: FirstPassGroupingCollector supports ignoring docs without group field (Binlong Gao)
Review Comment:
let's put this under 10.4, 10.3 is already cut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "gaobinlong (via GitHub)" <gi...@apache.org>.
gaobinlong commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3311149760
Thanks @romseygeek , I've made some change to make the return value of advanceTo() consistent among the group selectors, please help to take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3269591854
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on PR #15167:
URL: https://github.com/apache/lucene/pull/15167#issuecomment-3372165226
Thanks @gaobinlong!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] FirstPassGroupingCollector supports ignoring docs without group field [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek merged PR #15167:
URL: https://github.com/apache/lucene/pull/15167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org