You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2024/12/13 02:28:26 UTC
[PR] topn with granularity regression fixes (druid)
clintropolis opened a new pull request, #17565:
URL: https://github.com/apache/druid/pull/17565
### Description
changes:
* fix issue where topN with query granularity other than ALL would use the heap algorithm when it was actual able to use the pooled algorithm, and incorrectly used the pool algorithm in cases where it must use the heap algorithm, a regression from #16533
* fix issue where topN with query granularity other than ALL could incorrectly process values in the wrong time bucket, another regression from #16533
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1883700723
##########
processing/src/main/java/org/apache/druid/query/topn/TopNQueryEngine.java:
##########
@@ -275,7 +275,7 @@ private static boolean canUsePooledAlgorithm(
final int numBytesToWorkWith = resultsBuf.capacity();
final int numValuesPerPass = numBytesPerRecord > 0 ? numBytesToWorkWith / numBytesPerRecord : cardinality;
- return numValuesPerPass <= cardinality;
+ return numValuesPerPass >= cardinality;
Review Comment:
yea this is fair, it is not really a problem in practice because currently the column capabilities should not report as dictionary encoded if the cardinality is -1, so we return false earlier in the method before we get here, but this seems worth explicitly checking for just in case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1884586877
##########
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByQueryEngine.java:
##########
@@ -391,7 +391,7 @@ public boolean hasNext()
if (delegate != null && delegate.hasNext()) {
return true;
} else {
- if (!cursor.isDone() && granularizer.currentOffsetWithinBucket()) {
+ if (granularizer.currentOffsetWithinBucket()) {
Review Comment:
yea, i noticed granularizer.currentOffsetWithinBucket also checks isDone so it seemed nicer this way, this didn't cause any issues, just noticed while i was looking over stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1884586877
##########
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByQueryEngine.java:
##########
@@ -391,7 +391,7 @@ public boolean hasNext()
if (delegate != null && delegate.hasNext()) {
return true;
} else {
- if (!cursor.isDone() && granularizer.currentOffsetWithinBucket()) {
+ if (granularizer.currentOffsetWithinBucket()) {
Review Comment:
yea, i noticed granularizer.currentOffsetWithinBucket also checks isDone so it seemed nicer this way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1884592374
##########
processing/src/main/java/org/apache/druid/query/topn/Generic1AggPooledTopNScannerPrototype.java:
##########
@@ -54,25 +54,27 @@ public long scanAndAggregate(
{
long processedRows = 0;
int positionToAllocate = 0;
- while (!cursor.isDoneOrInterrupted()) {
- final IndexedInts dimValues = dimensionSelector.getRow();
- final int dimSize = dimValues.size();
- for (int i = 0; i < dimSize; i++) {
- int dimIndex = dimValues.get(i);
- int position = positions[dimIndex];
- if (position >= 0) {
- aggregator.aggregate(resultsBuffer, position);
- } else if (position == TopNAlgorithm.INIT_POSITION_VALUE) {
- positions[dimIndex] = positionToAllocate;
- position = positionToAllocate;
- aggregator.init(resultsBuffer, position);
- aggregator.aggregate(resultsBuffer, position);
- positionToAllocate += aggregatorSize;
+ if (granularizer.currentOffsetWithinBucket()) {
Review Comment:
yea, this was a bug, i did it as an if statement wrapping the loop so we wouldn't need to check `currentOffsetWithinBucket()` twice per loop since the granularizer advance methods also call `currentOffsetWithinBucket`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1887723392
##########
processing/src/main/java/org/apache/druid/query/QueryContexts.java:
##########
@@ -89,6 +89,7 @@ public class QueryContexts
public static final String UNCOVERED_INTERVALS_LIMIT_KEY = "uncoveredIntervalsLimit";
public static final String MIN_TOP_N_THRESHOLD = "minTopNThreshold";
public static final String CATALOG_VALIDATION_ENABLED = "catalogValidationEnabled";
+ public static final String TOPN_USE_MULTI_PASS_POOLED_QUERY_GRANULARITY = "topNuseMultiPassPooledQueryGranularity";
Review Comment:
Please add a javadoc comment here explaining what this is for, and linking to this PR (or a related issue). It's helpful to have that kind of thing for any undocumented parameter.
##########
integration-tests/src/test/resources/queries/twitterstream_queries.json:
##########
@@ -19,8 +19,8 @@
}
],
"context": {
- "useCache": "true",
- "populateCache": "true",
+ "useCache": "false",
Review Comment:
why change this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1887883616
##########
integration-tests/src/test/resources/queries/twitterstream_queries.json:
##########
@@ -19,8 +19,8 @@
}
],
"context": {
- "useCache": "true",
- "populateCache": "true",
+ "useCache": "false",
Review Comment:
was trying to see if could allow the tests to pass without setting the new context parameter, but it didn't seem to help, so changed to use the new context parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1888911023
##########
processing/src/main/java/org/apache/druid/query/CursorGranularizer.java:
##########
@@ -112,12 +115,14 @@ public static CursorGranularizer create(
private CursorGranularizer(
Cursor cursor,
+ Granularity granularity,
Review Comment:
@clintropolis Do we need this field ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe merged PR #17565:
URL: https://github.com/apache/druid/pull/17565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1883491417
##########
processing/src/main/java/org/apache/druid/query/topn/TopNQueryEngine.java:
##########
@@ -275,7 +275,7 @@ private static boolean canUsePooledAlgorithm(
final int numBytesToWorkWith = resultsBuf.capacity();
final int numValuesPerPass = numBytesPerRecord > 0 ? numBytesToWorkWith / numBytesPerRecord : cardinality;
- return numValuesPerPass <= cardinality;
+ return numValuesPerPass >= cardinality;
Review Comment:
this comparision change caught my eye...the new cmp seem to be more in line with the intention - but it seems to me that
depending on some conditions [here](https://github.com/apache/druid/blob/24e5d8a9e81fffd656e9544ae16bea53d7e60808/processing/src/main/java/org/apache/druid/query/topn/TopNQueryEngine.java#L111-L115) ; this `cardinality`'s value might be `-1` in some cases
is that ok to answer `true` if `cardinality` is `-1` ?
note: if `cardinality == -1` ; then both the old and the new logic returns `true` - is that ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1888954721
##########
processing/src/main/java/org/apache/druid/query/CursorGranularizer.java:
##########
@@ -112,12 +115,14 @@ public static CursorGranularizer create(
private CursorGranularizer(
Cursor cursor,
+ Granularity granularity,
Review Comment:
https://github.com/apache/druid/pull/17579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on PR #17565:
URL: https://github.com/apache/druid/pull/17565#issuecomment-2548830135
Since @gianm comments were non blocking, going ahead with merge since this blocks 31.0.1 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1884554341
##########
processing/src/main/java/org/apache/druid/query/topn/TopNQueryEngine.java:
##########
@@ -245,6 +245,11 @@ private static boolean canUsePooledAlgorithm(
final int numBytesPerRecord
)
{
+ if (cardinality < 0) {
+ // unknown cardinality doesn't work with the pooled algorith which requires an exact count of dictionary ids
Review Comment:
algorithm (spelling)
##########
processing/src/main/java/org/apache/druid/query/topn/Generic1AggPooledTopNScannerPrototype.java:
##########
@@ -54,25 +54,27 @@ public long scanAndAggregate(
{
long processedRows = 0;
int positionToAllocate = 0;
- while (!cursor.isDoneOrInterrupted()) {
- final IndexedInts dimValues = dimensionSelector.getRow();
- final int dimSize = dimValues.size();
- for (int i = 0; i < dimSize; i++) {
- int dimIndex = dimValues.get(i);
- int position = positions[dimIndex];
- if (position >= 0) {
- aggregator.aggregate(resultsBuffer, position);
- } else if (position == TopNAlgorithm.INIT_POSITION_VALUE) {
- positions[dimIndex] = positionToAllocate;
- position = positionToAllocate;
- aggregator.init(resultsBuffer, position);
- aggregator.aggregate(resultsBuffer, position);
- positionToAllocate += aggregatorSize;
+ if (granularizer.currentOffsetWithinBucket()) {
Review Comment:
was the `granularizer.currentOffsetWithinBucket()` check added here (& the other prototypes) to enable skipping of empty granular buckets?
##########
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByQueryEngine.java:
##########
@@ -391,7 +391,7 @@ public boolean hasNext()
if (delegate != null && delegate.hasNext()) {
return true;
} else {
- if (!cursor.isDone() && granularizer.currentOffsetWithinBucket()) {
+ if (granularizer.currentOffsetWithinBucket()) {
Review Comment:
was the `cursor.isDone()` here removed simply because it's redundant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] topn with granularity regression fixes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #17565:
URL: https://github.com/apache/druid/pull/17565#discussion_r1884592374
##########
processing/src/main/java/org/apache/druid/query/topn/Generic1AggPooledTopNScannerPrototype.java:
##########
@@ -54,25 +54,27 @@ public long scanAndAggregate(
{
long processedRows = 0;
int positionToAllocate = 0;
- while (!cursor.isDoneOrInterrupted()) {
- final IndexedInts dimValues = dimensionSelector.getRow();
- final int dimSize = dimValues.size();
- for (int i = 0; i < dimSize; i++) {
- int dimIndex = dimValues.get(i);
- int position = positions[dimIndex];
- if (position >= 0) {
- aggregator.aggregate(resultsBuffer, position);
- } else if (position == TopNAlgorithm.INIT_POSITION_VALUE) {
- positions[dimIndex] = positionToAllocate;
- position = positionToAllocate;
- aggregator.init(resultsBuffer, position);
- aggregator.aggregate(resultsBuffer, position);
- positionToAllocate += aggregatorSize;
+ if (granularizer.currentOffsetWithinBucket()) {
Review Comment:
yea, i did it as an if statement wrapping the loop so we wouldn't need to check `currentOffsetWithinBucket()` twice per loop since the granularizer advance methods also call `currentOffsetWithinBucket`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org