You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/04/30 05:12:01 UTC
[GitHub] [druid] clintropolis opened a new pull request #11185: fix bug with aggregator expressions on realtime index with string columns always producing 0 values
clintropolis opened a new pull request #11185:
URL: https://github.com/apache/druid/pull/11185
Fixes #8970 (well it was already fixed somewhere along the way, but fixes another issue discovered in the comments)
### Description
This PR fixes an issue when using expression virtual columns that reference string values on realtime query segments, where an automatic transform to handle possible multi-valued string inputs forces array output from the selector, which the aggregators are not prepared to handle. See https://github.com/apache/druid/issues/8970#issuecomment-829762963 for details.
This PR adds a modification that fixes the side-effect of these transformations, to allow array typed `ExprEval` to provide scalar values if and only if the array has a single element. Otherwise it will retain its current behavior where methods like `getLong` and `getDouble` return default zero values. This changes makes array expression results mirror the the behavior of scalar expression results, where scalar expressions are currently allowed to present themselves as single element arrays (which exists to handle cases where array functions are used and the input value is a string column is multi-valued in one segment, but single valued in another).
This PR has:
- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #11185: fix bug with aggregator expressions on realtime index with string columns always producing 0 values
Posted by GitBox <gi...@apache.org>.
clintropolis merged pull request #11185:
URL: https://github.com/apache/druid/pull/11185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #11185: fix bug with aggregator expressions on realtime index with string columns always producing 0 values
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #11185:
URL: https://github.com/apache/druid/pull/11185#discussion_r627848046
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprEval.java
##########
@@ -1148,6 +1283,66 @@ public ExprType type()
return ExprType.STRING_ARRAY;
}
+ @Override
+ public boolean isNumericNull()
+ {
+ if (value != null && value.length == 1) {
+ return computeNumber(value[0]) == null;
Review comment:
`computeNumber` seems expensive. Since the caller pattern is likely calling this and then one of `asX()` method below, should we cache the result?
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprEval.java
##########
@@ -1025,6 +1034,54 @@ public ExprType type()
return ExprType.LONG_ARRAY;
}
+ @Override
+ public int asInt()
+ {
+ if (value != null && value.length == 1) {
+ if (value[0] == null) {
Review comment:
nit: you can use `isNumericNull()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #11185: fix bug with aggregator expressions on realtime index with string columns always producing 0 values
Posted by GitBox <gi...@apache.org>.
clintropolis commented on pull request #11185:
URL: https://github.com/apache/druid/pull/11185#issuecomment-842556588
thanks for review all :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #11185: fix bug with aggregator expressions on realtime index with string columns always producing 0 values
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #11185:
URL: https://github.com/apache/druid/pull/11185#discussion_r633747391
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprEval.java
##########
@@ -543,7 +543,18 @@ public String asString()
}
/**
- * returns true if numeric primitive value for this ExprEval is null, otherwise false.
+ * The method returns true if numeric primitive value for this {@link ExprEval} is null, otherwise false.
+ *
+ * If this method returns false, then the values returned by {@link #asLong()}, {@link #asDouble()},
+ * and {@link #asInt()} are "valid", since this method is primarily used during {@link Expr} evaluation to decide
+ * if primitive numbers can be fetched to use.
+ *
+ * If a type cannot sanely convert into a primitive numeric value, then this method should always return true so that
+ * these primitive numeric getters are not called, since returning false is assumed to mean these values are valid.
+ *
+ * Note that all types must still return values for {@link #asInt()}, {@link #asLong()}}, and {@link #asDouble()},
+ * since this can still happen if {@link NullHandling#sqlCompatible()} is false, but it should be assumed that this
+ * can only happen in that mode and 0s are typical and expected for values that would otherwise be null.
Review comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org