You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/10/09 20:07:03 UTC
[GitHub] [druid] clintropolis opened a new pull request #10499: support for vectorizing expressions with non-existent inputs, more consistent type handling for non-vectorized expressions
clintropolis opened a new pull request #10499:
URL: https://github.com/apache/druid/pull/10499
### Description
This PR adds support for vectorizing expressions in cases where inputs are missing, using either null or default values as inputs, depending on the value of `druid.generic.useDefaultValueForNull`.
This PR also makes non-vectorized expression type handling a bit more consistent across different types of expressions. Major changes here include operator expressions will now try to preserve the type when one of the arguments is null instead of always producing double values, and math functions now follow logic similar to the operators.
Tagging PR as release notes/incompatible because the changes cause some expressions to output slightly different results (typically longs instead of doubles). Examples:
`longColumn + nonExistentColumn -> longColumn + 0L` instead of `(double) longColumn + 0.0`
and math functions will produce output from non-existent inputs in default mode instead of always producing zeros:
`max(longColumn, nonExistentColumn) -> max(longColumn, 0L)`
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #10499: support for vectorizing expressions with non-existent inputs, more consistent type handling for non-vectorized expressions
Posted by GitBox <gi...@apache.org>.
jon-wei commented on a change in pull request #10499:
URL: https://github.com/apache/druid/pull/10499#discussion_r510514679
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprTypeConversion.java
##########
@@ -31,22 +31,40 @@
* Infer the output type of a list of possible 'conditional' expression outputs (where any of these could be the
* output expression if the corresponding case matching expression evaluates to true)
*/
- static ExprType conditional(Expr.InputBindingTypes inputTypes, List<Expr> args)
+ static ExprType conditional(Expr.InputBindingInspector inspector, List<Expr> args)
{
ExprType type = null;
for (Expr arg : args) {
if (arg.isNullLiteral()) {
continue;
}
if (type == null) {
- type = arg.getOutputType(inputTypes);
+ type = arg.getOutputType(inspector);
} else {
- type = doubleMathFunction(type, arg.getOutputType(inputTypes));
+ type = function(type, arg.getOutputType(inspector));
}
}
return type;
}
+ /**
+ * Given 2 'input' types, which might not be fully trustable, choose the most appropriate combined type for
+ * non-vectorized, per-row type detection. In this mode, null values are {@link ExprType#STRING} typed, despite
+ * potentially coming from an underlying numeric column. This method is not well suited for array handling
+ */
+ public static ExprType autoDetect(ExprEval result, ExprEval other)
Review comment:
Can you add a part to the javadoc about when the input types would not be trustable (is it because of the string nulls from numeric columns, or are there other cases)?
##########
File path: processing/src/main/java/org/apache/druid/segment/virtual/ExpressionPlanner.java
##########
@@ -135,18 +137,29 @@ public static ExpressionPlan plan(ColumnInspector inspector, Expr expression)
traits.add(ExpressionPlan.Trait.NON_SCALAR_INPUTS);
}
- if (!maybeMultiValued.isEmpty()) {
+ if (!noCapabilities.isEmpty()) {
traits.add(ExpressionPlan.Trait.UNKNOWN_INPUTS);
}
+ if (!maybeMultiValued.isEmpty()) {
+ traits.add(ExpressionPlan.Trait.INCOMPLETE_INPUTS);
+ }
+
// if expression needs transformed, lets do it
if (!needsApplied.isEmpty()) {
traits.add(ExpressionPlan.Trait.NEEDS_APPLIED);
}
}
- // only set output type
- if (ExpressionPlan.none(traits, ExpressionPlan.Trait.UNKNOWN_INPUTS, ExpressionPlan.Trait.NEEDS_APPLIED)) {
+ // only set output type if we are pretty confident about input types
+ final boolean shoulComputeOutput = ExpressionPlan.none(
Review comment:
shoulComputeOutput -> shouldComputeOutput
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprTypeConversion.java
##########
@@ -74,6 +93,15 @@ public static ExprType operator(@Nullable ExprType type, @Nullable ExprType othe
return ExprType.STRING;
}
+ // non-vectorized expressions
+ if (type == ExprType.STRING) {
Review comment:
Can you update the javadoc for this method with the mixed type case and the reasoning behind preferring the non-string type?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #10499: support for vectorizing expressions with non-existent inputs, more consistent type handling for non-vectorized expressions
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #10499:
URL: https://github.com/apache/druid/pull/10499#discussion_r510585207
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprTypeConversion.java
##########
@@ -74,6 +93,15 @@ public static ExprType operator(@Nullable ExprType type, @Nullable ExprType othe
return ExprType.STRING;
}
+ // non-vectorized expressions
+ if (type == ExprType.STRING) {
Review comment:
Oh, this check isn't actually necessary anymore, it was from an intermediary state this branch was in prior to adding the `autoDetect` function when I was instead trying to use this in the `eval` of operator expressions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #10499: support for vectorizing expressions with non-existent inputs, more consistent type handling for non-vectorized expressions
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10499:
URL: https://github.com/apache/druid/pull/10499#discussion_r508194339
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprTypeConversion.java
##########
@@ -31,22 +31,40 @@
* Infer the output type of a list of possible 'conditional' expression outputs (where any of these could be the
* output expression if the corresponding case matching expression evaluates to true)
*/
- static ExprType conditional(Expr.InputBindingTypes inputTypes, List<Expr> args)
+ static ExprType conditional(Expr.InputBindingInspector inspector, List<Expr> args)
{
ExprType type = null;
for (Expr arg : args) {
if (arg.isNullLiteral()) {
continue;
}
if (type == null) {
- type = arg.getOutputType(inputTypes);
+ type = arg.getOutputType(inspector);
} else {
- type = doubleMathFunction(type, arg.getOutputType(inputTypes));
+ type = function(type, arg.getOutputType(inspector));
}
}
return type;
}
+ /**
+ * Given 2 'input' types, which might not be fully trustable, choose the most appropriate combined type for
+ * non-vectorized, per-row type detection. In this mode, null values are {@link ExprType#STRING} typed, despite
+ * potentially coming from an underlying numeric column. This method is not well suited for array handling
+ */
+ public static ExprType autoDetect(ExprEval result, ExprEval other)
Review comment:
What do `result` and `other` mean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #10499: support for vectorizing expressions with non-existent inputs, more consistent type handling for non-vectorized expressions
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #10499:
URL: https://github.com/apache/druid/pull/10499#discussion_r510585299
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprTypeConversion.java
##########
@@ -31,22 +31,40 @@
* Infer the output type of a list of possible 'conditional' expression outputs (where any of these could be the
* output expression if the corresponding case matching expression evaluates to true)
*/
- static ExprType conditional(Expr.InputBindingTypes inputTypes, List<Expr> args)
+ static ExprType conditional(Expr.InputBindingInspector inspector, List<Expr> args)
{
ExprType type = null;
for (Expr arg : args) {
if (arg.isNullLiteral()) {
continue;
}
if (type == null) {
- type = arg.getOutputType(inputTypes);
+ type = arg.getOutputType(inspector);
} else {
- type = doubleMathFunction(type, arg.getOutputType(inputTypes));
+ type = function(type, arg.getOutputType(inspector));
}
}
return type;
}
+ /**
+ * Given 2 'input' types, which might not be fully trustable, choose the most appropriate combined type for
+ * non-vectorized, per-row type detection. In this mode, null values are {@link ExprType#STRING} typed, despite
+ * potentially coming from an underlying numeric column. This method is not well suited for array handling
+ */
+ public static ExprType autoDetect(ExprEval result, ExprEval other)
Review comment:
>Can you add a part to the javadoc about when the input types would not be trustable (is it because of the string nulls from numeric columns, or are there other cases)?
I have this blurb:
> In this mode, null values are {@link ExprType#STRING} typed, despite potentially coming from an underlying numeric column
but I have added the missing column case too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #10499: support for vectorizing expressions with non-existent inputs, more consistent type handling for non-vectorized expressions
Posted by GitBox <gi...@apache.org>.
clintropolis merged pull request #10499:
URL: https://github.com/apache/druid/pull/10499
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #10499: support for vectorizing expressions with non-existent inputs, more consistent type handling for non-vectorized expressions
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #10499:
URL: https://github.com/apache/druid/pull/10499#discussion_r508800090
##########
File path: core/src/main/java/org/apache/druid/math/expr/ExprTypeConversion.java
##########
@@ -31,22 +31,40 @@
* Infer the output type of a list of possible 'conditional' expression outputs (where any of these could be the
* output expression if the corresponding case matching expression evaluates to true)
*/
- static ExprType conditional(Expr.InputBindingTypes inputTypes, List<Expr> args)
+ static ExprType conditional(Expr.InputBindingInspector inspector, List<Expr> args)
{
ExprType type = null;
for (Expr arg : args) {
if (arg.isNullLiteral()) {
continue;
}
if (type == null) {
- type = arg.getOutputType(inputTypes);
+ type = arg.getOutputType(inspector);
} else {
- type = doubleMathFunction(type, arg.getOutputType(inputTypes));
+ type = function(type, arg.getOutputType(inspector));
}
}
return type;
}
+ /**
+ * Given 2 'input' types, which might not be fully trustable, choose the most appropriate combined type for
+ * non-vectorized, per-row type detection. In this mode, null values are {@link ExprType#STRING} typed, despite
+ * potentially coming from an underlying numeric column. This method is not well suited for array handling
+ */
+ public static ExprType autoDetect(ExprEval result, ExprEval other)
Review comment:
ah those are not very good variable names, `eval` and `otherEval` probably would've been better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org