You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/08/25 08:47:47 UTC

[GitHub] [druid] clintropolis opened a new pull request, #12971: fix issue with SQL planner and null array constants

clintropolis opened a new pull request, #12971:
URL: https://github.com/apache/druid/pull/12971

   ### Description
   Fixes a bug in `DruidRexExecutor` when reducing constants, where null array constants result in a null pointer exception rather than creating a null literal. This issue was discovered with the `JSON_KEYS` function, but it could happen with expression which would produce a constant null array type.
   
   The added `JSON_KEYS` test would fail prior to the changes in this patch, the `JSON_PATHS` test is just for fun.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [x] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis merged pull request #12971: fix issue with SQL planner and null array constants

Posted by GitBox <gi...@apache.org>.
clintropolis merged PR #12971:
URL: https://github.com/apache/druid/pull/12971


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12971: fix issue with SQL planner and null array constants

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12971:
URL: https://github.com/apache/druid/pull/12971#discussion_r955528160


##########
sql/src/main/java/org/apache/druid/sql/calcite/planner/DruidRexExecutor.java:
##########
@@ -142,40 +142,42 @@ public void reduce(
           }
         } else if (sqlTypeName == SqlTypeName.ARRAY) {
           assert exprResult.isArray();
-          if (SqlTypeName.NUMERIC_TYPES.contains(constExp.getType().getComponentType().getSqlTypeName())) {
+          final Object[] array = exprResult.asArray();
+          if (array == null) {
+            literal = rexBuilder.makeNullLiteral(constExp.getType());
+          } else if (SqlTypeName.NUMERIC_TYPES.contains(constExp.getType().getComponentType().getSqlTypeName())) {
             if (exprResult.type().getElementType().is(ExprType.LONG)) {
-              List<BigDecimal> resultAsBigDecimalList = Arrays.stream(exprResult.asArray())
-                                                              .map(val -> {
-                                                                final Number longVal = (Number) val;
-                                                                if (longVal == null) {
-                                                                  return null;
-                                                                }
-                                                                return BigDecimal.valueOf(longVal.longValue());
-                                                              })
-                                                              .collect(Collectors.toList());
+              List<BigDecimal> resultAsBigDecimalList = new ArrayList<>(array.length);
+              for (Object val : array) {
+                final Number longVal = (Number) val;
+                if (longVal == null) {
+                  resultAsBigDecimalList.add(null);
+                } else {
+                  resultAsBigDecimalList.add(BigDecimal.valueOf(longVal.longValue()));
+                }
+              }
               literal = rexBuilder.makeLiteral(resultAsBigDecimalList, constExp.getType(), true);
             } else {
-              List<BigDecimal> resultAsBigDecimalList = Arrays.stream(exprResult.asArray()).map(
-                  val -> {
-                    final Number doubleVal = (Number) val;
-                    if (doubleVal == null) {
-                      return null;
-                    }
-                    if (Double.isNaN(doubleVal.doubleValue()) || Double.isInfinite(doubleVal.doubleValue())) {
-                      String expression = druidExpression.getExpression();
-                      throw new UnsupportedSQLQueryException(
-                          "'%s' contains an element that evaluates to '%s' which is not supported in SQL. You can either cast the element in the array to bigint or char or change the expression itself",
-                          expression,
-                          Double.toString(doubleVal.doubleValue())
-                      );
-                    }
-                    return BigDecimal.valueOf(doubleVal.doubleValue());
-                  }
-              ).collect(Collectors.toList());
+              List<BigDecimal> resultAsBigDecimalList = new ArrayList<>(array.length);
+              for (Object val : array) {
+                final Number doubleVal = (Number) val;
+                if (doubleVal == null) {
+                  resultAsBigDecimalList.add(null);
+                } else if (Double.isNaN(doubleVal.doubleValue()) || Double.isInfinite(doubleVal.doubleValue())) {
+                  String expression = druidExpression.getExpression();
+                  throw new UnsupportedSQLQueryException(
+                      "'%s' contains an element that evaluates to '%s' which is not supported in SQL. You can either cast the element in the array to bigint or char or change the expression itself",

Review Comment:
   I know you didn't change anything here, but you moved it so I get to comment 🙂
   
   "BIGINT or VARCHAR" would be better than "bigint or char".



##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/NestedDataOperatorConversions.java:
##########
@@ -207,7 +209,16 @@ public static class JsonKeysOperatorConversion implements SqlOperatorConversion
             )
         )
         .functionCategory(SqlFunctionCategory.USER_DEFINED_FUNCTION)
-        .returnTypeNullableArray(SqlTypeName.VARCHAR)
+        .returnTypeInference(

Review Comment:
   Hmm. If I read the code right, the difference is that `returnTypeNullableArray(SqlTypeName.VARCHAR)` yields `ARRAY<VARCHAR NULLABLE>` (nullable elements, non-nullable array), whereas, what you have here yields `ARRAY<VARCHAR NULLABLE> NULLABLE` (the array is itself nullable too).
   
   Suggestion: rename `returnTypeNullableArray` to `returnTypeArrayWithNullableElements` and then introduce `returnTypeNullableArrayWithNullableElements` that does the thing you have here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12971: fix issue with SQL planner and null array constants

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12971:
URL: https://github.com/apache/druid/pull/12971#discussion_r955524963


##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/NestedDataOperatorConversions.java:
##########
@@ -207,7 +209,16 @@ public static class JsonKeysOperatorConversion implements SqlOperatorConversion
             )
         )
         .functionCategory(SqlFunctionCategory.USER_DEFINED_FUNCTION)
-        .returnTypeNullableArray(SqlTypeName.VARCHAR)
+        .returnTypeInference(

Review Comment:
   `returnTypeNullableArray` behaves a bit different from this `returnTypeInference` because this is using `ReturnTypes.cascade` instead of `ReturnTypes.explicit`, and had some troubles with type mismatch nullable vs not until I made this change.
   
   It might make more sense to make `returnTypeNullableArray` always be cascade nullable instead of always nullable, but I didn't make that change for now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org