You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by "chucheng92 (via GitHub)" <gi...@apache.org> on 2023/12/12 08:15:22 UTC

[PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

chucheng92 opened a new pull request, #3579:
URL: https://github.com/apache/calcite/pull/3579

   https://issues.apache.org/jira/browse/CALCITE-6127


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1426188220


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   yes. make sense. I have updated the PR. In the additional commit, I have overridden the checkOperand methods. The logic of the overridden method is basically the same as the previous OperandTypeChecker, except that `SqlTypeUtil.isComparable` is replaced to `SqlTypeUtil.leastRestrictiveForComparison`. The first will reject row/struct type with NULL, the next will try to get `leastRestrictiveType`, the next is what we expect.
   
   If you have time, PTAL again. thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1427013786


##########
core/src/main/java/org/apache/calcite/sql/type/OperandTypes.java:
##########
@@ -1225,6 +1229,64 @@ private static class MapFromEntriesOperandTypeChecker
     }
   }
 
+  /**
+   * Operand type-checking strategy for a ARRAY function, it allows empty array.
+   *
+   * <p> The reason it overrides SameOperandTypeChecker#checkOperandTypesImpl is that it needs
+   * to handle the scenario where row/struct type and NULL exist simultaneously in array.
+   * This scenario need be supported, but will be rejected by the current checkOperandTypesImpl.

Review Comment:
   I think the last row of this comment should be removed. This would be appropriate in the PR, but not in a comment.



##########
core/src/main/java/org/apache/calcite/sql/type/OperandTypes.java:
##########
@@ -1225,6 +1229,64 @@ private static class MapFromEntriesOperandTypeChecker
     }
   }
 
+  /**
+   * Operand type-checking strategy for a ARRAY function, it allows empty array.
+   *
+   * <p> The reason it overrides SameOperandTypeChecker#checkOperandTypesImpl is that it needs
+   * to handle the scenario where row/struct type and NULL exist simultaneously in array.
+   * This scenario need be supported, but will be rejected by the current checkOperandTypesImpl.
+   */
+  private static class ArrayFunctionOperandTypeChecker
+      extends SameOperandTypeChecker {
+
+    ArrayFunctionOperandTypeChecker() {
+      // The args of array are non-fixed, so we set to -1 here. then operandCount
+      // can dynamically set according to the number of input args.
+      // details please see SameOperandTypeChecker#getOperandList.
+      super(-1);
+    }
+
+    @Override protected boolean checkOperandTypesImpl(
+        SqlOperatorBinding operatorBinding,
+        boolean throwOnFailure,
+        @Nullable SqlCallBinding callBinding) {
+      if (throwOnFailure && callBinding == null) {
+        throw new IllegalArgumentException(
+            "callBinding must be non-null in case throwOnFailure=true");
+      }
+      int nOperandsActual = nOperands;
+      if (nOperandsActual == -1) {
+        nOperandsActual = operatorBinding.getOperandCount();
+      }
+      RelDataType[] types = new RelDataType[nOperandsActual];
+      final List<Integer> operandList =
+          getOperandList(operatorBinding.getOperandCount());
+      for (int i : operandList) {
+        types[i] = operatorBinding.getOperandType(i);
+      }
+      int prev = -1;
+      for (int i : operandList) {
+        if (prev >= 0) {

Review Comment:
   Can't this be replaced with `i > 0`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424822417


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   thanks mihai. you are right. the typechecking has a bug.
   I have tested the calcite std array constructor and spark sql, it shows like below:
   
   ```
   0: jdbc:calcite:model=src/test/resources/mode> select array[row(1,2), null];
   +----------------+
   |     EXPR$0     |
   +----------------+
   | [{1, 2}, null] |
   +----------------+
   ```
   
   ```
   spark-sql (default)> select array((1,2), null);
   [{"col1":1,"col2":2},null]
   ```
   
   It is indeed supported to return this form. I will open another issue to tracking/fixing this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424822417


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   thanks mihai. you are right. the typechecking has a bug.
   I have tested the calcite std array constructor and spark sql, it shows like below:
   
   ```
   0: jdbc:calcite:model=src/test/resources/mode> select array[row(1,2), null];
   +----------------+
   |     EXPR$0     |
   +----------------+
   | [{1, 2}, null] |
   +----------------+
   ```
   
   ```
   spark-sql (default)> select array((1,2), null);
   [{"col1":1,"col2":2},null]
   ```
   
   It is indeed supported this form. 
   
   Considering this bug has always existed and is not the problem to be solved by this issue. I will open another issue to tracking/fixing this. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3579:
URL: https://github.com/apache/calcite/pull/3579#issuecomment-1851550069

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3579)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3579&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3579&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3579&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&types=CODE_SMELL)
   
   [![100.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/100-16px.png '100.0%')](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_coverage&view=list) [100.0% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1428365291


##########
core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java:
##########
@@ -1061,23 +1061,29 @@ static RelDataType deriveTypeSplit(SqlOperatorBinding operatorBinding,
   private static RelDataType arrayReturnType(SqlOperatorBinding opBinding) {
     final List<RelDataType> operandTypes = opBinding.collectOperandTypes();
 
-    // only numeric & character types check
+    // only numeric & character types check, this is a special spark array case
+    // the form like ARRAY(1, 2, '3') will return ["1", "2", "3"]
     boolean hasNumeric = false;
     boolean hasCharacter = false;
     boolean hasOthers = false;
     for (RelDataType type : operandTypes) {
       SqlTypeFamily family = type.getSqlTypeName().getFamily();
-      requireNonNull(family, "array element type family");
+      // some types such as Row, the family is null, fallback to normal inferred type logic
+      if (family == null) {
+        hasOthers = true;
+        break;
+      }
+      // skip it because we allow NULL literal
+      if (SqlTypeUtil.isNull(type)) {
+        continue;

Review Comment:
   Type families in Calcite are extremely confusing. I have filed a PR https://github.com/apache/calcite/pull/3411 trying to clarify some of their semantics, but that doesn't address all the issues. Unfortunately type families are used pervasively so there is no way to remove them. One thing that makes them confusing is that the SQL standard talks about type families, but the type families in Calcite do not align with the standard. For example, in Calcite DOUBLE has a type family of NUMERIC, whereas the standard says it should be APPROXIMATE_NUMERIC. You will also notice that there are two kinds of families, primary and secondary. Moreover, there are many types which do not have any family, as in this example. I find that code which deals directly with types is much clearer, as this example illustrates.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 merged PR #3579:
URL: https://github.com/apache/calcite/pull/3579


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1426188220


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   yes. I have updated the PR. In the additional commit, I have override the checkOperand methods. The logic of the overridden method is basically the same as the previous OperandTypeChecker, except that `SqlTypeUtil.isComparable` is replaced to `SqlTypeUtil.leastRestrictiveForComparison`. The first will reject row/struct type with NULL, the next will try to get `leastRestrictiveType`, the next is what we expect.
   
   If you have time, PTAL again. thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424796818


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   @mihaibudiu 
   My original intention is just to test the case where the array contains null, such as
   `array(1, null)`, `array('A', null)` ...
   they can get expected result (can deduce correct type). however `array(row(1), null)` will cause error, because the element type is not same.
   IMO, the end user may also use this form. Do you mean we don't need this `checkFail` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424822417


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   thanks mihai. you are right. the typechecking has a bug.
   I have tested the calcite std array constructor and spark sql, it shows like below:
   
   ```
   0: jdbc:calcite:model=src/test/resources/mode> select array[row(1,2), null];
   +----------------+
   |     EXPR$0     |
   +----------------+
   | [{1, 2}, null] |
   +----------------+
   ```
   
   ```
   spark-sql (default)> select array((1,2), null);
   [{"col1":1,"col2":2},null]
   ```
   
   It is indeed supported this form. I will open another issue to tracking/fixing this. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424808579


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   To be clearly:
   In fact, the error here is because of the effect of `type1.isStruct() != type2.isStruct()`, which works with `OperandTypeChecker`. I understand what you mean. You want `array(row(1, 2), null)` to return `RecordType(INTEGER EXPR $0, INTEGER EXPR$1) ARRAY NOT NULL`, row(1,2) & null will deduce leastRestrictive(or biggest) type row with nullable. In fact, it can indeed return this form in `derivedReturnType`. We only need to call derivedReturnType separately or comment out the line `type1.isStruct() != type2.isStruct()`.
   
   But for the `SameOperandTypeChecker` of array, it needs to throw this error during checkOperandTypes. I understand that our checkFail can completely check the error of `OperandTypeChecker`. what do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1427505058


##########
core/src/main/java/org/apache/calcite/sql/type/OperandTypes.java:
##########
@@ -1225,6 +1229,64 @@ private static class MapFromEntriesOperandTypeChecker
     }
   }
 
+  /**
+   * Operand type-checking strategy for a ARRAY function, it allows empty array.
+   *
+   * <p> The reason it overrides SameOperandTypeChecker#checkOperandTypesImpl is that it needs
+   * to handle the scenario where row/struct type and NULL exist simultaneously in array.
+   * This scenario need be supported, but will be rejected by the current checkOperandTypesImpl.

Review Comment:
   yes, removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424285138


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   Shouldn't this one pass with the type of elements  being a nullable row?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1426188220


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   yes. make sense. I have updated the PR. In the additional commit, I have overridden the checkOperand methods. The logic of the overridden method is basically the same as the previous OperandTypeChecker, except that `SqlTypeUtil.isComparable` is replaced to `SqlTypeUtil.leastRestrictiveForComparison`. The first will reject row/struct type with NULL, the next will try to get `leastRestrictiveType`, the logic of the next is what we expect.
   
   If you have time, PTAL again. thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1428792987


##########
core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java:
##########
@@ -1061,23 +1061,29 @@ static RelDataType deriveTypeSplit(SqlOperatorBinding operatorBinding,
   private static RelDataType arrayReturnType(SqlOperatorBinding opBinding) {
     final List<RelDataType> operandTypes = opBinding.collectOperandTypes();
 
-    // only numeric & character types check
+    // only numeric & character types check, this is a special spark array case
+    // the form like ARRAY(1, 2, '3') will return ["1", "2", "3"]
     boolean hasNumeric = false;
     boolean hasCharacter = false;
     boolean hasOthers = false;
     for (RelDataType type : operandTypes) {
       SqlTypeFamily family = type.getSqlTypeName().getFamily();
-      requireNonNull(family, "array element type family");
+      // some types such as Row, the family is null, fallback to normal inferred type logic
+      if (family == null) {
+        hasOthers = true;
+        break;
+      }
+      // skip it because we allow NULL literal
+      if (SqlTypeUtil.isNull(type)) {
+        continue;

Review Comment:
   hi, @JiajunBernoulli  sorry to ping you, any other comments? or could you give me an extra approve?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1427504943


##########
core/src/main/java/org/apache/calcite/sql/type/OperandTypes.java:
##########
@@ -1225,6 +1229,64 @@ private static class MapFromEntriesOperandTypeChecker
     }
   }
 
+  /**
+   * Operand type-checking strategy for a ARRAY function, it allows empty array.
+   *
+   * <p> The reason it overrides SameOperandTypeChecker#checkOperandTypesImpl is that it needs
+   * to handle the scenario where row/struct type and NULL exist simultaneously in array.
+   * This scenario need be supported, but will be rejected by the current checkOperandTypesImpl.
+   */
+  private static class ArrayFunctionOperandTypeChecker
+      extends SameOperandTypeChecker {
+
+    ArrayFunctionOperandTypeChecker() {
+      // The args of array are non-fixed, so we set to -1 here. then operandCount
+      // can dynamically set according to the number of input args.
+      // details please see SameOperandTypeChecker#getOperandList.
+      super(-1);
+    }
+
+    @Override protected boolean checkOperandTypesImpl(
+        SqlOperatorBinding operatorBinding,
+        boolean throwOnFailure,
+        @Nullable SqlCallBinding callBinding) {
+      if (throwOnFailure && callBinding == null) {
+        throw new IllegalArgumentException(
+            "callBinding must be non-null in case throwOnFailure=true");
+      }
+      int nOperandsActual = nOperands;
+      if (nOperandsActual == -1) {
+        nOperandsActual = operatorBinding.getOperandCount();
+      }
+      RelDataType[] types = new RelDataType[nOperandsActual];
+      final List<Integer> operandList =
+          getOperandList(operatorBinding.getOperandCount());
+      for (int i : operandList) {
+        types[i] = operatorBinding.getOperandType(i);
+      }
+      int prev = -1;
+      for (int i : operandList) {
+        if (prev >= 0) {

Review Comment:
   yes, we use `prev` to record previous index. but can also use `i` directly. updated the PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "JiajunBernoulli (via GitHub)" <gi...@apache.org>.
JiajunBernoulli commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424764366


##########
core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java:
##########
@@ -1061,23 +1061,29 @@ static RelDataType deriveTypeSplit(SqlOperatorBinding operatorBinding,
   private static RelDataType arrayReturnType(SqlOperatorBinding opBinding) {
     final List<RelDataType> operandTypes = opBinding.collectOperandTypes();
 
-    // only numeric & character types check
+    // only numeric & character types check, this is a special spark array case
+    // the form like ARRAY(1, 2, '3') will return ["1", "2", "3"]
     boolean hasNumeric = false;
     boolean hasCharacter = false;
     boolean hasOthers = false;
     for (RelDataType type : operandTypes) {
       SqlTypeFamily family = type.getSqlTypeName().getFamily();
-      requireNonNull(family, "array element type family");
+      // some types such as Row, the family is null, fallback to normal inferred type logic
+      if (family == null) {
+        hasOthers = true;
+        break;
+      }
+      // skip it because we allow NULL literal
+      if (SqlTypeUtil.isNull(type)) {
+        continue;

Review Comment:
   Why use `continue`?
   
   Old code in 1078~1080 line handle NULL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424822417


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   thanks mihai. you are right. the typechecking has a bug.
   I have tested the calcite std array constructor and spark sql, it shows like below:
   
   ```
   0: jdbc:calcite:model=src/test/resources/mode> select array[row(1,2), null];
   +----------------+
   |     EXPR$0     |
   +----------------+
   | [{1, 2}, null] |
   +----------------+
   ```
   
   ```
   spark-sql (default)> select array((1,2), null);
   [{"col1":1,"col2":2},null]
   ```
   
   It is indeed supported this form (the generated component type is nullable row/struct). 
   
   Considering this bug has always existed and is not the problem to be solved by this issue. I will open another issue to tracking/fixing this. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1426188220


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   yes. I have updated the PR. In the additional commit, I have overridden the checkOperand methods. The logic of the overridden method is basically the same as the previous OperandTypeChecker, except that `SqlTypeUtil.isComparable` is replaced to `SqlTypeUtil.leastRestrictiveForComparison`. The first will reject row/struct type with NULL, the next will try to get `leastRestrictiveType`, the next is what we expect.
   
   If you have time, PTAL again. thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424776526


##########
core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java:
##########
@@ -1061,23 +1061,29 @@ static RelDataType deriveTypeSplit(SqlOperatorBinding operatorBinding,
   private static RelDataType arrayReturnType(SqlOperatorBinding opBinding) {
     final List<RelDataType> operandTypes = opBinding.collectOperandTypes();
 
-    // only numeric & character types check
+    // only numeric & character types check, this is a special spark array case
+    // the form like ARRAY(1, 2, '3') will return ["1", "2", "3"]
     boolean hasNumeric = false;
     boolean hasCharacter = false;
     boolean hasOthers = false;
     for (RelDataType type : operandTypes) {
       SqlTypeFamily family = type.getSqlTypeName().getFamily();
-      requireNonNull(family, "array element type family");
+      // some types such as Row, the family is null, fallback to normal inferred type logic
+      if (family == null) {
+        hasOthers = true;
+        break;
+      }
+      // skip it because we allow NULL literal
+      if (SqlTypeUtil.isNull(type)) {
+        continue;

Review Comment:
   thanks for reviewing. the old code use `case NULL`, the NULL is SqlTypeFamily. however some other types such as UNKNOWN, the family is also NULL. here just use SqlTypeUtil#isNull to truly indicate NULL type. use continue will skip this null literal element.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424808579


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   To be clearly:
   In fact, the error here is because of the effect of `type1.isStruct() != type2.isStruct()`, which works with `OperandTypeChecker`. I understand what you mean. You want `array(row(1, 2), null)` to return `RecordType(INTEGER EXPR $0, INTEGER EXPR$1) ARRAY NOT NULL`, row(1,2) & null will deduce leastRestrictive(or biggest) type row with nullable. In fact, it can indeed return this form in `derivedReturnType`. We only need to call derivedReturnType separately or comment out the line `type1.isStruct() != type2.isStruct()`(just for testing).
   
   But for the `SameOperandTypeChecker` of array, it needs to throw this error during checkOperandTypes. I understand that our checkFail can check the error of `OperandTypeChecker`. what do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3579:
URL: https://github.com/apache/calcite/pull/3579#issuecomment-1855178220

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3579) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [3 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3579&resolved=false&inNewCodePeriod=true)  
   [90.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_coverage&view=list)  
   [20.6% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3579)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1426188220


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   yes. make sense. I have updated the PR. In the additional commit, I have overridden the checkOperand methods. The logic of the overridden method is basically the same as the previous OperandTypeChecker, except that `SqlTypeUtil.isComparable` is replaced to `SqlTypeUtil.leastRestrictiveForComparison`. The first will reject row/struct type with NULL, the next will try to get `leastRestrictiveType`, the logic of the next is what we expect, and it will not cause other side effects on the current existed function.
   
   If you have time, PTAL again. thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3579:
URL: https://github.com/apache/calcite/pull/3579#issuecomment-1855103589

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3579) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [3 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3579&resolved=false&inNewCodePeriod=true)  
   [90.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_coverage&view=list)  
   [20.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3579)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3579:
URL: https://github.com/apache/calcite/pull/3579#issuecomment-1857250056

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3579) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [3 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3579&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3579&resolved=false&inNewCodePeriod=true)  
   [90.5% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_coverage&view=list)  
   [17.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3579&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3579)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424813114


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   Yes, you are right, I think that the test should pass with the output you have given.
   This means that there is still one more bug left in the typechecking code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424776526


##########
core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java:
##########
@@ -1061,23 +1061,29 @@ static RelDataType deriveTypeSplit(SqlOperatorBinding operatorBinding,
   private static RelDataType arrayReturnType(SqlOperatorBinding opBinding) {
     final List<RelDataType> operandTypes = opBinding.collectOperandTypes();
 
-    // only numeric & character types check
+    // only numeric & character types check, this is a special spark array case
+    // the form like ARRAY(1, 2, '3') will return ["1", "2", "3"]
     boolean hasNumeric = false;
     boolean hasCharacter = false;
     boolean hasOthers = false;
     for (RelDataType type : operandTypes) {
       SqlTypeFamily family = type.getSqlTypeName().getFamily();
-      requireNonNull(family, "array element type family");
+      // some types such as Row, the family is null, fallback to normal inferred type logic
+      if (family == null) {
+        hasOthers = true;
+        break;
+      }
+      // skip it because we allow NULL literal
+      if (SqlTypeUtil.isNull(type)) {
+        continue;

Review Comment:
   thanks for reviewing. the old code use `case NULL`, the NULL is SqlTypeFamily. however some other types such as UNKNOWN, the family is also NULL. here just use SqlTypeUtil#isNull to truly indicate NULL type. use continue will skip this null literal element just like old logic in switch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1425664735


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   If the fix is not too complicated it should probably be part of this PR.
   Otherwise you have to check-in these tests which give the wrong result, and then modify them again. But perhaps it's reasonable to make a separate PR if it entails a lot of work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on PR #3579:
URL: https://github.com/apache/calcite/pull/3579#issuecomment-1859742624

   merging ...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on PR #3579:
URL: https://github.com/apache/calcite/pull/3579#issuecomment-1857260082

   hi, @mihaibudiu Sorry to ping you, all comments are resolved. If you have time, PTAL again. thanks~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424774439


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   @mihaibudiu thanks for reviewing. the array use `SameOperandTypeChecker`, will call
   https://github.com/apache/calcite/blob/56eb062ba2eae239e9fdda6891830cf2ec60b605/core/src/main/java/org/apache/calcite/sql/type/SqlTypeUtil.java#L1495
   
   when element is null and row will return false. then checkFail. 
   
   The reason is row type is a struct type. struct types require their fields are pairwise comparable in SqlTypeUtil. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424778822


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   With the rule you give it is impossible to create a constant array with type `RecordType(...) ARRAY` which contains a `NULL` value. In other words, I think that the example that fails should actually be legal code. Clearly the NULL type is not isStruct(), but they can be both unified by a type which is a nullable struct.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CALCITE-6127] The spark array function gives NullPointerException when element is row type [calcite]

Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3579:
URL: https://github.com/apache/calcite/pull/3579#discussion_r1424808579


##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -10537,6 +10537,29 @@ private static void checkArrayConcatAggFuncFails(SqlOperatorFixture t) {
         "[null, foo]", "CHAR(3) ARRAY NOT NULL");
     f2.checkScalar("array(null)",
         "[null]", "NULL ARRAY NOT NULL");
+    // check complex type
+    f2.checkScalar("array(row(1))", "[{1}]",
+        "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, null))", "[{1, null}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(null, 1))", "[{null, 1}]",
+        "RecordType(NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkScalar("array(row(1, 2))", "[{1, 2}]",
+        "RecordType(INTEGER NOT NULL EXPR$0, INTEGER NOT NULL EXPR$1) NOT NULL ARRAY NOT NULL");
+    f2.checkFails("^array(row(1, 2), null)^",
+        "Parameters must be of the same type", false);
+    f2.checkFails("^array(null, row(1, 2))^",

Review Comment:
   To be clearly:
   In fact, the error here is because of the effect of `type1.isStruct() != type2.isStruct()`, which works with `OperandTypeChecker`. I understand what you mean. You want `array(row(1, 2), null)` to return `RecordType(INTEGER EXPR $0, INTEGER EXPR$1) ARRAY NOT NULL`, row(1,2) & null will deduce leastRestrictive(or biggest) type row with nullable. In fact, it can indeed return this form in `derivedReturnType`. We only need to call derivedReturnType separately or comment out the line `type1.isStruct() != type2.isStruct()`(just for testing).
   
   But for the `SameOperandTypeChecker` of array, it needs to throw this error during checkOperandTypes. I understand that our checkFail can completely check the error of `OperandTypeChecker`. what do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org