You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/09/16 12:38:30 UTC
[GitHub] [flink] snuyanzin opened a new pull request, #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
snuyanzin opened a new pull request, #20850:
URL: https://github.com/apache/flink/pull/20850
## What is the purpose of the change
The PR is to update calcite dependency to 1.27
## Verifying this change
From one side it's partially covered by existing tests
From another side it would make sense to have multiple end-to-end user tests (probably still discussable )
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes )
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no )
- The serializers: ( no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972984794
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Filter.java:
##########
@@ -38,10 +38,13 @@
import org.apache.calcite.rex.RexShuttle;
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.util.Litmus;
+import org.checkerframework.checker.nullness.qual.Nullable;
import java.util.List;
import java.util.Objects;
+import static java.util.Objects.requireNonNull;
+
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Intersect.java:
##########
@@ -40,7 +40,7 @@
*/
public abstract class Intersect extends SetOp {
/** Creates an Intersect. */
- public Intersect(
+ protected Intersect(
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978511413
##########
flink-table/flink-sql-parser-hive/src/test/java/org/apache/flink/sql/parser/hive/FlinkHiveSqlParserImplTest.java:
##########
@@ -560,4 +560,36 @@ void testExplainUpsert() {
String expected = "EXPLAIN UPSERT INTO `EMPS1`\n" + "VALUES (ROW(1, 2))";
this.sql(sql).ok(expected);
}
+
+ /**
+ * Here we override the super method to avoid test error from `ARRAY_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testArrayAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `GROUP CONCAT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testGroupConcat() {}
+
+ /**
+ * Here we override the super method to avoid test error from `STRING_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testStringAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `EXPLAIN AS DOT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testExplainAsDot() {}
}
Review Comment:
https://issues.apache.org/jira/browse/CALCITE-4260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972986628
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -24,7 +24,6 @@
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;
import org.apache.calcite.linq4j.Ord;
-import org.apache.calcite.linq4j.function.Function2;
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java:
##########
@@ -5927,7 +5927,7 @@ public interface Config {
* Returns the factory to create {@link RelBuilder}, never null. Default is {@link
* RelFactories#LOGICAL_BUILDER}.
*/
- @ImmutableBeans.Property(required = true)
+ @ImmutableBeans.Property
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972990685
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/FlinkAggregateExpandDistinctAggregatesRule.java:
##########
@@ -331,6 +331,7 @@ private RelBuilder convertSingletonDistinct(
false,
aggCall.getArgList(),
-1,
+ aggCall.distinctKeys,
Review Comment:
Added distinctKeys https://issues.apache.org/jira/browse/CALCITE-4483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r999570972
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -193,6 +204,10 @@ under the License.
<groupId>org.apiguardian</groupId>
Review Comment:
added a comment in `flink-table/flink-table-planner/pom.xml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr closed pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr closed pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
URL: https://github.com/apache/flink/pull/20850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r991925372
##########
flink-table/pom.xml:
##########
@@ -77,6 +77,7 @@ under the License.
<properties>
<calcite.version>1.26.0</calcite.version>
+ <checker-qual.version>3.10.0</checker-qual.version>
<!-- Keep Janino in sync with calcite. -->
<janino.version>3.0.11</janino.version>
Review Comment:
No, within Calcite 1.27.0 it is still 3.0.11
It is changed in 1.28.0 within https://issues.apache.org/jira/browse/CALCITE-3745 to 3.1.6... (in fact it seems another pain point for future Calcite update e.g. 1.28.0) after that change https://github.com/janino-compiler/janino/pull/150/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993331987
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
yes that would be a reason to add the excelusion, better to triple check what maven contains in the `tree` and what is contained in the final jar files in the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r999569994
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -347,6 +367,7 @@ under the License.
<include>com.google.guava:failureaccess</include>
<include>commons-codec:commons-codec</include>
<include>commons-io:commons-io</include>
+ <include>org.checkerframework:checker-qual</include>
Review Comment:
you're right, i've double checked it and it's ok
added relocation as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972985822
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql/fun/SqlCastFunction.java:
##########
@@ -31,7 +31,6 @@
import org.apache.calcite.sql.SqlFunctionCategory;
import org.apache.calcite.sql.SqlIntervalQualifier;
import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlLiteral;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978587112
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/CalcTest.xml:
##########
@@ -262,7 +262,7 @@ LogicalProject(a=[$0], b=[$1], c=[$2])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[a, b, c], where=[(a IS NULL OR SEARCH(a, Sarg[1, 10]))])
+Calc(select=[a, b, c], where=[SEARCH(a, Sarg[1, 10; NULL AS TRUE])])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978592824
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/table/JoinTest.xml:
##########
@@ -84,15 +84,15 @@ Calc(select=[a, b, d, e])
<Resource name="ast">
<![CDATA[
LogicalProject(a=[$0], b=[$1], d=[$3], e=[$4])
-+- LogicalFilter(condition=[OR(AND(=($0, 0), =($3, 3)), SEARCH($0, Sarg[1]))])
++- LogicalFilter(condition=[OR(AND(=($0, 0), =($3, 3)), =($0, 1))])
+- LogicalJoin(condition=[true], joinType=[inner])
:- LogicalTableScan(table=[[default_catalog, default_database, T1]])
+- LogicalTableScan(table=[[default_catalog, default_database, T2]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Join(joinType=[InnerJoin], where=[(((a = 0) AND (d = 3)) OR SEARCH(a, Sarg[1]))], select=[a, b, d, e], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
+Join(joinType=[InnerJoin], where=[(((a = 0) AND (d = 3)) OR (a = 1))], select=[a, b, d, e], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978583021
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/CalcTest.xml:
##########
@@ -279,7 +279,7 @@ LogicalProject(a=[$0], b=[$1], c=[$2])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[a, b, c], where=[(a IS NULL OR SEARCH(a, Sarg[1, 10]))])
+Calc(select=[a, b, c], where=[SEARCH(a, Sarg[1, 10; NULL AS TRUE])])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978511255
##########
flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java:
##########
@@ -1950,6 +1950,38 @@ void testCreateTableAsSelectWithPartitionKey() {
"CREATE TABLE AS SELECT syntax does not support to create partitioned table yet."));
}
+ /**
+ * Here we override the super method to avoid test error from `ARRAY_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testArrayAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `GROUP CONCAT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testGroupConcat() {}
+
+ /**
+ * Here we override the super method to avoid test error from `STRING_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testStringAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `EXPLAIN AS DOT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testExplainAsDot() {}
Review Comment:
https://issues.apache.org/jira/browse/CALCITE-4260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r979468073
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/InputFormatCodeGenerator.scala:
##########
@@ -80,6 +80,7 @@ object InputFormatCodeGenerator {
@Override
public Object nextRecord(Object reuse) {
+ | ${ctx.reuseLocalVariableCode()}
Review Comment:
Probably as a result of https://issues.apache.org/jira/browse/CALCITE-4383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r994688394
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
just noted here about future rebase
Also after rebase to master some exclusions should be applied for flink-sql-client after this change https://issues.apache.org/jira/browse/FLINK-29486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992790858
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
🙈, you're right, thanks
removed it from here and flink-sql-parser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on PR #20850:
URL: https://github.com/apache/flink/pull/20850#issuecomment-1284170623
@flinkbot run azure
for some reason build has not started
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972990685
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/FlinkAggregateExpandDistinctAggregatesRule.java:
##########
@@ -331,6 +331,7 @@ private RelBuilder convertSingletonDistinct(
false,
aggCall.getArgList(),
-1,
+ aggCall.distinctKeys,
Review Comment:
Added distinctKeys https://issues.apache.org/jira/browse/CALCITE-4483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972986366
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql/fun/SqlDotOperator.java:
##########
@@ -34,6 +34,7 @@
import org.apache.calcite.sql.type.SqlSingleOperandTypeChecker;
import org.apache.calcite.sql.type.SqlTypeFamily;
import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.type.SqlTypeUtil;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978758053
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/table/SetOperatorsTest.xml:
##########
@@ -99,13 +99,13 @@ HashJoin(joinType=[LeftSemiJoin], where=[(c = a1)], select=[a, b, c], build=[rig
<TestCase name="testInWithProject">
<Resource name="ast">
<![CDATA[
-LogicalProject(b2=[AS(SEARCH($1, Sarg[1972-02-22 07:12:00.333:TIMESTAMP(3)]:TIMESTAMP(3)), _UTF-16LE'b2')])
+LogicalProject(b2=[AS(=($1, 1972-02-22 07:12:00.333), _UTF-16LE'b2')])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/table/SetOperatorsTest.xml:
##########
@@ -99,13 +99,13 @@ HashJoin(joinType=[LeftSemiJoin], where=[(c = a1)], select=[a, b, c], build=[rig
<TestCase name="testInWithProject">
<Resource name="ast">
<![CDATA[
-LogicalProject(b2=[AS(SEARCH($1, Sarg[1972-02-22 07:12:00.333:TIMESTAMP(3)]:TIMESTAMP(3)), _UTF-16LE'b2')])
+LogicalProject(b2=[AS(=($1, 1972-02-22 07:12:00.333), _UTF-16LE'b2')])
+- LogicalTableScan(table=[[default_catalog, default_database, A, source: [TestTableSource(a, b, c)]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[SEARCH(b, Sarg[1972-02-22 07:12:00.333]) AS b2])
+Calc(select=[(b = 1972-02-22 07:12:00.333) AS b2])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972989767
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/FlinkRexBuilder.java:
##########
@@ -102,10 +102,10 @@ public RexNode makeFieldAccess(RexNode expr, int i) {
* @return Simple literal, or cast simple literal
*/
@Override
- public RexNode makeZeroLiteral(RelDataType type) {
+ public RexLiteral makeZeroLiteral(RelDataType type) {
Review Comment:
As a result of https://github.com/apache/calcite/commit/56a86a032ac05ed522846910eea6f884b31820e3#diff-3be99bddc7edf13dc8198da7425d7e97c97237e3561c263fd74903b4a42d8cd9R401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972984498
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Correlate.java:
##########
@@ -34,12 +34,14 @@
import org.apache.calcite.sql.validate.SqlValidatorUtil;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.calcite.util.Litmus;
+import org.checkerframework.checker.nullness.qual.Nullable;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993950327
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
for `flink-table-planner` `mvn dependency:tree` gives
```
...
[INFO] +- org.apiguardian:apiguardian-api:jar:1.1.2:compile
...
```
compile is expected because of `org.apache.calcite.sql.validate.SqlValidatorImpl` where there are some apiguardian's annotations
for `flink-sql-parser` `mvn dependenc:tree` gives
```
...
[INFO] +- org.junit.vintage:junit-vintage-engine:jar:5.8.1:test
[INFO] | +- org.junit.platform:junit-platform-engine:jar:1.8.1:test
[INFO] | +- junit:junit:jar:4.13.2:test
[INFO] | \- org.apiguardian:apiguardian-api:jar:1.1.2:test
...
```
fpr `flink-sql-parser-hive` `mvn dependency tree` gives
```
...
[INFO] +- org.junit.vintage:junit-vintage-engine:jar:5.8.1:test
[INFO] | +- org.junit.platform:junit-platform-engine:jar:1.8.1:test
[INFO] | +- junit:junit:jar:4.13.2:test
[INFO] | \- org.apiguardian:apiguardian-api:jar:1.1.2:test
...
```
And neither jars from these 3 contains apiguardian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978587716
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/CalcTest.xml:
##########
@@ -344,13 +344,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
+- LogicalTableScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[IF(((c = '') OR c IS NULL), 'a', 'b') AS EXPR$0])
+Calc(select=[IF(SEARCH(c, Sarg[_UTF-16LE''; NULL AS TRUE]), 'a', 'b') AS EXPR$0])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992173879
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java:
##########
@@ -590,15 +611,15 @@ public RelRoot convertQuery(SqlNode query, final boolean needsValidation, final
SqlExplainLevel.EXPPLAN_ATTRIBUTES));
}
- final RelDataType validatedRowType = validator.getValidatedNodeType(query);
+ final RelDataType validatedRowType = validator().getValidatedNodeType(query);
List<RelHint> hints = new ArrayList<>();
if (query.getKind() == SqlKind.SELECT) {
final SqlSelect select = (SqlSelect) query;
if (select.hasHints()) {
hints = SqlUtil.getRelHint(hintStrategies, select.getHints());
}
}
-
+ // propagate the hints.
result = RelOptUtil.propagateRelHints(result, false);
// ----- FLINK MODIFICATION BEGIN -----
Review Comment:
Add affected lines and corresponding issues to the JavaDoc of this class.
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/utils/FlinkRexUtilTest.scala:
##########
@@ -484,9 +482,7 @@ class FlinkRexUtilTest {
rexBuilder.makeCall(GREATER_THAN, c, intLiteral(0)),
predicate19Layer1)
val newPredicate19 = simplify(rexBuilder, predicate19)
- assertEquals(
- rexBuilder.makeIn(c, Collections.singletonList[RexNode](intLiteral(1))).toString,
- newPredicate19.toString)
+ assertEquals(rexBuilder.makeCall(EQUALS, c, intLiteral(1)).toString, newPredicate19.toString)
// c >= 0 OR SEARCH(c, [0, 1])
// TODO `c >= 0 OR SEARCH(c, [0, 1])` should be simplified to c >= 0
Review Comment:
update comments
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/FlinkAggregateExpandDistinctAggregatesRule.java:
##########
@@ -484,6 +488,7 @@ private void rewriteUsingGroupingSets(RelOptRuleCall call, Aggregate aggregate)
false,
ImmutableIntList.copyOf(fullGroupSet),
-1,
+ ImmutableBitSet.of(),
Review Comment:
You can also use `null` here. Or is there a difference?
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -58,24 +57,19 @@
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
-import java.util.Objects;
import java.util.Set;
+import static java.util.Objects.requireNonNull;
+import static org.apache.calcite.linq4j.Nullness.castNonNull;
import static org.apache.calcite.rex.RexUnknownAs.FALSE;
import static org.apache.calcite.rex.RexUnknownAs.TRUE;
import static org.apache.calcite.rex.RexUnknownAs.UNKNOWN;
/**
* Context required to simplify a row-expression.
*
- * <p>Copied to fix Calcite 1.26 bugs, should be removed for the next Calcite upgrade.
- *
- * <p>Changes (line numbers are from the original RexSimplify file):
- *
- * <ol>
- * <li>CALCITE-4364 & FLINK-19811: Line 1307, Line 1764, Line 2638 ~ Line 2656.
- * <li>CALCITE-4446 & FLINK-22015: Line 2542 ~ Line 2548, Line 2614 ~ Line 2619.
- * </ol>
+ * <p>Currently, is a workaround for <a href="https://issues.apache.org/jira/browse/FLINK-29237">
Review Comment:
I see a lot of changes in this class. I guess most of them are due to the copy of Calcite code over to Flink. Could you add comments here which lines are still different in Flink? Similar as the old comment.
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -127,22 +127,22 @@ under the License.
<version>${calcite.version}</version>
<exclusions>
<!--
- "mvn dependency:tree" as of Calcite 1.26.0:
-
- [INFO] +- org.apache.calcite:calcite-core:jar:1.26.0:compile
- [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.26.0:compile
- [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
- [INFO] | +- org.apiguardian:apiguardian-api:jar:1.1.0:compile
- [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:runtime
+ "mvn dependency:tree" as of Calcite 1.27.0:
+
+ [INFO] +- org.apache.calcite:calcite-core:jar:1.27.0:compile
+ [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.27.0:compile
+ [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:compile
+ [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.2:compile
+ [INFO] | +- org.apache.calcite.avatica:avatica-core:jar:1.18.0:compile
Review Comment:
why was `avatica-core` not required before? Did we just miss it in the previous list?
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/catalog/JavaCatalogTableTest.xml:
##########
@@ -76,12 +75,11 @@ LogicalProject(EXPR$0=[$0])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT($f3) AS EXPR$0])
+GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT(*) AS EXPR$0])
Review Comment:
We need to verify whether compiled JSON plans still work for this change. The legacy window syntax is not supported in compiled JSON plans anyway.
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql2rel/RelDecorrelator.java:
##########
@@ -278,20 +283,42 @@ protected RelNode decorrelate(RelNode root) {
FilterCorrelateRule.Config.DEFAULT
.withRelBuilderFactory(f)
.toRule())
+ /* ----- FLINK MODIFICATION BEGIN -----
Review Comment:
Can we agree on always adding the line numbers and related issue also to the class JavaDoc? It is very tedious to search in classes.
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/join/NestedLoopSemiAntiJoinTest.xml:
##########
@@ -673,8 +673,8 @@ Calc(select=[b])
: +- HashAggregate(isMerge=[true], groupBy=[EXPR$0], select=[EXPR$0])
: +- Exchange(distribution=[hash[EXPR$0]])
: +- LocalHashAggregate(groupBy=[EXPR$0], select=[EXPR$0])
- : +- Calc(select=[1 AS EXPR$0])(reuse_id=[1])
- : +- LegacyTableSourceScan(table=[[default_catalog, default_database, t, source: [TestTableSource(i, j, k)]]], fields=[i, j, k])
+ : +- Calc(select=[1 AS EXPR$0])
Review Comment:
This might need investigation. It looks like some quality checks don't work anymore? Otherwise the planner should detect similar nodes.
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/rules/logical/subquery/SubQueryAntiJoinTest.xml:
##########
@@ -1833,8 +1833,8 @@ LogicalProject(b=[$1])
: +- LogicalJoin(condition=[=($0, $5)], joinType=[left])
: :- LogicalJoin(condition=[true], joinType=[inner])
: : :- LogicalTableScan(table=[[default_catalog, default_database, l, source: [TestTableSource(a, b, c)]]])
- : : +- LogicalAggregate(group=[{}], c=[COUNT()], ck=[COUNT($0)])
- : : +- LogicalProject(EXPR$0=[1])
+ : : +- LogicalProject(c=[$0], ck=[$0])
Review Comment:
Is this also tested somewhere with IT cases? We should make sure that ExecNodes are still working properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993052648
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -127,22 +127,22 @@ under the License.
<version>${calcite.version}</version>
<exclusions>
<!--
- "mvn dependency:tree" as of Calcite 1.26.0:
-
- [INFO] +- org.apache.calcite:calcite-core:jar:1.26.0:compile
- [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.26.0:compile
- [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
- [INFO] | +- org.apiguardian:apiguardian-api:jar:1.1.0:compile
- [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:runtime
+ "mvn dependency:tree" as of Calcite 1.27.0:
+
+ [INFO] +- org.apache.calcite:calcite-core:jar:1.27.0:compile
+ [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.27.0:compile
+ [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:compile
+ [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.2:compile
+ [INFO] | +- org.apache.calcite.avatica:avatica-core:jar:1.18.0:compile
Review Comment:
Ok sounds like a mistake in the previous version upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r991888500
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
Just to double check: Do we really need an exclusion if the dep is already set to provided?
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -193,6 +204,10 @@ under the License.
<groupId>org.apiguardian</groupId>
Review Comment:
Is this exclusion still valid because it seems Calcite uses apiguardian in version 1.1.2 now?
##########
flink-table/pom.xml:
##########
@@ -77,6 +77,7 @@ under the License.
<properties>
<calcite.version>1.26.0</calcite.version>
+ <checker-qual.version>3.10.0</checker-qual.version>
<!-- Keep Janino in sync with calcite. -->
<janino.version>3.0.11</janino.version>
Review Comment:
Has the Janino version changed?
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -161,6 +168,10 @@ under the License.
<groupId>org.apiguardian</groupId>
<artifactId>apiguardian-api</artifactId>
</exclusion>
+ <exclusion>
Review Comment:
Update the `"mvn dependency:tree" as of Calcite 1.26.0:` above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992562130
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -127,22 +127,22 @@ under the License.
<version>${calcite.version}</version>
<exclusions>
<!--
- "mvn dependency:tree" as of Calcite 1.26.0:
-
- [INFO] +- org.apache.calcite:calcite-core:jar:1.26.0:compile
- [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.26.0:compile
- [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
- [INFO] | +- org.apiguardian:apiguardian-api:jar:1.1.0:compile
- [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:runtime
+ "mvn dependency:tree" as of Calcite 1.27.0:
+
+ [INFO] +- org.apache.calcite:calcite-core:jar:1.27.0:compile
+ [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.27.0:compile
+ [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:compile
+ [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.2:compile
+ [INFO] | +- org.apache.calcite.avatica:avatica-core:jar:1.18.0:compile
Review Comment:
may be...
I've just checked output of `mvn dependency:tree` for flink-table-planner for calcite and it shows (avatica-core is included)
```
[INFO] +- org.apache.calcite:calcite-core:jar:1.26.0:compile
[INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.26.0:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.4:compile
[INFO] | +- org.apache.calcite.avatica:avatica-core:jar:1.17.0:compile
[INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:runtime
[INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.13.4:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.13.4:compile
[INFO] | +- com.jayway.jsonpath:json-path:jar:2.4.0:runtime
[INFO] | | \- net.minidev:json-smart:jar:2.3:runtime
[INFO] | | \- net.minidev:accessors-smart:jar:1.2:runtime
[INFO] | | \- org.ow2.asm:asm:jar:5.0.4:runtime
[INFO] | +- commons-codec:commons-codec:jar:1.15:runtime
[INFO] | \- commons-io:commons-io:jar:2.11.0:compile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r999264790
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -347,6 +367,7 @@ under the License.
<include>com.google.guava:failureaccess</include>
<include>commons-codec:commons-codec</include>
<include>commons-io:commons-io</include>
+ <include>org.checkerframework:checker-qual</include>
Review Comment:
I fear it will not work after relocation because of this line at `core/src/main/java/org/apache/calcite/util/ImmutableBeans.java` (line numbers 144-147)
```java
final boolean required = propertyType.isPrimitive()
|| !hasAnnotation(
method.getAnnotatedReturnType(),
"org.checkerframework.checker.nullness.qual.Nullable");
```
i will also double check it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r999264790
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -347,6 +367,7 @@ under the License.
<include>com.google.guava:failureaccess</include>
<include>commons-codec:commons-codec</include>
<include>commons-io:commons-io</include>
+ <include>org.checkerframework:checker-qual</include>
Review Comment:
I fear it will not work after relocation because of this line at `core/src/main/java/org/apache/calcite/util/ImmutableBeans.java`
```java
final boolean required = propertyType.isPrimitive()
|| !hasAnnotation(
method.getAnnotatedReturnType(),
"org.checkerframework.checker.nullness.qual.Nullable");
```
i will also double check it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r979468073
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/InputFormatCodeGenerator.scala:
##########
@@ -80,6 +80,7 @@ object InputFormatCodeGenerator {
@Override
public Object nextRecord(Object reuse) {
+ | ${ctx.reuseLocalVariableCode()}
Review Comment:
Probably as a result of https://issues.apache.org/jira/browse/CALCITE-4383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978511255
##########
flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java:
##########
@@ -1950,6 +1950,38 @@ void testCreateTableAsSelectWithPartitionKey() {
"CREATE TABLE AS SELECT syntax does not support to create partitioned table yet."));
}
+ /**
+ * Here we override the super method to avoid test error from `ARRAY_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testArrayAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `GROUP CONCAT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testGroupConcat() {}
+
+ /**
+ * Here we override the super method to avoid test error from `STRING_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testStringAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `EXPLAIN AS DOT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testExplainAsDot() {}
Review Comment:
Explain as dot was introduced in Calcite at https://issues.apache.org/jira/browse/CALCITE-4260
##########
flink-table/flink-sql-parser-hive/src/test/java/org/apache/flink/sql/parser/hive/FlinkHiveSqlParserImplTest.java:
##########
@@ -560,4 +560,36 @@ void testExplainUpsert() {
String expected = "EXPLAIN UPSERT INTO `EMPS1`\n" + "VALUES (ROW(1, 2))";
this.sql(sql).ok(expected);
}
+
+ /**
+ * Here we override the super method to avoid test error from `ARRAY_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testArrayAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `GROUP CONCAT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testGroupConcat() {}
+
+ /**
+ * Here we override the super method to avoid test error from `STRING_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testStringAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `EXPLAIN AS DOT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testExplainAsDot() {}
}
Review Comment:
Explain as dot was introduced in Calcite at https://issues.apache.org/jira/browse/CALCITE-4260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r994327282
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/join/NestedLoopSemiAntiJoinTest.xml:
##########
@@ -673,8 +673,8 @@ Calc(select=[b])
: +- HashAggregate(isMerge=[true], groupBy=[EXPR$0], select=[EXPR$0])
: +- Exchange(distribution=[hash[EXPR$0]])
: +- LocalHashAggregate(groupBy=[EXPR$0], select=[EXPR$0])
- : +- Calc(select=[1 AS EXPR$0])(reuse_id=[1])
- : +- LegacyTableSourceScan(table=[[default_catalog, default_database, t, source: [TestTableSource(i, j, k)]]], fields=[i, j, k])
+ : +- Calc(select=[1 AS EXPR$0])
Review Comment:
The same reason as mentioned in comment above:
added optimization in Calcite to remove args for nonDistinct `SqlAggCountFunction` and successors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972995310
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/SubQueryDecorrelator.java:
##########
@@ -451,7 +452,7 @@ public Frame decorrelateRel(LogicalProject rel) {
// Project projects the original expressions,
// plus any correlated variables the input wants to pass along.
- final List<Pair<RexNode, String>> projects = new ArrayList<>();
+ final List<Pair<RexNode, ? extends @Nullable String>> projects = new ArrayList<>();
Review Comment:
Changed at https://issues.apache.org/jira/browse/CALCITE-4199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r973030091
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ValuesCodeGenerator.scala:
##########
@@ -43,9 +44,17 @@ object ValuesCodeGenerator {
val generatedRecords = tuples.map {
r =>
exprGenerator.generateResultExpression(
- r.map(exprGenerator.generateExpression),
+ r.zipWithIndex.map {
Review Comment:
Probably as a result of https://issues.apache.org/jira/browse/CALCITE-4383
it requires same type now... To achieve there is a cast to least restrictive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993071814
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/catalog/JavaCatalogTableTest.xml:
##########
@@ -76,12 +75,11 @@ LogicalProject(EXPR$0=[$0])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT($f3) AS EXPR$0])
+GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT(*) AS EXPR$0])
Review Comment:
here probably I would love to get some input how I can check it or where to read how to check it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992266138
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/join/NestedLoopSemiAntiJoinTest.xml:
##########
@@ -673,8 +673,8 @@ Calc(select=[b])
: +- HashAggregate(isMerge=[true], groupBy=[EXPR$0], select=[EXPR$0])
: +- Exchange(distribution=[hash[EXPR$0]])
: +- LocalHashAggregate(groupBy=[EXPR$0], select=[EXPR$0])
- : +- Calc(select=[1 AS EXPR$0])(reuse_id=[1])
- : +- LegacyTableSourceScan(table=[[default_catalog, default_database, t, source: [TestTableSource(i, j, k)]]], fields=[i, j, k])
+ : +- Calc(select=[1 AS EXPR$0])
Review Comment:
This might need investigation. It looks like some equality checks don't work anymore? Otherwise the planner should detect similar nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993941510
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/catalog/JavaCatalogTableTest.xml:
##########
@@ -76,12 +75,11 @@ LogicalProject(EXPR$0=[$0])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT($f3) AS EXPR$0])
+GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT(*) AS EXPR$0])
Review Comment:
After debugging I found a reason for that. It is invoked by the change within [1] . To be more precise by optimization within these lines in
_org.apache.calcite.tools.RelBuilder_ (line number 3402 - 3406)
```java
if (aggFunction instanceof SqlCountAggFunction && !distinct) {
args = args.stream()
.filter(r::fieldIsNullable)
.collect(Util.toImmutableList());
}
```
I did a double check for that: copied a class to Flink repo and commented only this line and then no need to change rules (this, two below where you also commented, and some more, full changeset [2]), ci for that is also green except one non related Hive issue [3]
[1] https://github.com/apache/calcite/commit/a680abe0528faacf8d284419c76c36bbc1a65aa3
[2] https://github.com/snuyanzin/flink/commit/481543210509f682ac89ef3ce42d2f324937bbc4
[3] https://dev.azure.com/snuyanzin/flink/_build/results?buildId=1450&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993950327
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
`mvn dependency:tree` gives
```
...
[INFO] +- com.google.guava:guava:jar:29.0-jre:compile
[INFO] | +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] | \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
...
[INFO] +- org.checkerframework:checker-qual:jar:3.10.0:compile
...
```
and in jar there is e.g. present this class `org.checkerframework.framework.qual.PurityUnqualified` which is a result of changes in 3.10.0 according to changelog https://github.com/typetools/checker-framework/blob/master/docs/CHANGELOG.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972984498
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Correlate.java:
##########
@@ -34,12 +34,14 @@
import org.apache.calcite.sql.validate.SqlValidatorUtil;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.calcite.util.Litmus;
+import org.checkerframework.checker.nullness.qual.Nullable;
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Intersect.java:
##########
@@ -40,7 +40,7 @@
*/
public abstract class Intersect extends SetOp {
/** Creates an Intersect. */
- public Intersect(
+ protected Intersect(
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Filter.java:
##########
@@ -38,10 +38,13 @@
import org.apache.calcite.rex.RexShuttle;
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.util.Litmus;
+import org.checkerframework.checker.nullness.qual.Nullable;
import java.util.List;
import java.util.Objects;
+import static java.util.Objects.requireNonNull;
+
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993330634
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/catalog/JavaCatalogTableTest.xml:
##########
@@ -76,12 +75,11 @@ LogicalProject(EXPR$0=[$0])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT($f3) AS EXPR$0])
+GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT(*) AS EXPR$0])
Review Comment:
I don't fully understand what has changed. Somehow fields are not part of agg call anymore? we need to check if this is a pure logical node change or whether this also affects the structure of exec nodes. exec nodes need a stable JSON ser/de logic. I will try to help with this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r973030091
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ValuesCodeGenerator.scala:
##########
@@ -43,9 +44,17 @@ object ValuesCodeGenerator {
val generatedRecords = tuples.map {
r =>
exprGenerator.generateResultExpression(
- r.map(exprGenerator.generateExpression),
+ r.zipWithIndex.map {
Review Comment:
Probably as a result of https://issues.apache.org/jira/browse/CALCITE-4383
it requires same type now... To achieve there is a cast to least restrictive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992291970
##########
flink-table/pom.xml:
##########
@@ -77,6 +77,7 @@ under the License.
<properties>
<calcite.version>1.26.0</calcite.version>
+ <checker-qual.version>3.10.0</checker-qual.version>
<!-- Keep Janino in sync with calcite. -->
<janino.version>3.0.11</janino.version>
Review Comment:
Good to know. Step by step ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on PR #20850:
URL: https://github.com/apache/flink/pull/20850#issuecomment-1277125751
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #20850:
URL: https://github.com/apache/flink/pull/20850#issuecomment-1249318051
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "317a15da60181a853ca8bec7974071d29c4970a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "317a15da60181a853ca8bec7974071d29c4970a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 317a15da60181a853ca8bec7974071d29c4970a3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r999569412
##########
flink-table/pom.xml:
##########
@@ -76,7 +76,8 @@ under the License.
</dependencyManagement>
<properties>
- <calcite.version>1.26.0</calcite.version>
+ <calcite.version>1.27.0</calcite.version>
+ <checker-qual.version>3.10.0</checker-qual.version>
Review Comment:
yes, makes sense, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972995680
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/calcite/FlinkLogicalRelFactories.scala:
##########
@@ -79,11 +79,11 @@ object FlinkLogicalRelFactories {
/** Implementation of [[ProjectFactory]] that returns a [[FlinkLogicalCalc]]. */
class ProjectFactoryImpl extends ProjectFactory {
- def createProject(
+ override def createProject(
input: RelNode,
hints: util.List[RelHint],
childExprs: util.List[_ <: RexNode],
- fieldNames: util.List[String]): RelNode = {
+ fieldNames: util.List[_ <: String]): RelNode = {
Review Comment:
Changed at https://issues.apache.org/jira/browse/CALCITE-4199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r973032086
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/schema/FlinkTableFunction.scala:
##########
@@ -37,14 +37,13 @@ import java.util.Collections
*/
abstract class FlinkTableFunction(tableFunction: functions.TableFunction[_]) extends TableFunction {
- override def getElementType(arguments: util.List[AnyRef]): Type = classOf[Array[Object]]
+ override def getElementType(arguments: util.List[_]): Type = classOf[Array[Object]]
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972985031
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Minus.java:
##########
@@ -44,7 +44,7 @@
*/
public abstract class Minus extends SetOp {
- public Minus(
+ protected Minus(
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Sort.java:
##########
@@ -33,9 +33,11 @@
import org.apache.calcite.rel.hint.Hintable;
import org.apache.calcite.rel.hint.RelHint;
import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexLiteral;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.rex.RexShuttle;
import org.apache.calcite.util.Util;
+import org.checkerframework.checker.nullness.qual.Nullable;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978592501
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/table/JoinTest.xml:
##########
@@ -84,15 +84,15 @@ Calc(select=[a, b, d, e])
<Resource name="ast">
<![CDATA[
LogicalProject(a=[$0], b=[$1], d=[$3], e=[$4])
-+- LogicalFilter(condition=[OR(AND(=($0, 0), =($3, 3)), SEARCH($0, Sarg[1]))])
++- LogicalFilter(condition=[OR(AND(=($0, 0), =($3, 3)), =($0, 1))])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972985031
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Minus.java:
##########
@@ -44,7 +44,7 @@
*/
public abstract class Minus extends SetOp {
- public Minus(
+ protected Minus(
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Sort.java:
##########
@@ -33,9 +33,11 @@
import org.apache.calcite.rel.hint.Hintable;
import org.apache.calcite.rel.hint.RelHint;
import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexLiteral;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.rex.RexShuttle;
import org.apache.calcite.util.Util;
+import org.checkerframework.checker.nullness.qual.Nullable;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972985282
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Values.java:
##########
@@ -36,6 +36,7 @@
import org.apache.calcite.sql.SqlExplainLevel;
import org.apache.calcite.sql.type.SqlTypeUtil;
import org.apache.calcite.util.Pair;
+import org.checkerframework.checker.nullness.qual.Nullable;
import java.util.Collections;
import java.util.List;
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Window.java:
##########
@@ -46,6 +46,7 @@
import org.apache.calcite.util.ImmutableIntList;
import org.apache.calcite.util.Litmus;
import org.apache.calcite.util.Util;
+import org.checkerframework.checker.nullness.qual.Nullable;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993204783
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/FlinkAggregateExpandDistinctAggregatesRule.java:
##########
@@ -484,6 +488,7 @@ private void rewriteUsingGroupingSets(RelOptRuleCall call, Aggregate aggregate)
false,
ImmutableIntList.copyOf(fullGroupSet),
-1,
+ ImmutableBitSet.of(),
Review Comment:
it seems like a bit gray area...
However I did a test with Calcite tests like what is the diff if I put `null` or `ImmutableBitSet.of()` and some tests about parsing/dialect started to fail because of generation `WITHIN ()`. So looks like it's better to have null, also noticed that in javadoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993051603
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -193,6 +204,10 @@ under the License.
<groupId>org.apiguardian</groupId>
Review Comment:
It would say it make sense. The more helpful comments in pom files, the better. They get messy quite quickly which is why we constantly need to try to keep them readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r999571540
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
applied this change since rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978583509
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/CalcTest.xml:
##########
@@ -361,13 +361,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/CalcTest.xml:
##########
@@ -361,13 +361,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
+- LogicalTableScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[IF(((c = '') OR c IS NULL), 'a', 'b') AS EXPR$0])
+Calc(select=[IF(SEARCH(c, Sarg[_UTF-16LE''; NULL AS TRUE]), 'a', 'b') AS EXPR$0])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978587582
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/CalcTest.xml:
##########
@@ -344,13 +344,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/CalcTest.xml:
##########
@@ -344,13 +344,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
+- LogicalTableScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[IF(((c = '') OR c IS NULL), 'a', 'b') AS EXPR$0])
+Calc(select=[IF(SEARCH(c, Sarg[_UTF-16LE''; NULL AS TRUE]), 'a', 'b') AS EXPR$0])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992793159
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -193,6 +204,10 @@ under the License.
<groupId>org.apiguardian</groupId>
Review Comment:
I've just made it available for tests here
In fact 1.1.2 is used because of junit-jupiter 5.8.2 since this commit https://github.com/apache/flink/pull/17728
There is already a related comment above
`<!-- Excluding org.apiguardian:apiguardian-api v1.1.0 because org.junit.jupiter:junit-jupiter contains newer version v1.1.2 -->`
Or does it make sense to add something similar to `flink-table/flink-table-planner/pom.xml`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978587582
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/CalcTest.xml:
##########
@@ -344,13 +344,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978583509
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/CalcTest.xml:
##########
@@ -361,13 +361,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/CalcTest.xml:
##########
@@ -279,7 +279,7 @@ LogicalProject(a=[$0], b=[$1], c=[$2])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[a, b, c], where=[(a IS NULL OR SEARCH(a, Sarg[1, 10]))])
+Calc(select=[a, b, c], where=[SEARCH(a, Sarg[1, 10; NULL AS TRUE])])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972982268
##########
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/validate/FlinkSqlConformance.java:
##########
@@ -176,4 +178,9 @@ public boolean allowPluralTimeUnits() {
public boolean allowQualifyingCommonColumn() {
return true;
}
+
+ @Override
+ public SqlLibrary semantics() {
Review Comment:
Added at https://issues.apache.org/jira/browse/CALCITE-4427
##########
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/planner/delegation/hive/HiveParserCalcitePlanner.java:
##########
@@ -1208,6 +1208,7 @@ private RelNode genGBRelNode(
false,
gbKeyIndices,
-1,
+ null,
Review Comment:
Added distinctKeys https://issues.apache.org/jira/browse/CALCITE-4483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r994327282
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/join/NestedLoopSemiAntiJoinTest.xml:
##########
@@ -673,8 +673,8 @@ Calc(select=[b])
: +- HashAggregate(isMerge=[true], groupBy=[EXPR$0], select=[EXPR$0])
: +- Exchange(distribution=[hash[EXPR$0]])
: +- LocalHashAggregate(groupBy=[EXPR$0], select=[EXPR$0])
- : +- Calc(select=[1 AS EXPR$0])(reuse_id=[1])
- : +- LegacyTableSourceScan(table=[[default_catalog, default_database, t, source: [TestTableSource(i, j, k)]]], fields=[i, j, k])
+ : +- Calc(select=[1 AS EXPR$0])
Review Comment:
The same reason as mentioned in comment above:
added optimization in Calcite to filter nullable field args for nonDistinct `SqlAggCountFunction` and successors
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/rules/logical/subquery/SubQueryAntiJoinTest.xml:
##########
@@ -1833,8 +1833,8 @@ LogicalProject(b=[$1])
: +- LogicalJoin(condition=[=($0, $5)], joinType=[left])
: :- LogicalJoin(condition=[true], joinType=[inner])
: : :- LogicalTableScan(table=[[default_catalog, default_database, l, source: [TestTableSource(a, b, c)]]])
- : : +- LogicalAggregate(group=[{}], c=[COUNT()], ck=[COUNT($0)])
- : : +- LogicalProject(EXPR$0=[1])
+ : : +- LogicalProject(c=[$0], ck=[$0])
Review Comment:
The same reason as mentioned in comment above:
added optimization in Calcite to filter nullable field args for nonDistinct `SqlAggCountFunction` and successors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972982268
##########
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/validate/FlinkSqlConformance.java:
##########
@@ -176,4 +178,9 @@ public boolean allowPluralTimeUnits() {
public boolean allowQualifyingCommonColumn() {
return true;
}
+
+ @Override
+ public SqlLibrary semantics() {
Review Comment:
Added at https://issues.apache.org/jira/browse/CALCITE-4427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972986628
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -24,7 +24,6 @@
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;
import org.apache.calcite.linq4j.Ord;
-import org.apache.calcite.linq4j.function.Function2;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972987190
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java:
##########
@@ -5927,7 +5927,7 @@ public interface Config {
* Returns the factory to create {@link RelBuilder}, never null. Default is {@link
* RelFactories#LOGICAL_BUILDER}.
*/
- @ImmutableBeans.Property(required = true)
+ @ImmutableBeans.Property
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r991880868
##########
flink-table/flink-sql-parser-hive/src/test/java/org/apache/flink/sql/parser/hive/FlinkHiveSqlParserImplTest.java:
##########
@@ -46,6 +46,38 @@ void testTableHintsInInsert() {}
@Test
void testDescribeSchema() {}
+ /**
+ * Here we override the super method to avoid test error from `ARRAY_AGG` supported in original
Review Comment:
nit: simplify the comments to only an inline comment `// ARRAY_AGG` the explanation in the inline comment above should clarify the rest to avoid code duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992793159
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -193,6 +204,10 @@ under the License.
<groupId>org.apiguardian</groupId>
Review Comment:
I've just made it available for tests here
In fact 1.1.2 is used because of junit-jupiter 5.8.2 since this commit https://github.com/apache/flink/pull/17728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r991916758
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -193,6 +204,10 @@ under the License.
<groupId>org.apiguardian</groupId>
Review Comment:
1.1.2 appeared with Calcite 1.31.0 at https://issues.apache.org/jira/browse/CALCITE-5196
since this PR is about update only to 1.27.0 then it is still valid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992790858
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
~~🙈, you're right, thanks~~
~~removed it from here and flink-sql-parser~~
UPDATE: or wait, probably I remember the reason: ci fails during validation of NOTICE files [1]
That was the reason I put exclusions here
[1] https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41903&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb&l=4587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978587112
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/CalcTest.xml:
##########
@@ -262,7 +262,7 @@ LogicalProject(a=[$0], b=[$1], c=[$2])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[a, b, c], where=[(a IS NULL OR SEARCH(a, Sarg[1, 10]))])
+Calc(select=[a, b, c], where=[SEARCH(a, Sarg[1, 10; NULL AS TRUE])])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978511413
##########
flink-table/flink-sql-parser-hive/src/test/java/org/apache/flink/sql/parser/hive/FlinkHiveSqlParserImplTest.java:
##########
@@ -560,4 +560,36 @@ void testExplainUpsert() {
String expected = "EXPLAIN UPSERT INTO `EMPS1`\n" + "VALUES (ROW(1, 2))";
this.sql(sql).ok(expected);
}
+
+ /**
+ * Here we override the super method to avoid test error from `ARRAY_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testArrayAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `GROUP CONCAT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testGroupConcat() {}
+
+ /**
+ * Here we override the super method to avoid test error from `STRING_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testStringAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `EXPLAIN AS DOT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testExplainAsDot() {}
}
Review Comment:
Explain as dot was introduced in Calcite at https://issues.apache.org/jira/browse/CALCITE-4260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978511255
##########
flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java:
##########
@@ -1950,6 +1950,38 @@ void testCreateTableAsSelectWithPartitionKey() {
"CREATE TABLE AS SELECT syntax does not support to create partitioned table yet."));
}
+ /**
+ * Here we override the super method to avoid test error from `ARRAY_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testArrayAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `GROUP CONCAT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testGroupConcat() {}
+
+ /**
+ * Here we override the super method to avoid test error from `STRING_AGG` supported in original
+ * calcite.
+ */
+ @Disabled
+ @Test
+ void testStringAgg() {}
+
+ /**
+ * Here we override the super method to avoid test error from `EXPLAIN AS DOT` supported in
+ * original calcite.
+ */
+ @Disabled
+ @Test
+ void testExplainAsDot() {}
Review Comment:
Explain as dot was introduced in Calcite at https://issues.apache.org/jira/browse/CALCITE-4260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978763440
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/join/JoinTest.xml:
##########
@@ -72,17 +72,17 @@ Join(joinType=[InnerJoin], where=[(((a = 1) AND (x = 1)) OR ((a = 2) AND y IS NU
<Resource name="ast">
<![CDATA[
LogicalProject(a1=[$0], b1=[$3])
-+- LogicalJoin(condition=[OR(AND(=($0, 1), =($3, 1)), SEARCH($1, Sarg[2L:BIGINT]:BIGINT))], joinType=[inner])
++- LogicalJoin(condition=[OR(AND(=($0, 1), =($3, 1)), =($1, 2))], joinType=[inner])
:- LogicalTableScan(table=[[default_catalog, default_database, A, source: [TestTableSource(a1, a2, a3)]]])
+- LogicalTableScan(table=[[default_catalog, default_database, B, source: [TestTableSource(b1, b2, b3)]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
Calc(select=[a1, b1])
-+- Join(joinType=[InnerJoin], where=[(((a1 = 1) AND (b1 = 1)) OR SEARCH(a2, Sarg[2]))], select=[a1, a2, b1], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
++- Join(joinType=[InnerJoin], where=[(((a1 = 1) AND (b1 = 1)) OR (a2 = 2))], select=[a1, a2, b1], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/join/JoinTest.xml:
##########
@@ -72,17 +72,17 @@ Join(joinType=[InnerJoin], where=[(((a = 1) AND (x = 1)) OR ((a = 2) AND y IS NU
<Resource name="ast">
<![CDATA[
LogicalProject(a1=[$0], b1=[$3])
-+- LogicalJoin(condition=[OR(AND(=($0, 1), =($3, 1)), SEARCH($1, Sarg[2L:BIGINT]:BIGINT))], joinType=[inner])
++- LogicalJoin(condition=[OR(AND(=($0, 1), =($3, 1)), =($1, 2))], joinType=[inner])
:- LogicalTableScan(table=[[default_catalog, default_database, A, source: [TestTableSource(a1, a2, a3)]]])
+- LogicalTableScan(table=[[default_catalog, default_database, B, source: [TestTableSource(b1, b2, b3)]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
Calc(select=[a1, b1])
-+- Join(joinType=[InnerJoin], where=[(((a1 = 1) AND (b1 = 1)) OR SEARCH(a2, Sarg[2]))], select=[a1, a2, b1], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
++- Join(joinType=[InnerJoin], where=[(((a1 = 1) AND (b1 = 1)) OR (a2 = 2))], select=[a1, a2, b1], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
:- Exchange(distribution=[single])
- : +- Calc(select=[a1, a2], where=[((a1 = 1) OR SEARCH(a2, Sarg[2]))])
+ : +- Calc(select=[a1, a2], where=[((a1 = 1) OR (a2 = 2))])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on PR #20850:
URL: https://github.com/apache/flink/pull/20850#issuecomment-1284138735
Thanks for your feedback
rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r994327542
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/rules/logical/subquery/SubQueryAntiJoinTest.xml:
##########
@@ -1833,8 +1833,8 @@ LogicalProject(b=[$1])
: +- LogicalJoin(condition=[=($0, $5)], joinType=[left])
: :- LogicalJoin(condition=[true], joinType=[inner])
: : :- LogicalTableScan(table=[[default_catalog, default_database, l, source: [TestTableSource(a, b, c)]]])
- : : +- LogicalAggregate(group=[{}], c=[COUNT()], ck=[COUNT($0)])
- : : +- LogicalProject(EXPR$0=[1])
+ : : +- LogicalProject(c=[$0], ck=[$0])
Review Comment:
The same reason as mentioned in comment above:
added optimization in Calcite to remove args for nonDistinct SqlAggCountFunction and successors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r998991245
##########
flink-table/pom.xml:
##########
@@ -76,7 +76,8 @@ under the License.
</dependencyManagement>
<properties>
- <calcite.version>1.26.0</calcite.version>
+ <calcite.version>1.27.0</calcite.version>
+ <checker-qual.version>3.10.0</checker-qual.version>
Review Comment:
Remove this? It is only used at one location so we can directly define the version there?
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -347,6 +367,7 @@ under the License.
<include>com.google.guava:failureaccess</include>
<include>commons-codec:commons-codec</include>
<include>commons-io:commons-io</include>
+ <include>org.checkerframework:checker-qual</include>
Review Comment:
Shall we relocate this dependency as well in `<relocation>`?
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/catalog/JavaCatalogTableTest.xml:
##########
@@ -76,12 +75,11 @@ LogicalProject(EXPR$0=[$0])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT($f3) AS EXPR$0])
+GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT(*) AS EXPR$0])
Review Comment:
Thanks for the investigation. Given that this seems to be just a logical optimization, the change should be ok. I will perform a last investigation today before I give my +1 to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r999569994
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -347,6 +367,7 @@ under the License.
<include>com.google.guava:failureaccess</include>
<include>commons-codec:commons-codec</include>
<include>commons-io:commons-io</include>
+ <include>org.checkerframework:checker-qual</include>
Review Comment:
you're right add relocation as well
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -347,6 +367,7 @@ under the License.
<include>com.google.guava:failureaccess</include>
<include>commons-codec:commons-codec</include>
<include>commons-io:commons-io</include>
+ <include>org.checkerframework:checker-qual</include>
Review Comment:
you're right added relocation as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
twalthr commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992294313
##########
flink-table/flink-sql-parser/pom.xml:
##########
@@ -193,6 +204,10 @@ under the License.
<groupId>org.apiguardian</groupId>
Review Comment:
But in `flink-table/flink-table-planner/pom.xml` you set it to 1.1.2 explicitly, maybe we should add some comment in this file for the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992562130
##########
flink-table/flink-table-planner/pom.xml:
##########
@@ -127,22 +127,22 @@ under the License.
<version>${calcite.version}</version>
<exclusions>
<!--
- "mvn dependency:tree" as of Calcite 1.26.0:
-
- [INFO] +- org.apache.calcite:calcite-core:jar:1.26.0:compile
- [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.26.0:compile
- [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile
- [INFO] | +- org.apiguardian:apiguardian-api:jar:1.1.0:compile
- [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:runtime
- [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:runtime
+ "mvn dependency:tree" as of Calcite 1.27.0:
+
+ [INFO] +- org.apache.calcite:calcite-core:jar:1.27.0:compile
+ [INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.27.0:compile
+ [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:compile
+ [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.2:compile
+ [INFO] | +- org.apache.calcite.avatica:avatica-core:jar:1.18.0:compile
Review Comment:
may be...
I've just checked for `master` output of `mvn dependency:tree` for flink-table-planner for calcite and it shows (avatica-core is included)
```
[INFO] +- org.apache.calcite:calcite-core:jar:1.26.0:compile
[INFO] | +- org.apache.calcite:calcite-linq4j:jar:1.26.0:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.4:compile
[INFO] | +- org.apache.calcite.avatica:avatica-core:jar:1.17.0:compile
[INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:runtime
[INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.13.4:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.13.4:compile
[INFO] | +- com.jayway.jsonpath:json-path:jar:2.4.0:runtime
[INFO] | | \- net.minidev:json-smart:jar:2.3:runtime
[INFO] | | \- net.minidev:accessors-smart:jar:1.2:runtime
[INFO] | | \- org.ow2.asm:asm:jar:5.0.4:runtime
[INFO] | +- commons-codec:commons-codec:jar:1.15:runtime
[INFO] | \- commons-io:commons-io:jar:2.11.0:compile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r992790858
##########
flink-table/flink-sql-parser-hive/pom.xml:
##########
@@ -56,6 +56,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>provided</scope>
+ <exclusions>
Review Comment:
~~🙈, you're right, thanks~~
~~removed it from here and flink-sql-parser~~
or wait, probably I remember the reason: ci fails during validation of NOTICE files [1]
That was the reason I put exclusions here
[1] https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41903&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb&l=4587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r991925372
##########
flink-table/pom.xml:
##########
@@ -77,6 +77,7 @@ under the License.
<properties>
<calcite.version>1.26.0</calcite.version>
+ <checker-qual.version>3.10.0</checker-qual.version>
<!-- Keep Janino in sync with calcite. -->
<janino.version>3.0.11</janino.version>
Review Comment:
No, within Calcite 1.27.0 it is still 3.0.11
It is changed in 1.28.0 within https://issues.apache.org/jira/browse/CALCITE-3745 to 3.1.6... (in fact it seems another pain point for future Calcite update e.g. 1.28.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r978583648
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/CalcTest.xml:
##########
@@ -361,13 +361,13 @@ Calc(select=[a, (b + 1) AS EXPR$1], where=[(b > 2)])
</Resource>
<Resource name="ast">
<![CDATA[
-LogicalProject(EXPR$0=[IF(OR(=($2, _UTF-16LE''), IS NULL($2)), _UTF-16LE'a', _UTF-16LE'b')])
+LogicalProject(EXPR$0=[IF(SEARCH($2, Sarg[_UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"; NULL AS TRUE]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), _UTF-16LE'a', _UTF-16LE'b')])
+- LogicalTableScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-Calc(select=[IF(((c = '') OR c IS NULL), 'a', 'b') AS EXPR$0])
+Calc(select=[IF(SEARCH(c, Sarg[_UTF-16LE''; NULL AS TRUE]), 'a', 'b') AS EXPR$0])
Review Comment:
As a result of https://issues.apache.org/jira/browse/CALCITE-4325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972985822
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql/fun/SqlCastFunction.java:
##########
@@ -31,7 +31,6 @@
import org.apache.calcite.sql.SqlFunctionCategory;
import org.apache.calcite.sql.SqlIntervalQualifier;
import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlLiteral;
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql/fun/SqlDotOperator.java:
##########
@@ -34,6 +34,7 @@
import org.apache.calcite.sql.type.SqlSingleOperandTypeChecker;
import org.apache.calcite.sql.type.SqlTypeFamily;
import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.type.SqlTypeUtil;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972981423
##########
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/planner/delegation/hive/HiveParserCalcitePlanner.java:
##########
@@ -1208,6 +1208,7 @@ private RelNode genGBRelNode(
false,
gbKeyIndices,
-1,
+ null,
Review Comment:
Added distinctKeys https://issues.apache.org/jira/browse/CALCITE-4483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r972985282
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Values.java:
##########
@@ -36,6 +36,7 @@
import org.apache.calcite.sql.SqlExplainLevel;
import org.apache.calcite.sql.type.SqlTypeUtil;
import org.apache.calcite.util.Pair;
+import org.checkerframework.checker.nullness.qual.Nullable;
import java.util.Collections;
import java.util.List;
Review Comment:
Changes in this file are from calcite 1.27.0
##########
flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/core/Window.java:
##########
@@ -46,6 +46,7 @@
import org.apache.calcite.util.ImmutableIntList;
import org.apache.calcite.util.Litmus;
import org.apache.calcite.util.Util;
+import org.checkerframework.checker.nullness.qual.Nullable;
Review Comment:
Changes in this file are from calcite 1.27.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20850: [FLINK-20873][Table SQl/API] Update to calcite 1.27
Posted by GitBox <gi...@apache.org>.
snuyanzin commented on code in PR #20850:
URL: https://github.com/apache/flink/pull/20850#discussion_r993941510
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/catalog/JavaCatalogTableTest.xml:
##########
@@ -76,12 +75,11 @@ LogicalProject(EXPR$0=[$0])
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
-GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT($f3) AS EXPR$0])
+GroupWindowAggregate(window=[TumblingGroupWindow('w, proctime, 600000)], select=[COUNT(*) AS EXPR$0])
Review Comment:
After debugging I found a reason for that. It is invoked by the change within [1] . To be more precise by optimization within these lines in
_org.apache.calcite.tools.RelBuilder_ (line number 3402 - 3406)
```java
if (aggFunction instanceof SqlCountAggFunction && !distinct) {
args = args.stream()
.filter(r::fieldIsNullable)
.collect(Util.toImmutableList());
}
```
I did a double check for that: copied a class to Flink repo and commented only this line and then no need to change rules (this, two below where you also commented, and some more, full changeset [2]), ci for that is also green except one non related Hive issue [3]
[1] https://github.com/apache/calcite/commit/a680abe0528faacf8d284419c76c36bbc1a65aa3
[2] https://github.com/snuyanzin/flink/commit/481543210509f682ac89ef3ce42d2f324937bbc4
[3] https://dev.azure.com/snuyanzin/flink/_build/results?buildId=1448&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org