You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "xuyangzhong (via GitHub)" <gi...@apache.org> on 2023/11/20 12:03:19 UTC

[PR] [FLINK-33490][table-planner] Validate the name conflicts when creating view [flink]

xuyangzhong opened a new pull request, #23760:
URL: https://github.com/apache/flink/pull/23760

   ## What is the purpose of the change
   
   Many databases also throw exception when view name conflicts, e.g. mysql, postgres. We should follow this behavior in Flink.
   
   
   ## Brief change log
   
     - *validate the duplicated name while creating view*
     - *add tests*
   
   
   ## Verifying this change
   
   The tests are added.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? no
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the name conflicts when creating view [flink]

Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on code in PR #23760:
URL: https://github.com/apache/flink/pull/23760#discussion_r1399405605


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlNodeConvertUtils.java:
##########
@@ -63,6 +69,32 @@ static CatalogView toCatalogView(
         // This bug is fixed in CALCITE-3877 of Calcite 1.23.0.
         String originalQuery = context.toQuotedSqlString(query);
         SqlNode validateQuery = context.getSqlValidator().validate(query);
+
+        // Check name is unique.
+        // Don't rely on the calcite because if the field names are duplicate, calcite will add
+        // index to identify the duplicate names.
+        SqlValidatorNamespace namespace = context.getSqlValidator().getNamespace(validateQuery);
+        Map<String, Integer> nameToPos = new HashMap<>();
+        for (int i = 0;
+                i < Objects.requireNonNull(namespace).getType().getFieldList().size();
+                i++) {
+            String columnName = namespace.getType().getFieldList().get(i).getName();

Review Comment:
   Does it mean to check the column name conflict? The PR description refer to view name conflict.



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlNodeConvertUtils.java:
##########
@@ -63,6 +69,32 @@ static CatalogView toCatalogView(
         // This bug is fixed in CALCITE-3877 of Calcite 1.23.0.
         String originalQuery = context.toQuotedSqlString(query);
         SqlNode validateQuery = context.getSqlValidator().validate(query);
+
+        // Check name is unique.

Review Comment:
   I would suggest refactoring the code into a private method for better readability, i.e. clean code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the name conflicts when creating view [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on code in PR #23760:
URL: https://github.com/apache/flink/pull/23760#discussion_r1401572450


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlNodeConvertUtils.java:
##########
@@ -63,6 +69,32 @@ static CatalogView toCatalogView(
         // This bug is fixed in CALCITE-3877 of Calcite 1.23.0.
         String originalQuery = context.toQuotedSqlString(query);
         SqlNode validateQuery = context.getSqlValidator().validate(query);
+
+        // Check name is unique.
+        // Don't rely on the calcite because if the field names are duplicate, calcite will add
+        // index to identify the duplicate names.
+        SqlValidatorNamespace namespace = context.getSqlValidator().getNamespace(validateQuery);
+        Map<String, Integer> nameToPos = new HashMap<>();
+        for (int i = 0;
+                i < Objects.requireNonNull(namespace).getType().getFieldList().size();
+                i++) {
+            String columnName = namespace.getType().getFieldList().get(i).getName();

Review Comment:
   Yes, I'll change the title of this pr to make it clearer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "fsk119 (via GitHub)" <gi...@apache.org>.
fsk119 merged PR #23760:
URL: https://github.com/apache/flink/pull/23760


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "fsk119 (via GitHub)" <gi...@apache.org>.
fsk119 commented on code in PR #23760:
URL: https://github.com/apache/flink/pull/23760#discussion_r1402925888


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlNodeConvertUtils.java:
##########
@@ -63,6 +69,32 @@ static CatalogView toCatalogView(
         // This bug is fixed in CALCITE-3877 of Calcite 1.23.0.
         String originalQuery = context.toQuotedSqlString(query);
         SqlNode validateQuery = context.getSqlValidator().validate(query);
+
+        // Check name is unique.
+        // Don't rely on the calcite because if the field names are duplicate, calcite will add
+        // index to identify the duplicate names.
+        SqlValidatorNamespace namespace = context.getSqlValidator().getNamespace(validateQuery);
+        Map<String, Integer> nameToPos = new HashMap<>();
+        for (int i = 0;
+                i < Objects.requireNonNull(namespace).getType().getFieldList().size();
+                i++) {
+            String columnName = namespace.getType().getFieldList().get(i).getName();
+            if (nameToPos.containsKey(columnName)) {
+                SqlSelect select = (SqlSelect) query;

Review Comment:
   It may be a SqlCall, e.g. Union or CTE. Please add more test cases here to verify the behaviour.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on PR #23760:
URL: https://github.com/apache/flink/pull/23760#issuecomment-1822203012

   Hi, @JingGe . Thanks for your review. I have refactored it. Can you do the final check?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on PR #23760:
URL: https://github.com/apache/flink/pull/23760#issuecomment-1839930471

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on code in PR #23760:
URL: https://github.com/apache/flink/pull/23760#discussion_r1402846477


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlDdlToOperationConverterTest.java:
##########
@@ -2210,6 +2211,32 @@ public void testAlterTableDropPartitions() throws Exception {
                         "ALTER TABLE cat1.db1.tb1 DROP IF EXISTS PARTITION (b=1, c=2) PARTITION (b=2)");
     }
 
+    @Test
+    public void testCreateViewWithDuplicateFieldName() {
+        Map<String, String> prop = new HashMap<>();
+        prop.put("connector", "values");
+        prop.put("bounded", "true");
+        CatalogTable catalogTable =
+                CatalogTable.of(
+                        Schema.newBuilder()
+                                .column("id", DataTypes.BIGINT().notNull())
+                                .column("uid", DataTypes.BIGINT().notNull())
+                                .build(),
+                        null,
+                        Collections.emptyList(),
+                        prop);
+
+        catalogManager.createTable(
+                catalogTable, ObjectIdentifier.of("builtin", "default", "id_table"), false);
+
+        assertThatThrownBy(

Review Comment:
   Does it make sense to add one happy path test to make sure everything is still fine if no column name conflict found?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the name conflicts when creating view [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23760:
URL: https://github.com/apache/flink/pull/23760#issuecomment-1818948928

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "c5a3d3af2c79f572f93456d285b229ae5ccc1cfb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c5a3d3af2c79f572f93456d285b229ae5ccc1cfb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c5a3d3af2c79f572f93456d285b229ae5ccc1cfb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on code in PR #23760:
URL: https://github.com/apache/flink/pull/23760#discussion_r1431313795


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlDdlToOperationConverterTest.java:
##########
@@ -2210,6 +2211,32 @@ public void testAlterTableDropPartitions() throws Exception {
                         "ALTER TABLE cat1.db1.tb1 DROP IF EXISTS PARTITION (b=1, c=2) PARTITION (b=2)");
     }
 
+    @Test
+    public void testCreateViewWithDuplicateFieldName() {
+        Map<String, String> prop = new HashMap<>();
+        prop.put("connector", "values");
+        prop.put("bounded", "true");
+        CatalogTable catalogTable =
+                CatalogTable.of(
+                        Schema.newBuilder()
+                                .column("id", DataTypes.BIGINT().notNull())
+                                .column("uid", DataTypes.BIGINT().notNull())
+                                .build(),
+                        null,
+                        Collections.emptyList(),
+                        prop);
+
+        catalogManager.createTable(
+                catalogTable, ObjectIdentifier.of("builtin", "default", "id_table"), false);
+
+        assertThatThrownBy(

Review Comment:
   I think the existent tests can already cover this happy path. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on PR #23760:
URL: https://github.com/apache/flink/pull/23760#issuecomment-1863752804

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on PR #23760:
URL: https://github.com/apache/flink/pull/23760#issuecomment-1838565826

   Hi, @fsk119, I update this pr. Can you do the final check?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org