You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "spena (via GitHub)" <gi...@apache.org> on 2024/08/23 18:36:40 UTC

[PR] [FLINK-35709][Table SQL / API] Allow defining a schema in REPLACE TABLE AS (RTAS) [flink]

spena opened a new pull request, #25247:
URL: https://github.com/apache/flink/pull/25247

   ## What is the purpose of the change
   
   Allow defining new columns and/or overriding source column types in the CREATE clause on RTAS statements.
   Syntax supported:
   
   ```
   CREATE OR REPLACE TABLE  table_name 
     [( <column_definition>[, ...n], [<watermark_definition>], [<table_constraint>])]
   [DISTRIBUTED BY [HASH|RANGE|RANDOM](cols) [INTO n BUCKETS]]
   AS SELECT query_expression;
   ```
   
   Column definition supports adding new columns and/or defining columns already existing in the SELECT clause. Also, allows to pass PRIMARY KEY and WATERMARK columns.
   
   
   ## Brief change log
   
     - Added support for column definition, watermarks, primary keys and distribution in the RTAS statement
   
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
   - Added unit tests to validation and converter classes
   - Added IT tests that verify data is written and read correctly
   
   ## Does this pull request potentially affect one of the following parts:
   
    - Dependencies (does it add or upgrade a dependency): no
   - The public API, i.e., is any changed class annotated with @Public(Evolving): no
   - The serializers: no
   - The runtime per-record code paths (performance sensitive): no
   - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
   - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? yes
     - If yes, how is the feature documented? JavaDocs (will follow-up with another PR to update docs)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-35709][Table SQL / API] Allow defining a schema in REPLACE TABLE AS (RTAS) [flink]

Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr merged PR #25247:
URL: https://github.com/apache/flink/pull/25247


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-35709][Table SQL / API] Allow defining a schema in REPLACE TABLE AS (RTAS) [flink]

Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #25247:
URL: https://github.com/apache/flink/pull/25247#discussion_r1730233268


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/MergeTableAsUtil.java:
##########
@@ -63,9 +79,90 @@ public class MergeTableAsUtil {
     }
 
     /**
-     * Merges the schema part of the {@code sqlCreateTableAs} with the {@code sourceSchema}.
+     * Rewrites the query operation to include only the fields that may be persisted in the sink.
+     */
+    public PlannerQueryOperation maybeRewriteQuery(
+            CatalogManager catalogManager,
+            FlinkPlannerImpl flinkPlanner,
+            PlannerQueryOperation origQueryOperation,
+            SqlNode origQueryNode,
+            ResolvedCatalogTable sinkTable) {
+        FlinkCalciteSqlValidator sqlValidator = flinkPlanner.getOrCreateSqlValidator();
+        SqlRewriterUtils rewriterUtils = new SqlRewriterUtils(sqlValidator);
+        FlinkTypeFactory typeFactory = (FlinkTypeFactory) sqlValidator.getTypeFactory();
+
+        // Only fields that may be persisted will be included in the select query
+        RowType sinkRowType =
+                ((RowType) sinkTable.getResolvedSchema().toSinkRowDataType().getLogicalType());
+
+        Map<String, Integer> sourceFields =
+                IntStream.range(0, origQueryOperation.getResolvedSchema().getColumnNames().size())
+                        .boxed()
+                        .collect(
+                                Collectors.toMap(
+                                        origQueryOperation.getResolvedSchema().getColumnNames()
+                                                ::get,
+                                        Function.identity()));
+
+        // assignedFields contains the new sink fields that are not present in the source
+        // and that will be included in the select query
+        LinkedHashMap<Integer, SqlNode> assignedFields = new LinkedHashMap<>();
+
+        // targetPositions contains the positions of the source fields that will be
+        // included in the select query
+        List<Object> targetPositions = new ArrayList<>();

Review Comment:
   Since the type of `sourceFields` is `Map<String, Integer>` , would it more clear to use `List<Integer>`?
   ```suggestion
           List<Integer> targetPositions = new ArrayList<>();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-35709][Table SQL / API] Allow defining a schema in REPLACE TABLE AS (RTAS) [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #25247:
URL: https://github.com/apache/flink/pull/25247#issuecomment-2307651013

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a3b134c444aaa07e0cd513db2b3bdd9d218865d2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a3b134c444aaa07e0cd513db2b3bdd9d218865d2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a3b134c444aaa07e0cd513db2b3bdd9d218865d2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-35709][Table SQL / API] Allow defining a schema in REPLACE TABLE AS (RTAS) [flink]

Posted by "spena (via GitHub)" <gi...@apache.org>.
spena commented on code in PR #25247:
URL: https://github.com/apache/flink/pull/25247#discussion_r1732752338


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/MergeTableAsUtil.java:
##########
@@ -63,9 +79,90 @@ public class MergeTableAsUtil {
     }
 
     /**
-     * Merges the schema part of the {@code sqlCreateTableAs} with the {@code sourceSchema}.
+     * Rewrites the query operation to include only the fields that may be persisted in the sink.
+     */
+    public PlannerQueryOperation maybeRewriteQuery(
+            CatalogManager catalogManager,
+            FlinkPlannerImpl flinkPlanner,
+            PlannerQueryOperation origQueryOperation,
+            SqlNode origQueryNode,
+            ResolvedCatalogTable sinkTable) {
+        FlinkCalciteSqlValidator sqlValidator = flinkPlanner.getOrCreateSqlValidator();
+        SqlRewriterUtils rewriterUtils = new SqlRewriterUtils(sqlValidator);
+        FlinkTypeFactory typeFactory = (FlinkTypeFactory) sqlValidator.getTypeFactory();
+
+        // Only fields that may be persisted will be included in the select query
+        RowType sinkRowType =
+                ((RowType) sinkTable.getResolvedSchema().toSinkRowDataType().getLogicalType());
+
+        Map<String, Integer> sourceFields =
+                IntStream.range(0, origQueryOperation.getResolvedSchema().getColumnNames().size())
+                        .boxed()
+                        .collect(
+                                Collectors.toMap(
+                                        origQueryOperation.getResolvedSchema().getColumnNames()
+                                                ::get,
+                                        Function.identity()));
+
+        // assignedFields contains the new sink fields that are not present in the source
+        // and that will be included in the select query
+        LinkedHashMap<Integer, SqlNode> assignedFields = new LinkedHashMap<>();
+
+        // targetPositions contains the positions of the source fields that will be
+        // included in the select query
+        List<Object> targetPositions = new ArrayList<>();

Review Comment:
   I thought that too, but it doesn't work. The method `rewriterUtils.rewriteSelect()` in line 147 doesn't let me pass a List<Integer>. It is probably because it is a scala code. I even tried to do a casting `(List<Object>) targetPositions` and still it doesn't let me call it that way. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org