You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/09/08 13:55:29 UTC

[GitHub] [flink] lincoln-lil opened a new pull request, #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results

lincoln-lil opened a new pull request, #20791:
URL: https://github.com/apache/flink/pull/20791

   ## What is the purpose of the change
   Currently SinkUpsertMaterializer only update row by comparing the complete row in anycase, but this may cause wrong result if input has upsertKey and also non-deterministic column values, see such a case:
   
   ```java
   @Test
   public void testCdcWithNonDeterministicFuncSinkWithDifferentPk() {
   tEnv.createTemporaryFunction(
   "ndFunc", new JavaUserDefinedScalarFunctions.NonDeterministicUdf());
   
   String cdcDdl =
   "CREATE TABLE users (\n"
   + " user_id STRING,\n"
   + " user_name STRING,\n"
   + " email STRING,\n"
   + " balance DECIMAL(18,2),\n"
   + " primary key (user_id) not enforced\n"
   + ") WITH (\n"
   + " 'connector' = 'values',\n"
   + " 'changelog-mode' = 'I,UA,UB,D'\n"
   + ")";
   
   String sinkTableDdl =
   "CREATE TABLE sink (\n"
   + " user_id STRING,\n"
   + " user_name STRING,\n"
   + " email STRING,\n"
   + " balance DECIMAL(18,2),\n"
   + " PRIMARY KEY(email) NOT ENFORCED\n"
   + ") WITH(\n"
   + " 'connector' = 'values',\n"
   + " 'sink-insert-only' = 'false'\n"
   + ")";
   tEnv.executeSql(cdcDdl);
   tEnv.executeSql(sinkTableDdl);
   
   util.verifyJsonPlan(
   "insert into sink select user_id, ndFunc(user_name), email, balance from users");
   }
   ```
   
   for original cdc source records:
   
   ```gittext
   +I[user1, Tom, tom@gmail.com, 10.02],
   -D[user1, Tom, tom@gmail.com, 10.02],
   ```
   
   the above query cannot correctly delete the former insertion row because of the non-deterministic column value 'ndFunc(user_name)'
   
   this canbe solved by letting the SinkUpsertMaterializer be aware of input upsertKey and update by it. This pr aims to address it.
   
   ## Brief change log
   * Add projectRowType to RowTypeUtils and deprecate AggCodeGenHelper#projectRowType
   * Move non-deterministic test functions to userDefinedScalarFunctions
   * Add upsertKey info for SinkUpsertMaterializer and do update via upsertKey if has one (if no upsertKey,  code path keeps unchanged)
   
   ## Verifying this change
   add more related ut & it
   
   ## Does this pull request potentially affect one of the following parts:
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
     - The serializers: (no )
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
     - Does this pull request introduce a new feature? (no)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] godfreyhe commented on a diff in pull request #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results

Posted by GitBox <gi...@apache.org>.
godfreyhe commented on code in PR #20791:
URL: https://github.com/apache/flink/pull/20791#discussion_r970368440


##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/agg/batch/AggCodeGenHelper.scala:
##########
@@ -93,6 +93,8 @@ object AggCodeGenHelper {
       .asInstanceOf[Map[AggregateFunction[_, _], String]]
   }
 
+  /** @deprecated Use [[RowTypeUtils#projectRowType]] instead. */
+  @deprecated
   def projectRowType(rowType: RowType, mapping: Array[Int]): RowType = {

Review Comment:
   remove this method since it's only be used in tests



##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/TableSinkITCase.scala:
##########
@@ -157,6 +173,52 @@ class TableSinkITCase(mode: StateBackendMode) extends StreamingWithStateTestBase
     assertEquals(expected.sorted, result.sorted)
   }
 
+  @Test
+  def testCdcWithNonDeterministicFuncSinkWithDifferentPk(): Unit = {

Review Comment:
   cdc -> non-insert-source ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results

Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #20791:
URL: https://github.com/apache/flink/pull/20791#issuecomment-1240760497

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "add298232cfe66ffc5129b153fbc4c929ebd40b3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "add298232cfe66ffc5129b153fbc4c929ebd40b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * add298232cfe66ffc5129b153fbc4c929ebd40b3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lincoln-lil commented on a diff in pull request #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results

Posted by GitBox <gi...@apache.org>.
lincoln-lil commented on code in PR #20791:
URL: https://github.com/apache/flink/pull/20791#discussion_r970398258


##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/TableSinkITCase.scala:
##########
@@ -157,6 +173,52 @@ class TableSinkITCase(mode: StateBackendMode) extends StreamingWithStateTestBase
     assertEquals(expected.sorted, result.sorted)
   }
 
+  @Test
+  def testCdcWithNonDeterministicFuncSinkWithDifferentPk(): Unit = {

Review Comment:
   will rename it to  ChangelogSource



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lincoln-lil commented on pull request #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results

Posted by GitBox <gi...@apache.org>.
lincoln-lil commented on PR #20791:
URL: https://github.com/apache/flink/pull/20791#issuecomment-1246329609

   @godfreyhe thanks for reviewing this! I've updated the pr according to your comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] godfreyhe closed pull request #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results

Posted by GitBox <gi...@apache.org>.
godfreyhe closed pull request #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results
URL: https://github.com/apache/flink/pull/20791


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lincoln-lil commented on a diff in pull request #20791: [FLINK-28569][table-planner] Fix SinkUpsertMaterializer that should be aware of the input upsertKey if it is not empty to prevent wrong results

Posted by GitBox <gi...@apache.org>.
lincoln-lil commented on code in PR #20791:
URL: https://github.com/apache/flink/pull/20791#discussion_r970397264


##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/agg/batch/AggCodeGenHelper.scala:
##########
@@ -93,6 +93,8 @@ object AggCodeGenHelper {
       .asInstanceOf[Map[AggregateFunction[_, _], String]]
   }
 
+  /** @deprecated Use [[RowTypeUtils#projectRowType]] instead. */
+  @deprecated
   def projectRowType(rowType: RowType, mapping: Array[Int]): RowType = {

Review Comment:
   Ok, I'll remove it directly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org