You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "xishuaidelin (via GitHub)" <gi...@apache.org> on 2025/05/30 08:15:36 UTC

[PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

xishuaidelin opened a new pull request, #26616:
URL: https://github.com/apache/flink/pull/26616

   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   *This pull request aims to introduce the option and support key ordered async look up join in planner.*
   
   ## Brief change log
   
   1. introduce a new option for control the key ordered async lookup join.
   2. lookup join related exec nodes  are modified.
   
   
   ## Verifying this change
   
   1. option related test are added to verify the correctness of new option
   2. plan test is added to verify the changes of exec nodes.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**yes** / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115545347


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   nits:
   I was not sure what `true async lookup joins ` were. I suggest 
   `When true, async lookup joins follow the upsert key order in the CDC stream.`
   I suggest removing the brackets and having a new sentence.
   `If there is no defined upsert key, then the total record is considered as the upsert key.`
   
   nit: This feature does not works for insert-only. -> setting this for insert-only has no effect.
   
   Could we say why this has no effect for insert only? is it not possible to order the inserts by key in this case?  
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122774679


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   Hi, @davidradl .  Sorry for my late reply and thanks for your good catch here. 
   
   This optimization does not apply to insert-only streams because its purpose is to enable potential unordered optimizations in ordered scenarios. For insert-only stream, if allow_unordered is enabled then it has already undergone unordered processing. In another word, we don't expect to change the current behaviour on insert-only streams. And you could check LookupJoinUtil#getMergedAsyncOptions For details.
   
   I would clarify this in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122774679


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   Hi, @davidradl .  Sorry for my late reply and thanks for your good catch here. 
   
   This optimization is not applicable to insert-only streams, as each record in such streams is independent and does not affect the state of previous records. Moreover, since these records typically do not involve a primary key, no upsertKey can be derived. As a result, processing them in an unordered mode has no difference with processing them in key-ordered mode.
   
   I would clarify this in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2128191954


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -310,6 +314,21 @@ protected Transformation<RowData> createJoinTransformation(
         } else {
             StreamOperatorFactory<RowData> operatorFactory;
             if (isAsyncEnabled) {
+                if (asyncLookupOptions.keyOrdered) {
+                    return createKeyOrderedAsyncLookupJoin(

Review Comment:
   would it not be clearer to have a transformation to do the ordering then call the existing method to do the createAsyncLookupJoin at this level - rather than deeper in the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122774679


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   Hi, @davidradl .  Sorry for my late reply and thanks for your good catch here. 
   This optimization does not apply to insert-only streams because its purpose is to enable potential unordered optimizations in ordered scenarios. 
   For insert-only stream, if unordered is allowed then the it has already undergone unordered processing. So we don't consider insert-only streams here. 
   I will update this in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong merged PR #26616:
URL: https://github.com/apache/flink/pull/26616


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115564263


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -211,6 +214,64 @@ public Transformation<RowData> translateToPlanInternal(
                 planner, config, upsertMaterialize, lookupKeyContainsPrimaryKey);
     }
 
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        int[] shuffleKeys = inputUpsertKey;
+        // normally upsertKeys could not be null. If the job is restored from exec plan then

Review Comment:
   This comment could be clearer - can we say something like:
   
    If the job is restored from exec plan then the upsert keys could be null, because .... in this case we ....
   In all other cases the upsert keys are not null. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115606722


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/LookupJoinUtil.java:
##########
@@ -209,23 +210,30 @@ public static class AsyncLookupOptions {
         public static final String FIELD_NAME_CAPACITY = "capacity ";
         public static final String FIELD_NAME_TIMEOUT = "timeout";
         public static final String FIELD_NAME_OUTPUT_MODE = "output-mode";
+        public static final String FIELD_NAME_KEY_ORDERED_MODE = "key-ordered-mode";

Review Comment:
   why is this variable called a mode. I suggest IS_FIELD_NAME_KEY_ORDERED



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2134919493


##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/join/LookupJoinTest.scala:
##########
@@ -58,6 +58,28 @@ class LookupJoinTest extends TableTestBase with Serializable {
     util.addDataStream[(Int, String, Long, Double)]("T1", 'a, 'b, 'c, 'd)
     util.addDataStream[(Int, String, Int)]("nonTemporal", 'id, 'name, 'age)
 
+    util.addTable("""
+                    |CREATE TABLE UpsertTable (
+                    |  `id` INT,
+                    |  `name` STRING,
+                    |  `age` INT
+                    |) WITH (
+                    |  'connector' = 'values',
+                    |  'changelog-mode' = 'I,UA,UB,D'
+                    |)
+                    |""".stripMargin)
+
+    util.addTable("""
+                    |CREATE TABLE UpsertTableAppendOnly (

Review Comment:
   `Upsert` and `AppendOnly` conflicts.



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/LookupJoinUtil.java:
##########
@@ -209,23 +210,30 @@ public static class AsyncLookupOptions {
         public static final String FIELD_NAME_CAPACITY = "capacity ";
         public static final String FIELD_NAME_TIMEOUT = "timeout";
         public static final String FIELD_NAME_OUTPUT_MODE = "output-mode";
+        public static final String FIELD_NAME_KEY_ORDERED_MODE = "key-ordered-mode";

Review Comment:
   Agree with @davidradl. What about using:
   ```
   FIELD_NAME_IS_KEY_ORDERED =  "is-key-ordered"
   ```



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/batch/BatchExecLookupJoin.java:
##########
@@ -168,4 +169,22 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
             boolean lookupKeyContainsPrimaryKey) {
         return inputTransformation;
     }
+
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        throw new UnsupportedOperationException(

Review Comment:
   I known. Please add a test for it to ensure others will not break this limit in future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122774679


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   Hi, @davidradl .  Sorry for my late reply and thanks for your good catch here. 
   This optimization does not apply to insert-only streams because its purpose is to enable potential unordered optimizations in ordered scenarios. 
   For insert-only stream, if allow_unordered is enabled then it has already undergone unordered processing. So we don't consider insert-only streams here. 
   I will update this in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122878209


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -310,6 +314,21 @@ protected Transformation<RowData> createJoinTransformation(
         } else {
             StreamOperatorFactory<RowData> operatorFactory;
             if (isAsyncEnabled) {
+                if (asyncLookupOptions.keyOrdered) {
+                    return createKeyOrderedAsyncLookupJoin(

Review Comment:
   Actually we need to add a new transformation in key-ordered mode here. The existing createAsyncLookupJoin method is to generate a StreamOperatorFactory. A new method would be more clear .



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115564263


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -211,6 +214,64 @@ public Transformation<RowData> translateToPlanInternal(
                 planner, config, upsertMaterialize, lookupKeyContainsPrimaryKey);
     }
 
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        int[] shuffleKeys = inputUpsertKey;
+        // normally upsertKeys could not be null. If the job is restored from exec plan then

Review Comment:
   This comment could be clearer - can we say something like:
   
    If the job is restored from exec plan then the upsert keys could be null, because ....
   In all other cases the upsert keys are not null. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115619379


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -481,7 +516,14 @@ private StreamOperatorFactory<RowData> createAsyncLookupJoin(
                             isLeftOuterJoin,
                             asyncLookupOptions.asyncBufferCapacity);
         }
-
+        if (asyncLookupOptions.keyOrdered) {
+            checkNotNull(keySelector);

Review Comment:
   when can this be null? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122774679


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   Hi, @davidradl .  Sorry for my late reply and thanks for your good catch here. 
   This optimization does not apply to insert-only streams because its purpose is to enable potential unordered optimizations in ordered scenarios. For insert-only stream, if allow_unordered is enabled then it has already undergone unordered processing. In another word, we don't expect to change the current behaviour on insert-only streams.  
   
   `    private static AsyncDataStream.OutputMode convert(
               ChangelogMode inputChangelogMode,
               ExecutionConfigOptions.AsyncOutputMode asyncOutputMode) {
           if (inputChangelogMode.containsOnly(RowKind.INSERT)
                   && asyncOutputMode == ExecutionConfigOptions.AsyncOutputMode.ALLOW_UNORDERED) {
               return AsyncDataStream.OutputMode.UNORDERED;
           }
           return AsyncDataStream.OutputMode.ORDERED;
       }`
       
       
   I would clarify this in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2125997643


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -211,6 +214,64 @@ public Transformation<RowData> translateToPlanInternal(
                 planner, config, upsertMaterialize, lookupKeyContainsPrimaryKey);
     }
 
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        int[] shuffleKeys = inputUpsertKey;
+        // normally upsertKeys could not be null. If the job is restored from exec plan then
+        // upsertKeys could be null
+        if (shuffleKeys == null || shuffleKeys.length == 0) {
+            shuffleKeys = IntStream.range(0, inputRowType.getFieldCount()).toArray();
+        }
+
+        RowDataKeySelector keySelector =
+                getKeySelector(false, shuffleKeys, classLoader, inputRowType);
+
+        Transformation<RowData> partitionedTransform =
+                createPartitionTransformation(keySelector, inputTransformation, false, config);
+
+        StreamOperatorFactory<RowData> operatorFactory;
+
+        operatorFactory =
+                createAsyncLookupJoin(

Review Comment:
   The operator impl is not ready, right? Can we throw an exception here instead of using a wrong operator?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2134997737


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -310,6 +314,21 @@ protected Transformation<RowData> createJoinTransformation(
         } else {
             StreamOperatorFactory<RowData> operatorFactory;
             if (isAsyncEnabled) {
+                if (asyncLookupOptions.keyOrdered) {
+                    return createKeyOrderedAsyncLookupJoin(

Review Comment:
   For batch processing, the behavior is fundamentally different. This solution doesn't introduce more complexity; instead, it helps to keep the code clean by separating concerns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xuyangzhong (via GitHub)" <gi...@apache.org>.
xuyangzhong commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2125897133


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/batch/BatchExecLookupJoin.java:
##########
@@ -168,4 +169,22 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
             boolean lookupKeyContainsPrimaryKey) {
         return inputTransformation;
     }
+
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        throw new UnsupportedOperationException(

Review Comment:
   This exception message is quite confusing because the changelog mode under batch is always insert-only. How about using the following message for the IllegalStateException: "Batch mode should not use key-ordered async lookup joins. This is a bug. Please file an issue."?



##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/join/LookupJoinTest.scala:
##########
@@ -848,6 +859,17 @@ class LookupJoinTest extends TableTestBase with Serializable {
     util.verifyExecPlan(sql)
   }
 
+  @Test
+  def testJoinAsyncTableKeyOrdered(): Unit = {
+    util.tableEnv.getConfig
+      .set(ExecutionConfigOptions.TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED, Boolean.box(true))
+    val sql =
+      "SELECT /*+ LOOKUP('table'='D', 'async'='true', 'output-mode'='allow_unordered') */ * " +

Review Comment:
   Could you please add an ut about plan for append-only source?



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -294,4 +355,47 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
         transform.setStateKeyType(keySelector.getProducedType());
         return transform;
     }
+
+    private RowDataKeySelector getKeySelector(
+            boolean singleParallelism, int[] keys, ClassLoader classLoader, RowType inputRowType) {
+        RowDataKeySelector keySelector;
+        if (singleParallelism) {
+            keySelector = EmptyRowDataKeySelector.INSTANCE;
+        } else {
+            // make it a deterministic asc order
+            Arrays.sort(keys);
+            keySelector =
+                    KeySelectorUtil.getRowDataSelector(
+                            classLoader, keys, InternalTypeInfo.of(inputRowType));
+        }
+        return keySelector;
+    }
+
+    private Transformation<RowData> createPartitionTransformation(
+            RowDataKeySelector keySelector,
+            Transformation<RowData> inputTransformation,
+            boolean singleParallelism,
+            ExecNodeConfig config) {
+
+        final KeyGroupStreamPartitioner<RowData, RowData> partitioner =
+                new KeyGroupStreamPartitioner<>(
+                        keySelector, KeyGroupRangeAssignment.DEFAULT_LOWER_BOUND_MAX_PARALLELISM);
+        Transformation<RowData> partitionedTransform =
+                new PartitionTransformation<>(inputTransformation, partitioner);
+
+        createTransformationMeta(PARTITIONER_TRANSFORMATION, "Partitioner", "Partitioner", config)

Review Comment:
   Why we import this field from `CommonExecSink`...? Can we use a new & same one in `StreamExecLookupJoin`?



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -294,4 +355,47 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
         transform.setStateKeyType(keySelector.getProducedType());
         return transform;
     }
+
+    private RowDataKeySelector getKeySelector(

Review Comment:
   could you please extract the same logic in `createSyncLookupJoinWithState`?



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -211,6 +214,64 @@ public Transformation<RowData> translateToPlanInternal(
                 planner, config, upsertMaterialize, lookupKeyContainsPrimaryKey);
     }
 
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        int[] shuffleKeys = inputUpsertKey;
+        // normally upsertKeys could not be null. If the job is restored from exec plan then

Review Comment:
   actually, if we could not get the upsert key from its input, the upsert key may also be  null. 



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -294,4 +355,47 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
         transform.setStateKeyType(keySelector.getProducedType());
         return transform;
     }
+
+    private RowDataKeySelector getKeySelector(
+            boolean singleParallelism, int[] keys, ClassLoader classLoader, RowType inputRowType) {
+        RowDataKeySelector keySelector;
+        if (singleParallelism) {
+            keySelector = EmptyRowDataKeySelector.INSTANCE;

Review Comment:
   This logic has already existed in `KeySelectorUtil.getRowDataSelector`.



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -481,7 +516,14 @@ private StreamOperatorFactory<RowData> createAsyncLookupJoin(
                             isLeftOuterJoin,
                             asyncLookupOptions.asyncBufferCapacity);
         }
-
+        if (asyncLookupOptions.keyOrdered) {
+            checkNotNull(keySelector);
+            return new TableKeyedAsyncWaitOperatorFactory<>(
+                    asyncFunc,
+                    keySelector,
+                    asyncLookupOptions.asyncTimeout,
+                    asyncLookupOptions.asyncBufferCapacity);
+        }

Review Comment:
   Can we double check in this branch, the `keySelector` is always null?



##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/serde/AsyncLookupOptionsTest.java:
##########
@@ -136,5 +140,35 @@ void testAsyncLookupOptions() {
                         userConf,
                         ChangelogMode.all());
         assertThat(asyncLookupOptions.asyncOutputMode).isSameAs(AsyncDataStream.OutputMode.ORDERED);
+        assertFalse(asyncLookupOptions.keyOrdered);
+
+        TableConfig config = TableConfig.getDefault();
+        config.set(TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED, true);
+        asyncLookupOptions =
+                LookupJoinUtil.getMergedAsyncOptions(
+                        LookupJoinHintTestUtil.completeLookupHint, config, ChangelogMode.all());
+        assertTrue(asyncLookupOptions.asyncOutputMode == AsyncDataStream.OutputMode.ORDERED);

Review Comment:
   nit: use `assertThat...isEqualTo` to resolve the warnings.
   Others are same as well.



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/batch/BatchExecLookupJoin.java:
##########
@@ -168,4 +169,22 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
             boolean lookupKeyContainsPrimaryKey) {
         return inputTransformation;
     }
+
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        throw new UnsupportedOperationException(

Review Comment:
   Moreover, please add a test for batch.



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -294,4 +355,47 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
         transform.setStateKeyType(keySelector.getProducedType());
         return transform;
     }
+
+    private RowDataKeySelector getKeySelector(
+            boolean singleParallelism, int[] keys, ClassLoader classLoader, RowType inputRowType) {
+        RowDataKeySelector keySelector;
+        if (singleParallelism) {
+            keySelector = EmptyRowDataKeySelector.INSTANCE;
+        } else {
+            // make it a deterministic asc order
+            Arrays.sort(keys);
+            keySelector =
+                    KeySelectorUtil.getRowDataSelector(
+                            classLoader, keys, InternalTypeInfo.of(inputRowType));
+        }
+        return keySelector;
+    }
+
+    private Transformation<RowData> createPartitionTransformation(
+            RowDataKeySelector keySelector,
+            Transformation<RowData> inputTransformation,
+            boolean singleParallelism,

Review Comment:
   nit: Is this field always false?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2134993401


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -211,6 +214,64 @@ public Transformation<RowData> translateToPlanInternal(
                 planner, config, upsertMaterialize, lookupKeyContainsPrimaryKey);
     }
 
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        int[] shuffleKeys = inputUpsertKey;
+        // normally upsertKeys could not be null. If the job is restored from exec plan then

Review Comment:
   Fine. I'd like to update this commet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122906049


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -294,4 +355,47 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
         transform.setStateKeyType(keySelector.getProducedType());
         return transform;
     }
+
+    private RowDataKeySelector getKeySelector(
+            boolean singleParallelism, int[] keys, ClassLoader classLoader, RowType inputRowType) {
+        RowDataKeySelector keySelector;
+        if (singleParallelism) {
+            keySelector = EmptyRowDataKeySelector.INSTANCE;
+        } else {
+            // make it a deterministic asc order
+            Arrays.sort(keys);

Review Comment:
   They are the index of upsertKey.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122910551


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -481,7 +516,14 @@ private StreamOperatorFactory<RowData> createAsyncLookupJoin(
                             isLeftOuterJoin,
                             asyncLookupOptions.asyncBufferCapacity);
         }
-
+        if (asyncLookupOptions.keyOrdered) {
+            checkNotNull(keySelector);

Review Comment:
   async lookup join without key ordered mode



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115554582


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)

Review Comment:
   I wonder if the default should be true, as most of the time you want the ordering done to ease subsequent processing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on PR #26616:
URL: https://github.com/apache/flink/pull/26616#issuecomment-2955624549

   Hi @davidradl , is there anything else you'd like to add?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2134998765


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")

Review Comment:
   Thanks for the suggestion, but I believe ‘ordered’ is more appropriate here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122925969


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -481,7 +516,14 @@ private StreamOperatorFactory<RowData> createAsyncLookupJoin(
                             isLeftOuterJoin,
                             asyncLookupOptions.asyncBufferCapacity);
         }
-
+        if (asyncLookupOptions.keyOrdered) {
+            checkNotNull(keySelector);
+            return new TableKeyedAsyncWaitOperatorFactory<>(

Review Comment:
   TableKeyedAsyncWaitOperatorFactory serves for Table layer while AsyncWaitOperatorFactory serves for DataStream. We just want to implement without modify existing methods in DataStream.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115584086


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   Am I correct in saying , by default the order with be time order. With this option the keys will be ordered alphabetically if a string or numerically if a number. We should mention this in the description. Is it possible to have other types as keys?   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115584086


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   We should indicate why a use might set this. We think that there isa performance improvement in some cases, but the sorting must be extra processing. We should mention this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115591333


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -294,4 +355,47 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
         transform.setStateKeyType(keySelector.getProducedType());
         return transform;
     }
+
+    private RowDataKeySelector getKeySelector(
+            boolean singleParallelism, int[] keys, ClassLoader classLoader, RowType inputRowType) {
+        RowDataKeySelector keySelector;
+        if (singleParallelism) {
+            keySelector = EmptyRowDataKeySelector.INSTANCE;
+        } else {
+            // make it a deterministic asc order
+            Arrays.sort(keys);

Review Comment:
   do the integers representing the keys mean anything here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2131866667


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/batch/BatchExecLookupJoin.java:
##########
@@ -168,4 +169,22 @@ protected Transformation<RowData> createSyncLookupJoinWithState(
             boolean lookupKeyContainsPrimaryKey) {
         return inputTransformation;
     }
+
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        throw new UnsupportedOperationException(

Review Comment:
   Hi @xuyangzhong. Thansk for review. Actually this branch would never be reached because batch source is insert-only which would not optimized as async key ordered lookup join.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115564263


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecLookupJoin.java:
##########
@@ -211,6 +214,64 @@ public Transformation<RowData> translateToPlanInternal(
                 planner, config, upsertMaterialize, lookupKeyContainsPrimaryKey);
     }
 
+    @Override
+    protected Transformation<RowData> createKeyOrderedAsyncLookupJoin(
+            Transformation<RowData> inputTransformation,
+            RelOptTable temporalTable,
+            ExecNodeConfig config,
+            ClassLoader classLoader,
+            Map<Integer, LookupJoinUtil.LookupKey> allLookupKeys,
+            AsyncTableFunction<Object> asyncLookupFunction,
+            RelBuilder relBuilder,
+            RowType inputRowType,
+            RowType tableSourceRowType,
+            RowType resultRowType,
+            boolean isLeftOuterJoin,
+            LookupJoinUtil.AsyncLookupOptions asyncLookupOptions) {
+        int[] shuffleKeys = inputUpsertKey;
+        // normally upsertKeys could not be null. If the job is restored from exec plan then

Review Comment:
   This comment could be clearer - can we say something like:
   
    If the job is restored from exec plan then the upsert keys could be null, because .... in this case we ....
   In all other cases the upsert keys are not null. 
   
   nit we could write this as:
   
   ```
    int[] shuffleKeys = (inputUpsertKey == null || inputUpsertKey.length == 0)?
   IntStream.range(0, inputRowType.getFieldCount()).toArray():  inputUpsertKey;
   ```
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115554582


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)

Review Comment:
   I wonder if the default should be true, as most of the time you want the ordering done to ease subsequent processing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "xishuaidelin (via GitHub)" <gi...@apache.org>.
xishuaidelin commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2122774679


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "When true async lookup joins would follow the upsert key order in cdc stream (if no upsert key, "

Review Comment:
   Hi, @davidradl .  Sorry for my late reply and thanks for your good catch here. 
   This optimization does not apply to insert-only streams because its purpose is to enable potential unordered optimizations in ordered scenarios. 
   For insert-only stream, if allow_unordered is enabled then the it has already undergone unordered processing. So we don't consider insert-only streams here. 
   I will update this in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115548946


##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/ExecutionConfigOptions.java:
##########
@@ -374,6 +374,15 @@ public class ExecutionConfigOptions {
     // ------------------------------------------------------------------------
     //  Async Lookup Options
     // ------------------------------------------------------------------------
+    @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH_STREAMING)
+    public static final ConfigOption<Boolean> TABLE_EXEC_ASYNC_LOOKUP_KEY_ORDERED =
+            key("table.exec.async-lookup.key-ordered-enabled")

Review Comment:
   nit: I think `ordering` be better than `ordered` in the key name. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26616:
URL: https://github.com/apache/flink/pull/26616#issuecomment-2921592719

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5c55211e0976dde3e3c2e490772fe8aaa05c4636",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5c55211e0976dde3e3c2e490772fe8aaa05c4636",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5c55211e0976dde3e3c2e490772fe8aaa05c4636 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115558888


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -310,6 +314,21 @@ protected Transformation<RowData> createJoinTransformation(
         } else {
             StreamOperatorFactory<RowData> operatorFactory;
             if (isAsyncEnabled) {
+                if (asyncLookupOptions.keyOrdered) {
+                    return createKeyOrderedAsyncLookupJoin(

Review Comment:
   why is there a new method rather than adding parameter(s) to the existing createAsyncLookupJoin



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37875][table] Introduce key ordered async lookup join in planner [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26616:
URL: https://github.com/apache/flink/pull/26616#discussion_r2115626093


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecLookupJoin.java:
##########
@@ -481,7 +516,14 @@ private StreamOperatorFactory<RowData> createAsyncLookupJoin(
                             isLeftOuterJoin,
                             asyncLookupOptions.asyncBufferCapacity);
         }
-
+        if (asyncLookupOptions.keyOrdered) {
+            checkNotNull(keySelector);
+            return new TableKeyedAsyncWaitOperatorFactory<>(

Review Comment:
   I wonder if we need a new TableKeyedAsyncWaitOperatorFactory class, then there is the existing factory AsyncWaitOperatorFactory which seems the same apart from the keyselector parameter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org