[jira] [Created] (FLINK-28566) Adds materialization support to eliminate the non determinism generated by lookup join node

["lincoln lee (Jira)" <ji...@apache.org>]

lincoln lee created FLINK-28566:
-----------------------------------

             Summary: Adds materialization support to eliminate the non determinism generated by lookup join node
                 Key: FLINK-28566
                 URL: https://issues.apache.org/jira/browse/FLINK-28566
             Project: Flink
          Issue Type: Sub-task
          Components: Table SQL / Planner
            Reporter: lincoln lee
             Fix For: 1.16.0


In order to minimize the potential exceptions or data errors when many users use the update stream to lookup join an external 
table (essentially due to the non-deterministic result based on processing-time to lookup external tables). 
When update exists in the input stream and the lookup key does not contain the primary key of the external table,
FLINK automatically adds materialization of the update by default, so that it will only lookup the external table 
when the insert or update_after message arrives, and when the delete or update_before message arrives, it will 
directly querying the latest version of the locally materialized data and sent it to downstream operator.

To do so，we introduce a new option 'table.exec.lookup-join.upsert-materialize' and resue the `UpsertMaterialize`. By default, the materialize operator will be added when an update stream lookup an external table without containing its primary keys(includes no primary key defined). You can also choose no materialization(NONE) or force materialization(FORCE) which will always enable materialization except input is insert only.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)