[jira] [Created] (FLINK-40914) Support partial updates on LATERAL SNAPSHOT join build side

["Fabian Hueske (Jira)" <ji...@apache.org>]

Fabian Hueske created FLINK-40914:
-------------------------------------

             Summary: Support partial updates on LATERAL SNAPSHOT join build side
                 Key: FLINK-40914
                 URL: https://issues.apache.org/jira/browse/FLINK-40914
             Project: Flink
          Issue Type: Improvement
          Components: Table SQL / Runtime
            Reporter: Fabian Hueske
            Assignee: Fabian Hueske


LATERAL SNAPSHOT join maintains the build-side of the join in state.
Currently, the state is maintained as a MapState<Row, Long>, supporting any kind of tables as input. One drawback of this implementation is that retractions require full row matches, which is not needed if the input table has a primary key. 
Inputs that are read from upsert connectors with partial deletes (like Kafka) require an expensive ChangelogNormalize operator to derive full deletion rows.

We can significantly improve the performance of such jobs by directly supporting partial deletes in the LateralSnapshotJoin operator.

This requires:
 * Alternative state layout for build-side input with primary key
 * Abstraction of build-side updates and lookups
 * A switch (or new operator implementation) to determine the correct build-side mode
 * Planner changes to configure the switch (or chose the correct operator implementation)
 * Adjustment of LSJ logic in `StreamNonDeterministicUpdatePlanVisitor` (make it less restrictive if build-side has a PK)
 * Tests for the new mode / implementation



--
This message was sent by Atlassian Jira
(v8.20.10#820010)