[jira] [Created] (FLINK-40343) Support row‑level precise batch incremental update for Materialized Table with Left Join support

["colagy wang (Jira)" <ji...@apache.org>]

colagy wang created FLINK-40343:
-----------------------------------

             Summary: Support row‑level precise batch incremental update for Materialized Table with Left Join support
                 Key: FLINK-40343
                 URL: https://issues.apache.org/jira/browse/FLINK-40343
             Project: Flink
          Issue Type: Improvement
          Components: Table SQL / API
    Affects Versions: 2.3.0
         Environment: Apache Flink 2.3, Apache Paimon
            Reporter: colagy wang


h3. Background
Currently Flink Materialized Table provides two refresh modes:
 # {{{}REFRESH_MODE = CONTINUOUS{}}}: long‑running streaming job. It maintains full operator state for a long time to handle changelog updates. For scenarios where only sparse updates arrive occasionally (e.g. only a few records changed within 180 days), keeping large state for months brings excessive storage and CPU overhead.
 # {{{}REFRESH_MODE = FULL{}}}: periodic batch refresh. Existing batch incremental capability only works for simple append‑only projection/filter. Once the query contains {{{}JOIN{}}}, {{{}GROUP BY{}}}, batch incremental is silently degraded to full recomputation. Even if only a tiny fraction of rows/partitions are modified, the whole dataset and massive partitions have to be recomputed, resulting in extremely high compute cost.
h3. Business Pain Point
Take {{LEFT JOIN(order, product)}} as a typical business case:
 The business runs on a 15‑minute scheduling cycle. Only sparse partial updates come in each cycle.
 * Streaming mode: We have to maintain huge join state continuously, just for very few changed records, state cost is unacceptable for long‑term running.
 * Existing full batch mode: Every refresh triggers recomputation of massive unchanged partitions, compute cost is expensive and wasteful.
We expect a *row‑level precise batch incremental refresh mode* for Materialized Table, based on Paimon source table:
 # It does NOT require long‑lived streaming operator state; there is no large state to persist for months.
 # Only process rows/partitions which have real changes within the scheduling interval (for example every 15 minutes). Avoid recomputing massive untouched partitions.
 # Support incremental computation for {{{}LEFT JOIN{}}}. This feature can be disabled by default for safety, but should be available when users explicitly enable it.
h3. Industry Reference
Aliyun Ververica platform has already implemented similar precise batch‑incremental materialized table capability in production. It proves this technical solution is feasible and valuable for real‑world production workloads.
h3. Expected Benefit
 # Reduce compute resource cost: avoid full partition recomputation when only sparse updates occur.
 # Eliminate heavy long‑term streaming state maintenance cost for low‑change‑frequency business.
 # Expand batch incremental capability beyond simple filter/projection, support common join workloads such as {{{}LEFT JOIN{}}}.
h3. Suggested acceptance criteria
 * New refresh mode for Materialized Table: precise row‑level batch incremental update against Paimon source.
 * Support incremental {{LEFT JOIN}} calculation, guarded by an explicit config switch (disabled by default).
 * Only process actually‑modified data set in each scheduled batch run; skip untouched partitions/rows.
 * No requirement for persistent long‑running streaming operator state.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)