[jira] [Created] (FLINK-39584) Expand filter-merging for source-reuse

["Daniel Rossos (Jira)" <ji...@apache.org>]

Daniel Rossos created FLINK-39584:
-------------------------------------

             Summary: Expand filter-merging for source-reuse
                 Key: FLINK-39584
                 URL: https://issues.apache.org/jira/browse/FLINK-39584
             Project: Flink
          Issue Type: Improvement
          Components: Table SQL / Planner
            Reporter: Daniel Rossos


h1. Summary

ScanReuser does not have the option to unify scans of the same table that differ in their pushed-down filters. Sources with different FilterPushDownSpec values are treated as distinct because FilterPushDownSpec is included in the scan digest used by ReusableScanVisitor. This causes duplicate source reads even when the underlying data could be served by a single read.

For context, ScanReuser already unifies scans with different projections by creating a single scan with the superset of columns and adding per-consumer Calc nodes. The same pattern does not extend to filter differences today.
h1. 
Impact


For sources where data scanning is expensive computationally or monetarily (BigQuery Storage Read API sessions, JDBC query execution) this means N reads when one could suffice. Cost scales with the number of references rather than with the volume of unique data. This is especially pronounced in the multi-sink StatementSet fan-out pattern, where one upstream table feeds multiple downstream sinks with different filters.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)