[jira] [Created] (FLINK-39421) Metadata filter push-down for table sources

["Jim Hughes (Jira)" <ji...@apache.org>]

Jim Hughes created FLINK-39421:
----------------------------------

             Summary: Metadata filter push-down for table sources
                 Key: FLINK-39421
                 URL: https://issues.apache.org/jira/browse/FLINK-39421
             Project: Flink
          Issue Type: Improvement
          Components: Table SQL / Planner
            Reporter: Jim Hughes


Predicates on metadata columns (e.g., Kafka offset, timestamp, partition) are excluded from the existing SupportsFilterPushDown mechanism. FilterPushDownSpec serializes RexInputRef indices relative to the scan's produced row type, but metadata columns are appended after physical columns. When ProjectPushDownSpec narrows the row type during compiled plan restoration, metadata indices become out-of-bounds, causing ArrayIndexOutOfBoundsException.

This ticket introduces a dedicated metadata filter push-down path:
- New supportsMetadataFilterPushDown() and applyMetadataFilters() default methods on SupportsReadingMetadata (opt-in; existing sources unaffected)
- New MetadataFilterPushDownSpec that snapshots the row type at optimization time, solving the index mismatch
- Extension to PushFilterIntoTableSourceScanRule to classify predicates as physical, metadata, or mixed

See FLIP-574 (https://cwiki.apache.org/confluence/display/FLINK/FLIP-574+Metadata+Filter+Push-Down+for+Table+Sources) for full design.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)