[jira] [Created] (SPARK-59620) Late-materialization storage-filter pushdown via splicing

["Peter Toth (Jira)" <ji...@apache.org>]

Peter Toth created SPARK-59620:
----------------------------------

             Summary: Late-materialization storage-filter pushdown via splicing
                 Key: SPARK-59620
                 URL: https://issues.apache.org/jira/browse/SPARK-59620
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Peter Toth


A runtime bloom filter from join runtime filtering is applied today as a Filter above the scan. The scan still reads every value page of every row group, even where the filter drops almost every row. On a selective join over a wide table that read is the dominant cost.

This asks for such a filter to be pushed into the scan instead, so the vectorized Parquet reader can read the filter's key column first, decide which rows survive, and skip the column pages that no surviving row touches. A row group where nothing survives then costs one key-column read and no value-column IO at all.

The scan reports what it saved through new SQL metrics, so a user can see when the filter is paying off and when it is not.

Behind a new SQL config, off by default.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org