[jira] [Created] (SPARK-60448) Apply filter conditions on a storage filter's key columns in the Parquet reader

["Victor Sunderland (Jira)" <ji...@apache.org>]

Victor Sunderland created SPARK-60448:
-----------------------------------------

             Summary: Apply filter conditions on a storage filter's key columns in the Parquet reader
                 Key: SPARK-60448
                 URL: https://issues.apache.org/jira/browse/SPARK-60448
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Victor Sunderland


A follow-up to SPARK-59620.

With {{spark.sql.parquet.storageFilterPushdown.enabled}} on, the vectorized Parquet reader reads the join key column first, checks each row against the join's runtime bloom filter, and reads the other columns only for the rows that pass. Any other condition on that same key column, such as {{k % 50 = 0}}, is still evaluated only in the Filter after the scan. The reader therefore reads the other columns for rows that are dropped right afterwards.

This proposes offering such conditions to the reader along with the bloom filter. The reader already holds the key column, so evaluating one more condition on it adds no I/O, and the rows it rejects never have their other columns read. Concretely:
* A new hook, {{FileFormat.supportsStorageFilterOnKeys(expr)}}, false by default. Parquet accepts any condition it can evaluate as is (no subqueries, no unevaluable expressions such as Python UDFs).
* {{FileSourceStrategy}} offers a conjunct that references only the bloom filter's key columns after the bloom filter. {{IsNotNull}} is not offered, since the post-scan Filter needs it as is to mark the key column non-nullable.
* The reader evaluates these conditions only on the rows the bloom filter keeps, so that subexpression elimination does not compute what they share on every row.

This does not help a plain inner join, where Spark already infers such a condition onto the other side before it builds the bloom filter. It helps when the condition is not in the bloom filter, for example when the fact table is on the nullable side of a LEFT OUTER JOIN and the condition is inside a derived table, or for rows that pass only as bloom filter false positives. It saves the most with wide rows in small pages, where removing rows empties whole pages that the reader can then skip.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org