[jira] [Updated] (SPARK-58272) Enable runtime Bloom filters for safely materialized cached join inputs

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-58272:
----------------------------------
    Fix Version/s:     (was: 5.0.0)

> Enable runtime Bloom filters for safely materialized cached join inputs
> -----------------------------------------------------------------------
>
>                 Key: SPARK-58272
>                 URL: https://issues.apache.org/jira/browse/SPARK-58272
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Chao Sun
>            Assignee: Chao Sun
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> h3. Problem
> Spark injects a runtime Bloom filter only when the join creation side exposes a likely selective predicate above a scan. Persisting that selected input replaces its logical subtree with InMemoryRelation, hiding the predicate even after all cached partitions are materialized. Consequently a large shuffle join may process every application-side row despite the cache already containing the exact candidate join keys, size, and row count.
> {code:sql}
> CACHE TABLE selected_keys AS
> SELECT request_id FROM requests WHERE tenant_id = 42;
> SELECT events.*
> FROM events JOIN selected_keys
> ON events.request_id = selected_keys.request_id;
> {code}
> A visible requests filter can trigger a runtime Bloom filter; the equivalent materialized selected_keys input currently does not.
> h3. Proposed improvement
> Teach runtime-filter injection to recognize a fully materialized cached creation side with accurate, generation-consistent row-count and size statistics. Permit the optimization only when replaying the cached input is safe and there is evidence of pruning benefit, while retaining existing join-type, application-side scan-size, deterministic-key, and feature-enable safeguards.
> Safety must fail closed for incomplete cache materialization, partially consumed partitions, stale updates from an earlier cache generation, non-repeatable or opaque execution, best-effort file reads, and storage levels that cannot provide reliable replay. A dedicated size threshold can distinguish a materialized cache from a creation side that would need its original query recomputed.
> Prior discussion: https://github.com/apache/spark/pull/39377 and SPARK-41867. This proposal uses exact materialized cache statistics and explicit replay/correctness safeguards rather than generally looking through arbitrary cached lineage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org