[jira] [Created] (SPARK-59056) Pre-size HashSets in ParquetFilters IN/InSet predicate pushdown

["David Mollitor (Jira)" <ji...@apache.org>]

David Mollitor created SPARK-59056:
--------------------------------------

             Summary: Pre-size HashSets in ParquetFilters IN/InSet predicate pushdown
                 Key: SPARK-59056
                 URL: https://issues.apache.org/jira/browse/SPARK-59056
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: David Mollitor


h2. Summary

{{ParquetFilters.makeInPredicate}} builds a {{java.util.HashSet}} for every {{IN}} predicate pushed down to Parquet, without an initial capacity. The final element count is exactly {{values.length}} and is known before the set is populated, so the set can be pre-sized to avoid internal rehashing and the intermediate bucket-array allocations.
h2. Details

{{makeInPredicate}} has 12 near-identical sites of the form:
{code:scala}
val set = new HashSet[T]()
for (value <- values) {
  set.add(...)   // one element per entry in `values`
}
FilterApi.in(column(n), set)
{code}
Each {{HashSet}} starts at the default capacity (16) and rehashes as it grows toward
{{values.length}} elements. This set is only built when the {{IN}} list size exceeds
{{spark.sql.parquet.pushdown.inFilterThreshold}} (default 10); smaller lists take a per-element equality OR-chain and never build a {{{}HashSet{}}}. So by construction the set is always populated with more than the threshold's worth of values, and for larger lists (dozens to thousands – generated {{IN}} lists, dynamic filters) the default-capacity set rehashes several times per pushed-down predicate, on every Parquet file split scanned.
h2. Proposed change

Pre-size each set from the known element count using Guava's
{{Sets.newHashSetWithExpectedSize(values.length)}} (already a Spark dependency; consistent with the existing {{Maps.newHashMapWithExpectedSize}} usage in {{{}catalyst{}}}). The Guava helper is used rather than {{new HashSet(values.length)}} because {{{}java.util.HashSet{}}}'s int constructor treats the argument as bucket capacity, not expected size, so the naive form would still rehash at ~75% load.

The change is capacity-only and behavior-preserving – the resulting predicate semantics are identical.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org