You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2026/09/15 15:33:00 UTC

[jira] [Created] (SPARK-59541) Coalesce OR-connected equality and IN predicates on the same column into a single IN

David Mollitor created SPARK-59541:
--------------------------------------

             Summary: Coalesce OR-connected equality and IN predicates on the same column into a single IN
                 Key: SPARK-59541
                 URL: https://issues.apache.org/jira/browse/SPARK-59541
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: David Mollitor


h2. Background

Spark leaves OR-connected membership tests on the same column un-merged:
 * {{x = 1 OR x = 2 OR x = 3}} stays a nested {{Or(EqualTo, ...)}} - it is never folded to {{{}IN{}}}.
 * {{x IN (1, 2) OR x IN (3, 4)}} stays two separate {{In}} nodes.
 * {{x = 1 OR x IN (2, 3)}} stays {{{}Or(EqualTo, In){}}}.

{{OptimizeIn}} only processes a single {{In}} node at a time, so none of these are combined today.
h2. Proposal

A new logical optimizer rule {{CombineDisjunctiveInPredicates}} that coalesces OR-connected {{EqualTo}} and {{In}} disjuncts on the same deterministic, non-foldable subject into one {{{}In{}}}, letting the adjacent {{OptimizeIn}} finish the job (dedup, single-element -> {{{}EqualTo{}}}, large list -> {{{}InSet{}}}). {{EqualTo(x, c)}} is treated as a one-element membership {{{}In(x, [c]){}}}. Examples:
 * {{x = 1 OR x = 2 OR x = 3}} -> {{x IN (1, 2, 3)}}
 * {{x IN (1, 2) OR x IN (3, 4)}} -> {{x IN (1, 2, 3, 4)}}
 * {{x = 1 OR x IN (2, 3)}} -> {{x IN (1, 2, 3)}}

The rule runs immediately before {{OptimizeIn}} and is gated by an internal flag {{spark.sql.optimizer.combineDisjunctiveInPredicates.enabled}} (default true).
h2. Why are the changes needed

A single {{{}In{}}}/{{{}InSet{}}} is a strictly more optimized representation than an OR chain of the same predicates:
 * Single subject evaluation: {{In.eval}} evaluates the subject expression once; an {{Or}} of N {{{}EqualTo{}}}/{{{}In{}}} nodes re-evaluates it N times.
 * Set dispatch: once merged and above the conversion threshold, {{OptimizeIn}} turns the {{In}} into {{{}InSet{}}}, which uses a {{switch}} (compact ints) - dispatch that OR-connected nodes never get.
 * Avoids the whole-stage-codegen method-size cliff: a very large OR chain overflows the generated method-size limit and falls back to interpreted execution; the merged {{InSet}} stays compact.

Microbenchmark over a {{Long}} column, 10M rows, non-matching values (worst case), rule off (raw OR) vs rule on:
||N||raw OR (ms)||merged IN/InSet (ms)||
|20|39|40|
|100|148|54|
|500|65142 (interpreted fallback)|82|

Parity at small N and a large win as N grows.
h2. Correctness
 * Determinism guard (the sole correctness gate): the subject must be deterministic - {{In}} evaluates it once, the OR form once per disjunct - so {{rand() = 1 OR rand() = 2}} is left unchanged.
 * Three-valued logic: {{x IN A OR x IN B}} is equivalent to {{x IN (A union B)}} under nulls (both NULL when the subject is NULL; both FALSE when non-null and unmatched with no null member; both NULL when unmatched with a null member), matching {{{}In.eval{}}}.
 * Exclusions: {{EqualNullSafe}} (<=>) is never matched; structs are excluded on subject and members; foldable-only "subjects" are not grouped (avoids the degenerate {{1 IN (x, y)}} from {{{}x = 1 OR y = 1{}}}).
 * Fixed-point safety: fires only on groups with >= 2 candidate disjuncts, collapsing to one {{In}} that cannot re-match; returns the {{Or}} unchanged otherwise.

h2. Caveat: Parquet row-group pruning precision

Merging a large (> {{{}spark.sql.optimizer.inSetConversionThreshold{}}}) OR-of-equalities into {{IN}} > {{InSet}} can make Parquet row-group pruning coarser: a raw OR-of{{{}EqualTo{}}} pushes as {{FilterApi.or(eq, ...)}} (per-value), whereas the merged form pushes as {{FilterApi.in}} (a min/max envelope over the set). The internal flag allows disabling the rule where this matters.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org