You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2026/06/27 19:31:30 UTC

[PR] [SPARK-57727][SQL] Fix incorrect query result due to inferAdditionalConstraints incorrectly substituting attributes with non-binary-stable collations [spark]

jiwen624 opened a new pull request, #56836:
URL: https://github.com/apache/spark/pull/56836

   ### What changes were proposed in this pull request?
   Guard the attribute-substitution cases in `QueryPlanConstraints.inferAdditionalConstraints` with `isBinaryStable`, so the transitive inference is skipped when the equated attributes have a non-binary-stable type (e.g. a non-`UTF8_BINARY` collation). Binary-stable types are unaffected.
   
   This mirrors the existing guard in another rule `ConstantPropagation` (SPARK-55647).
   
   ### Why are the changes needed?
   `inferAdditionalConstraints` infers a predicate on `b` from `a = b` and a predicate on `a` by substituting `a` with `b`. This is only valid when `a` and `b` are byte-for-byte interchangeable. Under a non-binary-stable collation, `a = b` is a collation equality (e.g. `'hello' = 'HELLO'` under `UTF8_LCASE`), not byte equality, so substituting into a comparison evaluated in a different collation produces a wrong constraint and silently drops rows:
   ```sql
   CREATE TABLE t (a STRING COLLATE UTF8_LCASE, b STRING COLLATE UTF8_LCASE);
   INSERT INTO t VALUES ('hello', 'HELLO');
   SELECT a, b FROM t WHERE a = b AND a = 'hello' COLLATE UTF8_BINARY;
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   Yes, fixed incorrect query result.
   
   ### How was this patch tested?
   Added UT.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   Yes. Claude Code
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57727][SQL] Fix incorrect query result due to inferAdditionalConstraints incorrectly substituting attributes with non-binary-stable collations [spark]

Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #56836:
URL: https://github.com/apache/spark/pull/56836#issuecomment-4820868895

   While this fix should be correct but it blocks pushdown of same-collation, e.g.,:
   ```SQL
   -- a, b : UTF8_LCASE ; everything same collation
   ... t1.a = t2.b   AND   t1.a = 'x'      -- no COLLATE override
   infer: t2.b = 'x'   (all UTF8_LCASE)
   ```
   I'm thinking about a second approach, probably a bit more complex though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57727][SQL] Fix incorrect query result due to inferAdditionalConstraints incorrectly substituting attributes with non-binary-stable collations [spark]

Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #56836:
URL: https://github.com/apache/spark/pull/56836#issuecomment-4836761468

   This fix works with minimal changes but seems too conservative - it disables legit same-collation inferences.
   Opened: https://github.com/apache/spark/pull/56860 as an alternative.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org