Re: [PR] [SPARK-50593][SQL] SPJ: Support truncate transform via generalized ReducibleFunction API [spark]

["peter-toth (via GitHub)" <gi...@apache.org>]

peter-toth commented on PR #55885:
URL: https://github.com/apache/spark/pull/55885#issuecomment-5760079728

   @metanil a heads-up on sunchao's [P1 about the raw partition-key equality fast path](https://github.com/apache/spark/pull/55885#discussion_r3679486000): it is not specific to this PR.
   
   The same hole is live on master, through the identity-versus-transform arm SPARK-56182 added. A table partitioned by `identity(id)`, joined with one partitioned by a transform that reorders its key space, where both sides report the key list `[0, 1]`: the join returns 0 rows instead of 2, with no shuffle and no `GroupPartitionsExec` in the plan. I measured that on master and filed SPARK-59688, with the fix in #58943. `KeyedShuffleSpec.isCompatibleWith` there asks `areKeysCompatible` with `allowReduce = false`, so a pair whose keys still need reducing onto one key space is no longer read as lined up, and the join takes the push branch that reduces and regroups.
   
   So there is nothing for you to do here about that P1. Once #58943 lands it falls away on your rebase, and the gate widening in this PR inherits the fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org