You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Timothy Miller (Jira)" <ji...@apache.org> on 2023/03/13 21:51:00 UTC

[jira] [Created] (SPARK-42776) BroadcastHashJoinExec.requiredChildDistribution called before columnar replacement rules

Timothy Miller created SPARK-42776:
--------------------------------------

             Summary: BroadcastHashJoinExec.requiredChildDistribution called before columnar replacement rules
                 Key: SPARK-42776
                 URL: https://issues.apache.org/jira/browse/SPARK-42776
             Project: Spark
          Issue Type: Bug
          Components: Optimizer
    Affects Versions: 3.3.1
         Environment: I'm prototyping on a Mac, but that's not really relevant.
            Reporter: Timothy Miller


I am trying to replace BroadcastHashJoinExec with a columnar equivalent. However, I noticed that BroadcastHashJoinExec.requiredChildDistribution gets called BEFORE the columnar replacement rules. As a result, the object that gets broadcast is the plain old hashmap created from row data. By the time the columnar replacement rules are applied, it's too late to get Spark to broadcast any other kind of object.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org