[jira] [Created] (SPARK-30732) BroadcastExchangeExec does not fully honor "spark.broadcast.compress"

["Puneet (Jira)" <ji...@apache.org>]

Puneet created SPARK-30732:
------------------------------

             Summary: BroadcastExchangeExec does not fully honor "spark.broadcast.compress"
                 Key: SPARK-30732
                 URL: https://issues.apache.org/jira/browse/SPARK-30732
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Puneet


Setting {{spark.broadcast.compress}} to false disables compression while sending broadcast variable to executors ([https://spark.apache.org/docs/latest/configuration.html#compression-and-serialization])

However this does not disable compression for any child relations sent by the executors to the driver. 

Setting spark.boradcast.compress to false should disable both sides of the traffic, allowing users to disable compression for the whole broadcast join for example.

[https://github.com/puneetguptanitj/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L89]

^here `executeCollectIterator` calls `getByteArrayRdd` which by default always gets a compressed stream

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org