You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2017/10/09 22:24:00 UTC
[jira] [Updated] (SPARK-22170) Broadcast join holds an extra copy
of rows in driver memory
[ https://issues.apache.org/jira/browse/SPARK-22170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-22170:
----------------------------
Issue Type: Improvement (was: Bug)
> Broadcast join holds an extra copy of rows in driver memory
> -----------------------------------------------------------
>
> Key: SPARK-22170
> URL: https://issues.apache.org/jira/browse/SPARK-22170
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.0.2, 2.1.1, 2.2.0
> Reporter: Ryan Blue
> Assignee: Ryan Blue
> Fix For: 2.3.0
>
>
> I investigated a driver OOM that was building a large broadcast table with a memory profiler and found that a huge amount of memory is used while building a broadcast table. This is because [BroadcastExchangeExec uses {{executeCollect}}|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L76]. In {{executeCollect}}, all of the partitions are fetched as compressed blocks, then each block is decompressed (with a stream), and each row is copied to a new byte buffer and added to an ArrayBuffer, which is copied to an Array. This results in a huge amount of allocation: a buffer for each row in the broadcast. Those rows are only used to get copied into a {{BytesToBytesMap}} that will be broadcasted, so there is no need to keep them in memory.
> Replacing the array buffer step with an iterator reduces the amount of memory held while creating the map by not requiring all rows to be in memory. It also avoids allocating a large Array for the rows. In practice, a 16MB broadcast table used 100MB less memory with this approach, but the reduction depends on the size of rows and compression (16MB was in Parquet format).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org