You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "liupengcheng (JIRA)" <ji...@apache.org> on 2019/01/18 03:13:00 UTC
[jira] [Updated] (SPARK-26660) Add warning logs for large
taskBinary size
[ https://issues.apache.org/jira/browse/SPARK-26660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liupengcheng updated SPARK-26660:
---------------------------------
Attachment: screenshot-1.png
> Add warning logs for large taskBinary size
> ------------------------------------------
>
> Key: SPARK-26660
> URL: https://issues.apache.org/jira/browse/SPARK-26660
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0, 2.4.0
> Environment: Spark2.1
> Reporter: liupengcheng
> Priority: Minor
> Attachments: screenshot-1.png
>
>
> In our production environment, we encountered a OOM problem in ML application. After carefully investigation, it was found that some ML library may generate large ml model, thus causing executor may not able to deserialize it and result in OOM failures.
> !screenshot-1.png!
> In order to facilitate the debuging of memory problem caused by large taskBinary broadcast, we may should add same warning logs for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org