[jira] [Resolved] (SPARK-26660) Add warning logs for large taskBinary size

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-26660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-26660.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0

Issue resolved by pull request 23580
[https://github.com/apache/spark/pull/23580]

> Add warning logs for large taskBinary size
> ------------------------------------------
>
>                 Key: SPARK-26660
>                 URL: https://issues.apache.org/jira/browse/SPARK-26660
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.1.0, 2.4.0
>         Environment: Spark2.1
>            Reporter: liupengcheng
>            Assignee: liupengcheng
>            Priority: Minor
>             Fix For: 3.0.0
>
>         Attachments: screenshot-1.png
>
>
> In our production environment, we encountered a OOM problem in ML application. After carefully investigation, it was found that some ML library may generate large ml model, thus causing executor may not able to deserialize it and result in OOM failures.
> !screenshot-1.png!
> In order to facilitate the debuging of memory problem caused by large taskBinary broadcast, we may should add same warning logs for it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org