You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2016/09/06 22:56:21 UTC

[jira] [Resolved] (SPARK-15891) Make YARN logs less noisy

     [ https://issues.apache.org/jira/browse/SPARK-15891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcelo Vanzin resolved SPARK-15891.
------------------------------------
       Resolution: Fixed
         Assignee: Marcelo Vanzin
    Fix Version/s: 2.1.0

> Make YARN logs less noisy
> -------------------------
>
>                 Key: SPARK-15891
>                 URL: https://issues.apache.org/jira/browse/SPARK-15891
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 2.0.0
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>             Fix For: 2.1.0
>
>
> Spark can generate a lot of logs when running in YARN mode. The problem is already a little bad with normal configuration, but is even worse with dynamic allocation on.
> The first problem is that for every executor Spark launches, it will print the whole command and all the env variables it's setting, even though those are exactly the same for every executor. That's not too bad with a handful of executors, but get annoying pretty soon after that. Dynamic allocation makes that problem worse since there executors constantly being started and then going away.
> Also, there's a lot of logging generated by the dynamic allocation backend code in the YARN module. We should audit those and make sure they all make sense, and whether / how to reduce the amount of logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org