[jira] [Assigned] (SPARK-15891) Make YARN logs less noisy

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-15891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-15891:
------------------------------------

    Assignee:     (was: Apache Spark)

> Make YARN logs less noisy
> -------------------------
>
>                 Key: SPARK-15891
>                 URL: https://issues.apache.org/jira/browse/SPARK-15891
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 2.0.0
>            Reporter: Marcelo Vanzin
>            Priority: Minor
>
> Spark can generate a lot of logs when running in YARN mode. The problem is already a little bad with normal configuration, but is even worse with dynamic allocation on.
> The first problem is that for every executor Spark launches, it will print the whole command and all the env variables it's setting, even though those are exactly the same for every executor. That's not too bad with a handful of executors, but get annoying pretty soon after that. Dynamic allocation makes that problem worse since there executors constantly being started and then going away.
> Also, there's a lot of logging generated by the dynamic allocation backend code in the YARN module. We should audit those and make sure they all make sense, and whether / how to reduce the amount of logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org