[jira] [Resolved] (SPARK-8405) Show executor logs on Web UI when Yarn log aggregation is enabled

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-8405.
------------------------------
    Resolution: Not A Problem

> Show executor logs on Web UI when Yarn log aggregation is enabled
> -----------------------------------------------------------------
>
>                 Key: SPARK-8405
>                 URL: https://issues.apache.org/jira/browse/SPARK-8405
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.4.0
>            Reporter: Carson Wang
>         Attachments: SparkLogError.png
>
>
> When running Spark application in Yarn mode and Yarn log aggregation is enabled, customer is not able to view executor logs on the history server Web UI. The only way for customer to view the logs is through the Yarn command "yarn logs -applicationId <appId>".
> An screenshot of the error is attached. When you click an executor’s log link on the Spark history server, you’ll see the error if Yarn log aggregation is enabled. The log URL redirects user to the node manager’s UI. This works if the logs are located on that node. But since log aggregation is enabled, the local logs are deleted once log aggregation is completed. 
> The logs should be available through the web UIs just like other Hadoop components like MapReduce. For security reasons, end users may not be able to log into the nodes and run the yarn logs -applicationId command. The web UIs can be viewable and exposed through the firewall if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org