[jira] [Assigned] (SPARK-26792) Apply custom log URL to Spark UI

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-26792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcelo Vanzin reassigned SPARK-26792:
--------------------------------------

    Assignee: Jungtaek Lim

> Apply custom log URL to Spark UI
> --------------------------------
>
>                 Key: SPARK-26792
>                 URL: https://issues.apache.org/jira/browse/SPARK-26792
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>
> SPARK-23155 enables SHS to set up custom log URLs for incompleted / completed apps.
> While getting reviews from SPARK-23155, I've got two comments which applying custom log URLs to UI would help achieving it. Quoting these comments here:
> https://github.com/apache/spark/pull/23260#issuecomment-456827963
> {quote}
> Sorry I haven't had time to look through all the code so this might be a separate jira, but one thing I thought of here is it would be really nice not to have specifically stderr/stdout. users can specify any log4j.properties and some tools like oozie by default end up using hadoop log4j rather then spark log4j, so files aren't necessarily the same. Also users can put in other logs files so it would be nice to have links to those from the UI. It seems simpler if we just had a link to the directory and it read the files within there. Other things in Hadoop do it this way, but I'm not sure if that works well for other resource managers, any thoughts on that? As long as this doesn't prevent the above I can file a separate jira for it.
> {quote}
> https://github.com/apache/spark/pull/23260#issuecomment-456904716
> {quote}
> Hi Tom, +1: singling out stdout and stderr is definitely an annoyance. We
> typically configure Spark jobs to write the GC log and dump heap on OOM
> using <LOG_DIR>,  and/or we use the rolling file appender to deal with
> large logs during debugging. So linking the YARN container log overview
> page would make much more sense for us. We work it around with a custom
> submit process that logs all important URLs on the submit side log.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org