[jira] [Created] (SPARK-26792) Apply custom log URL to Spark UI

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

Jungtaek Lim created SPARK-26792:
------------------------------------

             Summary: Apply custom log URL to Spark UI
                 Key: SPARK-26792
                 URL: https://issues.apache.org/jira/browse/SPARK-26792
             Project: Spark
          Issue Type: Improvement
          Components: Web UI
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


SPARK-23155 enables SHS to set up custom log URLs for incompleted / completed apps.

While getting reviews from SPARK-23155, I've got two comments which applying custom log URLs to UI would help achieving it. Quoting these comments here:

https://github.com/apache/spark/pull/23260#issuecomment-456827963

{quote}
Sorry I haven't had time to look through all the code so this might be a separate jira, but one thing I thought of here is it would be really nice not to have specifically stderr/stdout. users can specify any log4j.properties and some tools like oozie by default end up using hadoop log4j rather then spark log4j, so files aren't necessarily the same. Also users can put in other logs files so it would be nice to have links to those from the UI. It seems simpler if we just had a link to the directory and it read the files within there. Other things in Hadoop do it this way, but I'm not sure if that works well for other resource managers, any thoughts on that? As long as this doesn't prevent the above I can file a separate jira for it.
{quote}

https://github.com/apache/spark/pull/23260#issuecomment-456904716

{quote}
Hi Tom, +1: singling out stdout and stderr is definitely an annoyance. We
typically configure Spark jobs to write the GC log and dump heap on OOM
using <LOG_DIR>,  and/or we use the rolling file appender to deal with
large logs during debugging. So linking the YARN container log overview
page would make much more sense for us. We work it around with a custom
submit process that logs all important URLs on the submit side log.
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org