[jira] [Created] (SPARK-6868) Container link broken on Spark UI Executors page when YARN is set to HTTPS_ONLY

["Dean Chen (JIRA)" <ji...@apache.org>]

Dean Chen created SPARK-6868:
--------------------------------

             Summary: Container link broken on Spark UI Executors page when YARN is set to HTTPS_ONLY
                 Key: SPARK-6868
                 URL: https://issues.apache.org/jira/browse/SPARK-6868
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.3.0, 1.2.1, 1.2.0, 1.1.1, 1.1.0
            Reporter: Dean Chen


The stdout and stderr log links on the executor page will use the http:// prefix even if the node manager does not support http and only https via setting yarn.http.policy=HTTPS_ONLY.

Unfortunately the unencrypted http link in that case does not return a 404 but a binary file containing random binary chars. This causes a lot of confusion for the end user since it seems like the log file exists and is just filled with garbage. (see attached screenshot)

The fix is to prefix container log links with https:// instead of http:// if yarn.http.policy=HTTPS_ONLY. YARN's job page has this exact logic as seen here: https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java#L108



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org