Executor links in Job History

[yohann jardin <yo...@hotmail.com>]

Hello,


I'm using Spark 2.1.0 and hadoop 2.2.0.

When I launch jobs on Yarn, I can retrieve their information on Spark History Server, except that the links to stdout/stderr of executors are wrong -> they lead to their url while the job was running.


We have the flag 'yarn.log-aggregation-enable' set to true and once a job is finished on Yarn, its logs are sent to HDFS.


On the client end, when I launch my job i set 'spark.eventLog.enabled' to true, and specify 'spark.eventLog.dir'. I can retrieve the DAG and such afterward on Spark History Server.


I checked http://spark.apache.org/docs/latest/running-on-yarn.html and http://spark.apache.org/docs/latest/monitoring.html

But I do not find what i'm missing to let Spark History Server redirect me to Yarn History Server with a valid link, to see the stdout/stderr logs of the executors.



Any idea?


Regards,

Yohann