Completed jobs' logs link not getting redirected to History Server

[Gaurav Chhabra <va...@gmail.com>]

Hello Everyone,


One of my Spark jobs failed. I tried checking the logs in "YARN >
ResourceManager Web UI > FAILED > Application ID" and then when i click on
the Application Masters *logs* link, i notice that it's not redirecting me
to the History Server URL.

When i manually changed the URL from:

https://<worker-node>:8044/node/containerlogs/<container-id>/<user>
to
https://
<JHS_HOST>:19890/jobhistory/logs/<worker-node>:8041/<container-id>/<container-id>/<user>


I see that the link works.

Note: *Spark History Server* and *Map-Reduce Job History Server* (JHS) are
both running on the same node i.e., <JHS_HOST>

I have another cluster where the redirection is working properly. I
searched online and checked several posts, and based on those suggestions,
compared all those settings in both the clusters but could not find
anything that was out of order. Following were the properties that i
compared:

---------------------------------------------------------------------------------
a) *yarn-site.xml*

yarn.log-aggregation-enable = true

yarn.nodemanager.address = <worker-node>:8041

yarn.nodemanager.webapp.https.address = <worker-node>:8044

yarn.log.server.url = https://<JHS_HOST>:19890/jobhistory/logs/

b) *mapred-site.xml *

mapreduce.jobhistory.webapp.https.address = <JHS_HOST>:19890

c) *spark-defaults.conf*

spark.yarn.historyServer.adddress is set to http://<JHS_HOST>:18089
---------------------------------------------------------------------------------

I read in the below link that it's actually the NodeManager http server
that does the job of redirection. Is there anything I can check in this
regard? Is there some specific service that should be running? I would also
like to point out that this worker node was recently added to the cluster
(old worker node has been decommissioned although the node is still up so i
can check any configuration, if required) and the *logs* link never worked
on this new worker-node.

http://spark.apache.org/docs/latest/running-on-yarn.html#debugging-your-application

I will really appreciate any pointers.


Regards.