View executor logs on YARN mode

[lonely Feb <lo...@gmail.com>]

Hi all, i sadly found on YARN mode i cannot view executor logs on YARN web
UI nor on SPARK history web UI. On YARN web UI i can only view AppMaster
logs and on SPARK history web UI i can only view Application metric
information. If i want to see whether a executor is being full GC i can
only use "yarn logs" command. It's very unfriendly. BTW, "yarn logs"
command requires a option of containerID which i could not found on YARN
web UI either. I need to grep it in AppMaster log.

I just wonder how do you handle this situation?

Re: View executor logs on YARN mode

[Brett Meyer <Br...@crowdstrike.com>]

You can view the logs for the particular containers on the YARN UI if you go
to the page for a specific node, and then from the Tools menu on the left,
select Local Logs.  There should be a userlogs directory which will contain
the specific application ids for each job that you run.  Inside the
directories for the applications, you can find the specific container logs.


From:  lonely Feb <lo...@gmail.com>
Date:  Wednesday, January 14, 2015 at 8:07 AM
To:  "user@spark.apache.org" <us...@spark.apache.org>
Subject:  View executor logs on YARN mode

Hi all, i sadly found on YARN mode i cannot view executor logs on YARN web
UI nor on SPARK history web UI. On YARN web UI i can only view AppMaster
logs and on SPARK history web UI i can only view Application metric
information. If i want to see whether a executor is being full GC i can only
use "yarn logs" command. It's very unfriendly. BTW, "yarn logs" command
requires a option of containerID which i could not found on YARN web UI
either. I need to grep it in AppMaster log.

I just wonder how do you handle this situation?