trouble accessing driver log files using rest-api

[Andy Davidson <An...@SantaCruzIntegration.com>]

I am running spark-1.6.1 and the stand alone cluster manager. I am running
into performance problems with spark streaming and added some extra metrics
to my log files. I submit my app in cluster mode. (I.e. The driver runs on a
slave not master)


I am not able to get the driver log files while the app is running using the
documented rest api
 <http://spark.apache.org/docs/latest/monitoring.html#rest-api>
http://spark.apache.org/docs/latest/monitoring.html#rest-api

I think the issue is the rest-api give you access to the app log files. I
need the driver log file?


$ curl  http://$host/api/v1/applications/

[ {

  "id" : "app-20160711185337-0049",

  "name" : "gnip1",

  "attempts" : [ {

    "startTime" : "2016-07-11T18:53:35.318GMT",

    "endTime" : "1969-12-31T23:59:59.999GMT",

    "sparkUser" : "",

    "completed" : false

  } ]

} ][ec2-user@ip-172-31-22-140 tmp]$



$ curl -o$outputFile http://$host/api/v1/applications/$appID/logs



$outputFile will always be an empty zip file



If I use executors/. I get info about the drivers and executors how ever no
way to Œget' the log files. The driver output does not have any executorLogs
and the workers executorLogs are version of the log files rendered in HTML
not the actual log file.




$ curl http://$host/api/v1/applications/$appID/executors [ { "id" :
"driver", "hostPort" : "172.31.23.203:33303", "rddBlocks" : 0, "memoryUsed"
: 0, "diskUsed" : 0, "activeTasks" : 0, "failedTasks" : 0, "completedTasks"
: 0, "totalTasks" : 0, "totalDuration" : 0, "totalInputBytes" : 0,
"totalShuffleRead" : 0, "totalShuffleWrite" : 0, "maxMemory" : 535953408,
"executorLogs" : { } }, { "id" : "1", "hostPort" :
"ip-172-31-23-200.us-west-1.compute.internal:51560", "rddBlocks" : 218,
"memoryUsed" : 452224280, "diskUsed" : 0, "activeTasks" : 1, "failedTasks" :
0, "completedTasks" : 27756, "totalTasks" : 27757, "totalDuration" :
1650935, "totalInputBytes" : 9619224986, "totalShuffleRead" : 0,
"totalShuffleWrite" : 507615, "maxMemory" : 535953408, "executorLogs" : {
"stdout" : 
"http://ec2-xxx.compute.amazonaws.com:8081/logPage/?appId=app-20160711185337
-0049&executorId=1&logType=stdout", "stderr" :
"http://ec2-xxx.us-west-1.compute.amazonaws.com:8081/logPage/?appId=app-2016
0711185337-0049&executorId=1&logType=stderr" }

Any suggestions would be greatly appreciated

Andy