You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Violet Vin <vi...@gmail.com> on 2022/10/28 08:04:45 UTC
How to find final status (Driver's) for an application
Hi Everyone,
I am setting up Spark 3.3 in standalone mode (
https://spark.apache.org/docs/latest/spark-standalone.html) and submitting
spark jobs programatically.
I do have java client program that polls Job status for the jobs submitted
programatically, for which i am using following REST endpoint. curl
http://192.168.1.139:8080/json/ which provide JSON response as following,
{
"url" : "spark://192.168.1.139:7077",
"workers" : [ { "id" : "x", "host" : "x", "port" : x, "webuiaddress" :
"x",
"cores" : x, "coresused" : x, "coresfree" : x, "memory"
: xx,
"memoryused" : xx, "memoryfree" : xx, "state" : "x",
"lastheartbeat" : x
}, { ...}, ],
"cores" : x,
"coresused" : x,
"memory" : x,
"memoryused" : x,
"activeapps" : [ ],
"completedapps" : [ { "starttime" : x, "id" : "app-xx-xxxx", "name" :
"abc", "user" : "xx",
"memoryperslave" : x, "submitdate" : "x","state" :
"FINISHED OR RUNNING", "duration" : x
}, {...}],
"activedrivers" : [ ],
"status" : "x"
}
In above response, I have observed state for completedapps is always
FINISHED even if application fails, while on UI (http://master:8080),
associated driver shows a failed state, as captured below.
Referring to above example, Currently, My java client gets status for
application (app-20190925115750-0003) FINISHED, even though it got failed
(encountered exception) and associated driver shows "FAILED" state. I
intend to show the final status in this case as FAILED.
It seems if i can co-relate, an application-id (app-20190925115750-0003) to
driver-id (driver-20190925115748-0003), I can report a "FAILED" (final)
status. I could not find any co-relation between them (appID --> driver ID).
Looking forward to your suggestions to resolving this or any possible
approaches to achieve this.
Regards,
Violet
Re: How to find final status (Driver's) for an application
Posted by Artemis User <ar...@dtechspace.com>.
The master UI doesn't return much details, not designed for this
purpose. You need to use the application-level/driver UI instead (on
port 4040/4041...). Please see online doc monitoring and
instrumentation for details
(https://spark.apache.org/docs/latest/monitoring.html#rest-api).
On 10/28/22 4:04 AM, Violet Vin wrote:
> Hi Everyone,
>
> I am setting up Spark 3.3 in standalone mode
> (https://spark.apache.org/docs/latest/spark-standalone.html) and
> submitting spark jobs programatically.
>
> I do have java client program that polls Job status for the jobs
> submitted programatically, for which i am using following REST
> endpoint. curl http://192.168.1.139:8080/json/ which provide JSON
> response as following,
>
> {
> "url" : "spark://192.168.1.139:7077 <http://192.168.1.139:7077>",
> "workers" : [ { "id" : "x", "host" : "x", "port" : x, "webuiaddress"
> : "x",
> "cores" : x, "coresused" : x, "coresfree" : x,
> "memory" : xx,
> "memoryused" : xx, "memoryfree" : xx, "state" :
> "x", "lastheartbeat" : x
> }, { ...}, ],
> "cores" : x,
> "coresused" : x,
> "memory" : x,
> "memoryused" : x,
> "activeapps" : [ ],
> "completedapps" : [ { "starttime" : x, "id" : "app-xx-xxxx", "name"
> : "abc", "user" : "xx",
> "memoryperslave" : x, "submitdate" :
> "x","state" : "FINISHED OR RUNNING", "duration" : x
> }, {...}],
> "activedrivers" : [ ],
> "status" : "x"
> }
>
> In above response, I have observed state for completedapps is always
> FINISHED even if application fails, while on UI (http://master:8080),
> associated driver shows a failed state, as captured below.
>
> Referring to above example, Currently, My java client gets status for
> application (app-20190925115750-0003) FINISHED, even though it got
> failed (encountered exception) and associated driver shows "FAILED"
> state. I intend to show the final status in this case as FAILED.
>
> It seems if i can co-relate, an application-id
> (app-20190925115750-0003) to driver-id (driver-20190925115748-0003), I
> can report a "FAILED" (final) status. I could not find any co-relation
> between them (appID --> driver ID).
>
> Looking forward to your suggestions to resolving this or any possible
> approaches to achieve this.
>
> Regards,
> Violet