You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Vladimir Tretyakov <vl...@sematext.com> on 2016/10/06 12:40:30 UTC
Spark REST API YARN client mode is not full?
Hi,
When I start Spark v1.6 (cdh5.8.0) in Yarn client mode I see that 4040
port is avaiable, but UI shows nothing and API returns not full information.
I started Spark application like this:
spark-submit --master yarn-client --class
org.apache.spark.examples.SparkPi
/usr/lib/spark/examples/lib/spark-examples-1.6.0-cdh5.8.0-hadoop2.6.0-cdh5.8.0.jar
10000
API returns me:
http://localhost:4040/api/v1/applications
[ {
"name" : "Spark Pi",
"attempts" : [ {
"startTime" : "2016-10-05T11:27:54.558GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"sparkUser" : "",
"completed" : false
} ]
} ]
Where is application id? How I can get more detailed information about
application without this id (I am talking about /applications/[app-id]/jobs,
/applications/[app-id]/stages etc urls from
http://spark.apache.org/docs/1.6.0/monitoring.html)?
UI also shows me empty pages.
Without appId we cannot use other REST API calls. Is there any other way to
get RUNNING application ids?
Please help me understand what's going on.
Re: Spark REST API YARN client mode is not full?
Posted by Vladimir Tretyakov <vl...@sematext.com>.
Thx for answer Vadim.
Started application as:
spark-submit --master yarn-client --class
org.apache.spark.examples.SparkPi
/usr/lib/spark/examples/lib/spark-examples-1.6.0-cdh5.8.0-hadoop2.6.0-cdh5.8.0.jar
30000
Performed few requests:
curl http://localhost:4040/api/v1/applications
<html>
<head>
<title>
Moved
</title>
</head>
<body>
<h1>
Moved
</h1>
<div>
Content has moved
<a href="
http://bfdad119e56a:8088/proxy/application_1475780604681_0011/api/v1/applications
">here</a>
</div>
</body>
</html>
curl
http://bfdad119e56a:8088/proxy/application_1475780604681_0013/api/v1/applications/
[ {
"id" : "application_1475780604681_0013",
"name" : "Spark Pi",
"attempts" : [ {
"startTime" : "2016-10-07T10:07:40.194GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"sparkUser" : "",
"completed" : false
} ]
} ]
Looks like REST API is available thru YARN proxy in my case, does this
related to CDH you think?
Do you say all are binding to 4040 in your case? Can you please share what
version of Spark you use? How do you start your application (YARN cluster
or client mode)?
On Thu, Oct 6, 2016 at 8:14 PM, Vadim Semenov <va...@datadoghq.com>
wrote:
> It may be related to the CDH version of spark you're using.
> When I use REST API I get yarn application id there
>
> Try opening http://localhost:4040/api/v1/applications/0/stages
>
> On Thu, Oct 6, 2016 at 8:40 AM, Vladimir Tretyakov <
> vladimir.tretyakov@sematext.com> wrote:
>
>> Hi,
>>
>> When I start Spark v1.6 (cdh5.8.0) in Yarn client mode I see that 4040
>> port is avaiable, but UI shows nothing and API returns not full
>> information.
>>
>> I started Spark application like this:
>>
>> spark-submit --master yarn-client --class
>> org.apache.spark.examples.SparkPi
>> /usr/lib/spark/examples/lib/spark-examples-1.6.0-cdh5.8.0-ha
>> doop2.6.0-cdh5.8.0.jar
>> 10000
>>
>> API returns me:
>>
>> http://localhost:4040/api/v1/applications
>>
>> [ {
>> "name" : "Spark Pi",
>> "attempts" : [ {
>> "startTime" : "2016-10-05T11:27:54.558GMT",
>> "endTime" : "1969-12-31T23:59:59.999GMT",
>> "sparkUser" : "",
>> "completed" : false
>> } ]
>> } ]
>>
>> Where is application id? How I can get more detailed information about
>> application without this id (I am talking about
>> /applications/[app-id]/jobs,
>> /applications/[app-id]/stages etc urls from
>> http://spark.apache.org/docs/1.6.0/monitoring.html)?
>>
>> UI also shows me empty pages.
>>
>> Without appId we cannot use other REST API calls. Is there any other way
>> to get RUNNING application ids?
>>
>> Please help me understand what's going on.
>>
>
>
Re: Spark REST API YARN client mode is not full?
Posted by Vadim Semenov <va...@datadoghq.com>.
It may be related to the CDH version of spark you're using.
When I use REST API I get yarn application id there
Try opening http://localhost:4040/api/v1/applications/0/stages
On Thu, Oct 6, 2016 at 8:40 AM, Vladimir Tretyakov <
vladimir.tretyakov@sematext.com> wrote:
> Hi,
>
> When I start Spark v1.6 (cdh5.8.0) in Yarn client mode I see that 4040
> port is avaiable, but UI shows nothing and API returns not full
> information.
>
> I started Spark application like this:
>
> spark-submit --master yarn-client --class
> org.apache.spark.examples.SparkPi
> /usr/lib/spark/examples/lib/spark-examples-1.6.0-cdh5.8.0-
> hadoop2.6.0-cdh5.8.0.jar
> 10000
>
> API returns me:
>
> http://localhost:4040/api/v1/applications
>
> [ {
> "name" : "Spark Pi",
> "attempts" : [ {
> "startTime" : "2016-10-05T11:27:54.558GMT",
> "endTime" : "1969-12-31T23:59:59.999GMT",
> "sparkUser" : "",
> "completed" : false
> } ]
> } ]
>
> Where is application id? How I can get more detailed information about
> application without this id (I am talking about
> /applications/[app-id]/jobs,
> /applications/[app-id]/stages etc urls from
> http://spark.apache.org/docs/1.6.0/monitoring.html)?
>
> UI also shows me empty pages.
>
> Without appId we cannot use other REST API calls. Is there any other way
> to get RUNNING application ids?
>
> Please help me understand what's going on.
>