You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2018/02/01 20:29:00 UTC
Flink REST API
Hi,
I have a triggered a Flink YARN Session on Hadoop yarn.
While I was able to trigger applications and run them. I wish to find the URL of REST API for the Flink YARN Sesssion I launched.
Can someone please help me point out on how to find the REST API Url for the Flink on YARN?
Thanks a lot.
Regards,
Raja.
Re: [EXTERNAL] Re: Flink REST API
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Raja,
How does the HTTP request fail? Can you post the output of:
curl -v http://host:port/jobs
There are a couple of other ways to find out the port number. For example,
when
the YARN session is initially deployed, the url of the web ui is logged to
stdout:
https://github.com/apache/flink/blob/release-1.4/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L691
However, in a HA setup the JobManager may be restarted on a different host
in
case of a failure.
You can also get the port number from the JobManager logs:
https://github.com/apache/flink/blob/release-1.4/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/utils/WebFrontendBootstrap.java#L129
Best,
Gary
On Fri, Feb 9, 2018 at 4:46 PM, Raja.Aravapalli <Ra...@target.com>
wrote:
> I am issuing a GET call to list running jobs on Flink Session.
>
>
>
> Another quick question, is there a way to check the port on which my Flink
> YARN Session is exposing REST API ?
>
>
>
> Because, I could figure out on UI either in YARN Resource Manager / Flink
> Web UI of YARN Session the port number I received from Yarn Client using
> the command *“yarn application -list”*
>
>
>
> Thanks a lot.
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *Gary Yao <ga...@data-artisans.com>
> *Date: *Friday, February 9, 2018 at 9:25 AM
> *To: *Raja Aravapalli <Ra...@target.com>
> *Cc: *"user@flink.apache.org" <us...@flink.apache.org>
> *Subject: *Re: [EXTERNAL] Re: Flink REST API
>
>
>
> Hi Raja,
>
>
>
> Can you tell me the API call that you are trying to issue? If it is not a
> GET
>
> request, it could be that you are suffering from this bug:
>
>
>
> https://issues.apache.org/jira/browse/YARN-2031
>
>
>
> In my case the tracking url shown on the resource manager ui is indeed one
> that
>
> targets the YARN proxy, i.e., POST/DELETE/PUT requests are not supported.
>
> However, the tracking url obtained from the YARN cli directly refers to the
>
> jobmanager.
>
>
>
> Best,
>
>
>
> Gary
>
>
>
> On Fri, Feb 9, 2018 at 3:54 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>
>
> Hi Gary,
>
>
>
> Thanks a lot. I am able to use REST API now.
>
>
>
> As you informed, I am able to query REST API, by capturing the
> tracking-url, I get by using the command *“yarn application -list”*
>
>
>
> But, however as I observe in the YARN Resource manager UI, I am not able
> to query using the tracking url I am observing in YARN Resource Manager.
> Not sure, If the Hadoop environment I am on, is using some proxies or etc.
>
> Do you have any thoughts on why this is happening different in YARN
> command line Vs YARN Resource Manager ?
>
>
>
>
>
> Thanks a lot again.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *Gary Yao <ga...@data-artisans.com>
> *Date: *Friday, February 2, 2018 at 10:20 AM
> *To: *Raja Aravapalli <Ra...@target.com>
> *Cc: *"user@flink.apache.org" <us...@flink.apache.org>
> *Subject: *[EXTERNAL] Re: Flink REST API
>
>
>
> Hi Raja,
>
>
>
> The registered tracking URL of the YARN application can be used to issue
> HTTP
>
> requests against the REST API. You can retrieve the URL by using the YARN
>
> client:
>
>
>
> yarn application -list
>
>
>
> In the output, the rightmost column shows the URL, e.g.,
>
>
>
> Application-Id ... Tracking-URL
>
> application_1517585200464_0001 ...
>
> http://ip-172-31-40-165.eu-central-1.compute.internal:38045
>
>
>
> You can also find the tracking URL in the Web UI of the YARN resource
> manager.
>
>
>
> Best,
>
>
>
> Gary
>
>
>
>
>
>
>
> On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
> Hi,
>
>
>
> I have a triggered a Flink YARN Session on Hadoop yarn.
>
>
>
> While I was able to trigger applications and run them. I wish to find the
> URL of REST API for the Flink YARN Sesssion I launched.
>
>
>
> Can someone please help me point out on how to find the REST API Url for
> the Flink on YARN?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
>
>
Re: [EXTERNAL] Re: Flink REST API
Posted by "Raja.Aravapalli" <Ra...@target.com>.
I am issuing a GET call to list running jobs on Flink Session.
Another quick question, is there a way to check the port on which my Flink YARN Session is exposing REST API ?
Because, I could figure out on UI either in YARN Resource Manager / Flink Web UI of YARN Session the port number I received from Yarn Client using the command “yarn application -list”
Thanks a lot.
Regards,
Raja.
From: Gary Yao <ga...@data-artisans.com>
Date: Friday, February 9, 2018 at 9:25 AM
To: Raja Aravapalli <Ra...@target.com>
Cc: "user@flink.apache.org" <us...@flink.apache.org>
Subject: Re: [EXTERNAL] Re: Flink REST API
Hi Raja,
Can you tell me the API call that you are trying to issue? If it is not a GET
request, it could be that you are suffering from this bug:
https://issues.apache.org/jira/browse/YARN-2031
In my case the tracking url shown on the resource manager ui is indeed one that
targets the YARN proxy, i.e., POST/DELETE/PUT requests are not supported.
However, the tracking url obtained from the YARN cli directly refers to the
jobmanager.
Best,
Gary
On Fri, Feb 9, 2018 at 3:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Gary,
Thanks a lot. I am able to use REST API now.
As you informed, I am able to query REST API, by capturing the tracking-url, I get by using the command “yarn application -list”
But, however as I observe in the YARN Resource manager UI, I am not able to query using the tracking url I am observing in YARN Resource Manager. Not sure, If the Hadoop environment I am on, is using some proxies or etc.
Do you have any thoughts on why this is happening different in YARN command line Vs YARN Resource Manager ?
Thanks a lot again.
Regards,
Raja.
From: Gary Yao <ga...@data-artisans.com>>
Date: Friday, February 2, 2018 at 10:20 AM
To: Raja Aravapalli <Ra...@target.com>>
Cc: "user@flink.apache.org<ma...@flink.apache.org>" <us...@flink.apache.org>>
Subject: [EXTERNAL] Re: Flink REST API
Hi Raja,
The registered tracking URL of the YARN application can be used to issue HTTP
requests against the REST API. You can retrieve the URL by using the YARN
client:
yarn application -list
In the output, the rightmost column shows the URL, e.g.,
Application-Id ... Tracking-URL
application_1517585200464_0001 ...
http://ip-172-31-40-165.eu-central-1.compute.internal:38045
You can also find the tracking URL in the Web UI of the YARN resource manager.
Best,
Gary
On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
I have a triggered a Flink YARN Session on Hadoop yarn.
While I was able to trigger applications and run them. I wish to find the URL of REST API for the Flink YARN Sesssion I launched.
Can someone please help me point out on how to find the REST API Url for the Flink on YARN?
Thanks a lot.
Regards,
Raja.
Re: [EXTERNAL] Re: Flink REST API
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Raja,
Can you tell me the API call that you are trying to issue? If it is not a
GET
request, it could be that you are suffering from this bug:
https://issues.apache.org/jira/browse/YARN-2031
In my case the tracking url shown on the resource manager ui is indeed one
that
targets the YARN proxy, i.e., POST/DELETE/PUT requests are not supported.
However, the tracking url obtained from the YARN cli directly refers to the
jobmanager.
Best,
Gary
On Fri, Feb 9, 2018 at 3:54 PM, Raja.Aravapalli <Ra...@target.com>
wrote:
>
>
> Hi Gary,
>
>
>
> Thanks a lot. I am able to use REST API now.
>
>
>
> As you informed, I am able to query REST API, by capturing the
> tracking-url, I get by using the command *“yarn application -list”*
>
>
>
> But, however as I observe in the YARN Resource manager UI, I am not able
> to query using the tracking url I am observing in YARN Resource Manager.
> Not sure, If the Hadoop environment I am on, is using some proxies or etc.
>
> Do you have any thoughts on why this is happening different in YARN
> command line Vs YARN Resource Manager ?
>
>
>
>
>
> Thanks a lot again.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *Gary Yao <ga...@data-artisans.com>
> *Date: *Friday, February 2, 2018 at 10:20 AM
> *To: *Raja Aravapalli <Ra...@target.com>
> *Cc: *"user@flink.apache.org" <us...@flink.apache.org>
> *Subject: *[EXTERNAL] Re: Flink REST API
>
>
>
> Hi Raja,
>
>
>
> The registered tracking URL of the YARN application can be used to issue
> HTTP
>
> requests against the REST API. You can retrieve the URL by using the YARN
>
> client:
>
>
>
> yarn application -list
>
>
>
> In the output, the rightmost column shows the URL, e.g.,
>
>
>
> Application-Id ... Tracking-URL
>
> application_1517585200464_0001 ...
>
> http://ip-172-31-40-165.eu-central-1.compute.internal:38045
>
>
>
> You can also find the tracking URL in the Web UI of the YARN resource
> manager.
>
>
>
> Best,
>
>
>
> Gary
>
>
>
>
>
>
>
> On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
> Hi,
>
>
>
> I have a triggered a Flink YARN Session on Hadoop yarn.
>
>
>
> While I was able to trigger applications and run them. I wish to find the
> URL of REST API for the Flink YARN Sesssion I launched.
>
>
>
> Can someone please help me point out on how to find the REST API Url for
> the Flink on YARN?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
Re: [EXTERNAL] Re: Flink REST API
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Hi Gary,
Thanks a lot. I am able to use REST API now.
As you informed, I am able to query REST API, by capturing the tracking-url, I get by using the command “yarn application -list”
But, however as I observe in the YARN Resource manager UI, I am not able to query using the tracking url I am observing in YARN Resource Manager. Not sure, If the Hadoop environment I am on, is using some proxies or etc.
Do you have any thoughts on why this is happening different in YARN command line Vs YARN Resource Manager ?
Thanks a lot again.
Regards,
Raja.
From: Gary Yao <ga...@data-artisans.com>
Date: Friday, February 2, 2018 at 10:20 AM
To: Raja Aravapalli <Ra...@target.com>
Cc: "user@flink.apache.org" <us...@flink.apache.org>
Subject: [EXTERNAL] Re: Flink REST API
Hi Raja,
The registered tracking URL of the YARN application can be used to issue HTTP
requests against the REST API. You can retrieve the URL by using the YARN
client:
yarn application -list
In the output, the rightmost column shows the URL, e.g.,
Application-Id ... Tracking-URL
application_1517585200464_0001 ...
http://ip-172-31-40-165.eu-central-1.compute.internal:38045
You can also find the tracking URL in the Web UI of the YARN resource manager.
Best,
Gary
On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
I have a triggered a Flink YARN Session on Hadoop yarn.
While I was able to trigger applications and run them. I wish to find the URL of REST API for the Flink YARN Sesssion I launched.
Can someone please help me point out on how to find the REST API Url for the Flink on YARN?
Thanks a lot.
Regards,
Raja.
Re: Flink REST API
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Raja,
The registered tracking URL of the YARN application can be used to issue
HTTP
requests against the REST API. You can retrieve the URL by using the YARN
client:
yarn application -list
In the output, the rightmost column shows the URL, e.g.,
Application-Id ... Tracking-URL
application_1517585200464_0001 ...
http://ip-172-31-40-165.eu-central-1.compute.internal:38045
You can also find the tracking URL in the Web UI of the YARN resource
manager.
Best,
Gary
On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli <Ra...@target.com>
wrote:
> Hi,
>
>
>
> I have a triggered a Flink YARN Session on Hadoop yarn.
>
>
>
> While I was able to trigger applications and run them. I wish to find the
> URL of REST API for the Flink YARN Sesssion I launched.
>
>
>
> Can someone please help me point out on how to find the REST API Url for
> the Flink on YARN?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>