You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vasily Melnik <Va...@glowbyteconsulting.com> on 2020/01/23 12:23:29 UTC
REST rescale with Flink on YARN
Hi all.
I'm using Flink 1.8 on YARN with CDH 5.12
When i try to perform rescale request:
curl -v -X PATCH
'<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3
<https://slack-redir.net/link?url=http%3A%2F%2Fstrm-flink-tm-dev-1.dwh.m1.tinkoff.cloud%3A8088%2Fproxy%2Fapplication_1576854986116_0079%2Fjobs%2F11dcfc3163936fc019e049fc841b075b%2Frescaling%3Fparallelism%3D3>'
i get a mistake:
*Method PATCH is not defined in RFC 2068 and is not supported by the
Servlet API*GET and POST methods work well.
The Server type in response is Jetty(6.1.26.cloudera.4).
How can i deal with this situation?
С уважением,
Василий Мельник
Re: REST rescale with Flink on YARN
Posted by Yang Wang <da...@gmail.com>.
Gary is right. You could also access the Yarn RM rest api to get the AM
original address.
http://hadoop.apache.org/docs/r2.8.5/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_API
Best,
Yang
Gary Yao <ga...@ververica.com> 于2020年1月28日周二 下午6:17写道：
> Hi,
>
> You can use
>
> yarn application -status <APPLICATION_ID>
>
> to find the host and port that the server is listening on (AM host & RPC
> Port). If you need to access that information programmatically, take a
> look at
> the YarnClient [1].
>
> Best,
> Gary
>
>
> [1]
> https://hadoop.apache.org/docs/r2.8.5/api/org/apache/hadoop/yarn/client/api/YarnClient.html
>
> On Thu, Jan 23, 2020 at 3:21 PM Vasily Melnik <
> Vasily.Melnik@glowbyteconsulting.com> wrote:
>
>> Hi all,
>> I've found some solution for this issue.
>> Problem is that with YARN ApplicationMaster URL we communicate with
>> JobManager via proxy which is implemented on Jetty 6 (for Hadoop 2.6).
>> So to use PATCH method we need to locate original JobManager URL.
>> Using /jobmanager/config API we could get only host, but web.port is
>> displayed as 0 (???)
>> To find actual web port, we should parse YARN logs for jobmanager, where
>> we can find something like this:
>>
>> *INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint -
>> Rest endpoint listening at <host>:<port>.*
>>
>> Maybe someone knows less complicated way to find actual REST URL under
>> YARN?
>>
>>
>>
>>
>> С уважением,
>> Василий Мельник
>>
>>
>> On Thu, 23 Jan 2020 at 15:32, Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> Older versions of Jetty don't support PATCH requests. You will either
>>> have to update it or create a custom Flink version that uses POST for the
>>> rescale operation.
>>>
>>> On 23/01/2020 13:23, Vasily Melnik wrote:
>>>
>>> Hi all.
>>> I'm using Flink 1.8 on YARN with CDH 5.12
>>> When i try to perform rescale request:
>>>
>>> curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3 <https://slack-redir.net/link?url=http%3A%2F%2Fstrm-flink-tm-dev-1.dwh.m1.tinkoff.cloud%3A8088%2Fproxy%2Fapplication_1576854986116_0079%2Fjobs%2F11dcfc3163936fc019e049fc841b075b%2Frescaling%3Fparallelism%3D3>'
>>>
>>> i get a mistake:
>>>
>>> *Method PATCH is not defined in RFC 2068 and is not supported by the
>>> Servlet API *GET and POST methods work well.
>>> The Server type in response is Jetty(6.1.26.cloudera.4).
>>>
>>> How can i deal with this situation?
>>>
>>> С уважением,
>>> Василий Мельник
>>>
>>>
>>>
Re: REST rescale with Flink on YARN
Posted by Gary Yao <ga...@ververica.com>.
Hi,
You can use
yarn application -status <APPLICATION_ID>
to find the host and port that the server is listening on (AM host & RPC
Port). If you need to access that information programmatically, take a look
at
the YarnClient [1].
Best,
Gary
[1]
https://hadoop.apache.org/docs/r2.8.5/api/org/apache/hadoop/yarn/client/api/YarnClient.html
On Thu, Jan 23, 2020 at 3:21 PM Vasily Melnik <
Vasily.Melnik@glowbyteconsulting.com> wrote:
> Hi all,
> I've found some solution for this issue.
> Problem is that with YARN ApplicationMaster URL we communicate with
> JobManager via proxy which is implemented on Jetty 6 (for Hadoop 2.6).
> So to use PATCH method we need to locate original JobManager URL.
> Using /jobmanager/config API we could get only host, but web.port is
> displayed as 0 (???)
> To find actual web port, we should parse YARN logs for jobmanager, where
> we can find something like this:
>
> *INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest
> endpoint listening at <host>:<port>.*
>
> Maybe someone knows less complicated way to find actual REST URL under
> YARN?
>
>
>
>
> С уважением,
> Василий Мельник
>
>
> On Thu, 23 Jan 2020 at 15:32, Chesnay Schepler <ch...@apache.org> wrote:
>
>> Older versions of Jetty don't support PATCH requests. You will either
>> have to update it or create a custom Flink version that uses POST for the
>> rescale operation.
>>
>> On 23/01/2020 13:23, Vasily Melnik wrote:
>>
>> Hi all.
>> I'm using Flink 1.8 on YARN with CDH 5.12
>> When i try to perform rescale request:
>>
>> curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3 <https://slack-redir.net/link?url=http%3A%2F%2Fstrm-flink-tm-dev-1.dwh.m1.tinkoff.cloud%3A8088%2Fproxy%2Fapplication_1576854986116_0079%2Fjobs%2F11dcfc3163936fc019e049fc841b075b%2Frescaling%3Fparallelism%3D3>'
>>
>> i get a mistake:
>>
>> *Method PATCH is not defined in RFC 2068 and is not supported by the
>> Servlet API *GET and POST methods work well.
>> The Server type in response is Jetty(6.1.26.cloudera.4).
>>
>> How can i deal with this situation?
>>
>> С уважением,
>> Василий Мельник
>>
>>
>>
Re: REST rescale with Flink on YARN
Posted by Vasily Melnik <Va...@glowbyteconsulting.com>.
Hi all,
I've found some solution for this issue.
Problem is that with YARN ApplicationMaster URL we communicate with
JobManager via proxy which is implemented on Jetty 6 (for Hadoop 2.6).
So to use PATCH method we need to locate original JobManager URL.
Using /jobmanager/config API we could get only host, but web.port is
displayed as 0 (???)
To find actual web port, we should parse YARN logs for jobmanager, where we
can find something like this:
*INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest
endpoint listening at <host>:<port>.*
Maybe someone knows less complicated way to find actual REST URL under
YARN?
С уважением,
Василий Мельник
On Thu, 23 Jan 2020 at 15:32, Chesnay Schepler <ch...@apache.org> wrote:
> Older versions of Jetty don't support PATCH requests. You will either have
> to update it or create a custom Flink version that uses POST for the
> rescale operation.
>
> On 23/01/2020 13:23, Vasily Melnik wrote:
>
> Hi all.
> I'm using Flink 1.8 on YARN with CDH 5.12
> When i try to perform rescale request:
>
> curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3 <https://slack-redir.net/link?url=http%3A%2F%2Fstrm-flink-tm-dev-1.dwh.m1.tinkoff.cloud%3A8088%2Fproxy%2Fapplication_1576854986116_0079%2Fjobs%2F11dcfc3163936fc019e049fc841b075b%2Frescaling%3Fparallelism%3D3>'
>
> i get a mistake:
>
> *Method PATCH is not defined in RFC 2068 and is not supported by the
> Servlet API *GET and POST methods work well.
> The Server type in response is Jetty(6.1.26.cloudera.4).
>
> How can i deal with this situation?
>
> С уважением,
> Василий Мельник
>
>
>
Re: REST rescale with Flink on YARN
Posted by Chesnay Schepler <ch...@apache.org>.
Older versions of Jetty don't support PATCH requests. You will either
have to update it or create a custom Flink version that uses POST for
the rescale operation.
On 23/01/2020 13:23, Vasily Melnik wrote:
> Hi all.
> I'm using Flink 1.8 on YARN with CDH 5.12
> When i try to perform rescale request:
> curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3 <https://slack-redir.net/link?url=http%3A%2F%2Fstrm-flink-tm-dev-1.dwh.m1.tinkoff.cloud%3A8088%2Fproxy%2Fapplication_1576854986116_0079%2Fjobs%2F11dcfc3163936fc019e049fc841b075b%2Frescaling%3Fparallelism%3D3>'
> i get a mistake:
> /Method PATCH is not defined in RFC 2068 and is not supported by the
> Servlet API
> /GET and POST methods work well.
> The Server type in response is Jetty(6.1.26.cloudera.4).
>
> How can i deal with this situation?
>
> С уважением,
> Василий Мельник
>