You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Juho Autio <ju...@rovio.com> on 2018/05/30 07:07:47 UTC
Re: REST API "broken" on YARN because POST is not allowed via YARN proxy
Hi, I tried to search Flink Jira for this but couldn't find a ticket to
match. If there's no ticket yet, did I understand correctly though, that
you would be open to support GET as an alternative method for all of
Flink's REST operations?
Now that 1.5 was released, it's a pity that this problem kind of prevents
us from upgrading – or spend time on creating a workaround.
On Thu, Apr 5, 2018 at 9:10 PM, Till Rohrmann <tr...@apache.org> wrote:
> This improvement is unfortunately out of scope for the 1.5 release since
> the feature freeze is already quite some time ago. But I hope that this
> improvement will make it into the 1.6 release.
>
> Cheers,
> Till
>
> On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <ju...@rovio.com> wrote:
>
>> Thanks for the answer. Wrapping with GET sounds good to me. You said next
>> version; do you mean that Flink 1.5 would already include this improvement
>> when it's released?
>>
>> On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi Juho,
>>>
>>> you are right that due to a limitation in the Yarn proxy [1] we cannot
>>> directly contact the cluster through the Yarn proxy.
>>>
>>> The way it works at the moment is that the Flink client retrieves the
>>> AM's hostname through the ApplicationReport and then directly talks to the
>>> AM. This of course requires that one can reach the respective Yarn
>>> container. This is a limitation, though not a regression, which we want to
>>> improve with the next version of Flink. An idea would be to wrap the REST
>>> calls in a GET call to make them pass through the Yarn proxy.
>>>
>>> [1] https://issues.apache.org/jira/browse/YARN-2084
>>>
>>> Cheers,
>>> Till
>>>
>>> On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>>
>>>> Hi Juho,
>>>>
>>>> Thanks for raising this point!
>>>>
>>>> I'll add Chesnay and Till to the thread who contributed to the REST API.
>>>>
>>>> Best, Fabian
>>>>
>>>> 2018-04-04 15:02 GMT+02:00 Juho Autio <ju...@rovio.com>:
>>>>
>>>>> I just learned that Flink savepoints API was refactored to require
>>>>> using HTTP POST.
>>>>>
>>>>> That's fine otherwise, but makes life harder when Flink is run on top
>>>>> of YARN.
>>>>>
>>>>> I've added example calls below to show how POST is declined by
>>>>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.
>>>>>
>>>>> Can you think of any solution to this? If I would be able to determine
>>>>> the actual host & port for Flink UI, I could use that instead of the proxy
>>>>> address. But that would probably require opening at least one more port, so
>>>>> it's not the optimal solution either. Ideally I would have Flink REST API
>>>>> completely accessible with GET and PUT methods.
>>>>>
>>>>> To me it seems like AWS EMR will also hit this issue as soon as they
>>>>> start supporting Flink 1.5, because they seem to run Flink as a YARN app.
>>>>>
>>>>>
>>>>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p
>>>>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web
>>>>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/
>>>>> WebAppProxyServlet.java#L296-L306
>>>>>
>>>>>
>>>>> $ http POST http://10.0.10.71:20888/proxy/
>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23
>>>>> d4f8b908/checkpoints
>>>>>
>>>>> HTTP/1.1 405 HTTP method POST is not supported by this URL
>>>>> Cache-Control: must-revalidate,no-cache,no-store
>>>>> Content-Length: 1523
>>>>> Content-Type: text/html; charset=iso-8859-1
>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>>> Pragma: no-cache
>>>>> Pragma: no-cache
>>>>>
>>>>> <html>
>>>>> <head>
>>>>> <meta http-equiv="Content-Type" content="text/html;
>>>>> charset=ISO-8859-1"/>
>>>>> <title>Error 405 HTTP method POST is not supported by this URL</title>
>>>>> </head>
>>>>> <body><h2>HTTP ERROR 405</h2>
>>>>> <p>Problem accessing /proxy/application_15228441533
>>>>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
>>>>> <pre> HTTP method POST is not supported by this URL</pre></p><hr
>>>>> /><i><small>Powered by Jetty://</small></i><br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>> <br/>
>>>>>
>>>>> </body>
>>>>> </html>
>>>>>
>>>>>
>>>>> $ http PUT http://10.0.10.71:20888/proxy/
>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23
>>>>> d4f8b908/checkpoints
>>>>>
>>>>> HTTP/1.1 404 Not Found
>>>>> Access-Control-Allow-Origin: *
>>>>> Cache-Control: no-cache
>>>>> Content-Length: 25
>>>>> Content-Type: application/json; charset=UTF-8
>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>>> Pragma: no-cache
>>>>> Pragma: no-cache
>>>>>
>>>>> {
>>>>> "errors": [
>>>>> "Not found."
>>>>> ]
>>>>> }
>>>>>
>>>>> ^ expected to get this from flink because there's no PUT /checkpoints.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: REST API "broken" on YARN because POST is not allowed via YARN proxy
Posted by Juho Autio <ju...@rovio.com>.
Thanks, Till!
On Wed, May 30, 2018 at 10:39 AM, Till Rohrmann <tr...@apache.org>
wrote:
> Hi Juho,
>
> I created a JIRA issue for the problem [1]. Yes, the solution would be to
> use only GET requests. Either we wrap our requests in a GET request or we
> change our handlers to accept GET requests only. One thing which we have to
> fix first is that also the jar file upload goes through REST.
>
> [1] https://issues.apache.org/jira/browse/FLINK-9478
>
> Cheers,
> Till
>
> On Wed, May 30, 2018 at 9:07 AM, Juho Autio <ju...@rovio.com> wrote:
>
>> Hi, I tried to search Flink Jira for this but couldn't find a ticket to
>> match. If there's no ticket yet, did I understand correctly though, that
>> you would be open to support GET as an alternative method for all of
>> Flink's REST operations?
>>
>> Now that 1.5 was released, it's a pity that this problem kind of prevents
>> us from upgrading – or spend time on creating a workaround.
>>
>>
>> On Thu, Apr 5, 2018 at 9:10 PM, Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> This improvement is unfortunately out of scope for the 1.5 release since
>>> the feature freeze is already quite some time ago. But I hope that this
>>> improvement will make it into the 1.6 release.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <ju...@rovio.com> wrote:
>>>
>>>> Thanks for the answer. Wrapping with GET sounds good to me. You said
>>>> next version; do you mean that Flink 1.5 would already include this
>>>> improvement when it's released?
>>>>
>>>> On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <tr...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Juho,
>>>>>
>>>>> you are right that due to a limitation in the Yarn proxy [1] we cannot
>>>>> directly contact the cluster through the Yarn proxy.
>>>>>
>>>>> The way it works at the moment is that the Flink client retrieves the
>>>>> AM's hostname through the ApplicationReport and then directly talks to the
>>>>> AM. This of course requires that one can reach the respective Yarn
>>>>> container. This is a limitation, though not a regression, which we want to
>>>>> improve with the next version of Flink. An idea would be to wrap the REST
>>>>> calls in a GET call to make them pass through the Yarn proxy.
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/YARN-2084
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Juho,
>>>>>>
>>>>>> Thanks for raising this point!
>>>>>>
>>>>>> I'll add Chesnay and Till to the thread who contributed to the REST
>>>>>> API.
>>>>>>
>>>>>> Best, Fabian
>>>>>>
>>>>>> 2018-04-04 15:02 GMT+02:00 Juho Autio <ju...@rovio.com>:
>>>>>>
>>>>>>> I just learned that Flink savepoints API was refactored to require
>>>>>>> using HTTP POST.
>>>>>>>
>>>>>>> That's fine otherwise, but makes life harder when Flink is run on
>>>>>>> top of YARN.
>>>>>>>
>>>>>>> I've added example calls below to show how POST is declined by
>>>>>>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.
>>>>>>>
>>>>>>> Can you think of any solution to this? If I would be able to
>>>>>>> determine the actual host & port for Flink UI, I could use that instead of
>>>>>>> the proxy address. But that would probably require opening at least one
>>>>>>> more port, so it's not the optimal solution either. Ideally I would have
>>>>>>> Flink REST API completely accessible with GET and PUT methods.
>>>>>>>
>>>>>>> To me it seems like AWS EMR will also hit this issue as soon as they
>>>>>>> start supporting Flink 1.5, because they seem to run Flink as a YARN app.
>>>>>>>
>>>>>>>
>>>>>>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p
>>>>>>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web
>>>>>>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/
>>>>>>> WebAppProxyServlet.java#L296-L306
>>>>>>>
>>>>>>>
>>>>>>> $ http POST http://10.0.10.71:20888/proxy/
>>>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23
>>>>>>> d4f8b908/checkpoints
>>>>>>>
>>>>>>> HTTP/1.1 405 HTTP method POST is not supported by this URL
>>>>>>> Cache-Control: must-revalidate,no-cache,no-store
>>>>>>> Content-Length: 1523
>>>>>>> Content-Type: text/html; charset=iso-8859-1
>>>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>>>>> Pragma: no-cache
>>>>>>> Pragma: no-cache
>>>>>>>
>>>>>>> <html>
>>>>>>> <head>
>>>>>>> <meta http-equiv="Content-Type" content="text/html;
>>>>>>> charset=ISO-8859-1"/>
>>>>>>> <title>Error 405 HTTP method POST is not supported by this
>>>>>>> URL</title>
>>>>>>> </head>
>>>>>>> <body><h2>HTTP ERROR 405</h2>
>>>>>>> <p>Problem accessing /proxy/application_15228441533
>>>>>>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
>>>>>>> <pre> HTTP method POST is not supported by this URL</pre></p><hr
>>>>>>> /><i><small>Powered by Jetty://</small></i><br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>> <br/>
>>>>>>>
>>>>>>> </body>
>>>>>>> </html>
>>>>>>>
>>>>>>>
>>>>>>> $ http PUT http://10.0.10.71:20888/proxy/
>>>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23
>>>>>>> d4f8b908/checkpoints
>>>>>>>
>>>>>>> HTTP/1.1 404 Not Found
>>>>>>> Access-Control-Allow-Origin: *
>>>>>>> Cache-Control: no-cache
>>>>>>> Content-Length: 25
>>>>>>> Content-Type: application/json; charset=UTF-8
>>>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>>> Pragma: no-cache
>>>>>>> Pragma: no-cache
>>>>>>>
>>>>>>> {
>>>>>>> "errors": [
>>>>>>> "Not found."
>>>>>>> ]
>>>>>>> }
>>>>>>>
>>>>>>> ^ expected to get this from flink because there's no PUT
>>>>>>> /checkpoints.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: REST API "broken" on YARN because POST is not allowed via YARN proxy
Posted by Till Rohrmann <tr...@apache.org>.
Hi Juho,
I created a JIRA issue for the problem [1]. Yes, the solution would be to
use only GET requests. Either we wrap our requests in a GET request or we
change our handlers to accept GET requests only. One thing which we have to
fix first is that also the jar file upload goes through REST.
[1] https://issues.apache.org/jira/browse/FLINK-9478
Cheers,
Till
On Wed, May 30, 2018 at 9:07 AM, Juho Autio <ju...@rovio.com> wrote:
> Hi, I tried to search Flink Jira for this but couldn't find a ticket to
> match. If there's no ticket yet, did I understand correctly though, that
> you would be open to support GET as an alternative method for all of
> Flink's REST operations?
>
> Now that 1.5 was released, it's a pity that this problem kind of prevents
> us from upgrading – or spend time on creating a workaround.
>
>
> On Thu, Apr 5, 2018 at 9:10 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> This improvement is unfortunately out of scope for the 1.5 release since
>> the feature freeze is already quite some time ago. But I hope that this
>> improvement will make it into the 1.6 release.
>>
>> Cheers,
>> Till
>>
>> On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <ju...@rovio.com> wrote:
>>
>>> Thanks for the answer. Wrapping with GET sounds good to me. You said
>>> next version; do you mean that Flink 1.5 would already include this
>>> improvement when it's released?
>>>
>>> On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Hi Juho,
>>>>
>>>> you are right that due to a limitation in the Yarn proxy [1] we cannot
>>>> directly contact the cluster through the Yarn proxy.
>>>>
>>>> The way it works at the moment is that the Flink client retrieves the
>>>> AM's hostname through the ApplicationReport and then directly talks to the
>>>> AM. This of course requires that one can reach the respective Yarn
>>>> container. This is a limitation, though not a regression, which we want to
>>>> improve with the next version of Flink. An idea would be to wrap the REST
>>>> calls in a GET call to make them pass through the Yarn proxy.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/YARN-2084
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Juho,
>>>>>
>>>>> Thanks for raising this point!
>>>>>
>>>>> I'll add Chesnay and Till to the thread who contributed to the REST
>>>>> API.
>>>>>
>>>>> Best, Fabian
>>>>>
>>>>> 2018-04-04 15:02 GMT+02:00 Juho Autio <ju...@rovio.com>:
>>>>>
>>>>>> I just learned that Flink savepoints API was refactored to require
>>>>>> using HTTP POST.
>>>>>>
>>>>>> That's fine otherwise, but makes life harder when Flink is run on top
>>>>>> of YARN.
>>>>>>
>>>>>> I've added example calls below to show how POST is declined by
>>>>>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.
>>>>>>
>>>>>> Can you think of any solution to this? If I would be able to
>>>>>> determine the actual host & port for Flink UI, I could use that instead of
>>>>>> the proxy address. But that would probably require opening at least one
>>>>>> more port, so it's not the optimal solution either. Ideally I would have
>>>>>> Flink REST API completely accessible with GET and PUT methods.
>>>>>>
>>>>>> To me it seems like AWS EMR will also hit this issue as soon as they
>>>>>> start supporting Flink 1.5, because they seem to run Flink as a YARN app.
>>>>>>
>>>>>>
>>>>>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p
>>>>>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web
>>>>>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/
>>>>>> WebAppProxyServlet.java#L296-L306
>>>>>>
>>>>>>
>>>>>> $ http POST http://10.0.10.71:20888/proxy/
>>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23
>>>>>> d4f8b908/checkpoints
>>>>>>
>>>>>> HTTP/1.1 405 HTTP method POST is not supported by this URL
>>>>>> Cache-Control: must-revalidate,no-cache,no-store
>>>>>> Content-Length: 1523
>>>>>> Content-Type: text/html; charset=iso-8859-1
>>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>>>> Pragma: no-cache
>>>>>> Pragma: no-cache
>>>>>>
>>>>>> <html>
>>>>>> <head>
>>>>>> <meta http-equiv="Content-Type" content="text/html;
>>>>>> charset=ISO-8859-1"/>
>>>>>> <title>Error 405 HTTP method POST is not supported by this URL</title>
>>>>>> </head>
>>>>>> <body><h2>HTTP ERROR 405</h2>
>>>>>> <p>Problem accessing /proxy/application_15228441533
>>>>>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
>>>>>> <pre> HTTP method POST is not supported by this URL</pre></p><hr
>>>>>> /><i><small>Powered by Jetty://</small></i><br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>> <br/>
>>>>>>
>>>>>> </body>
>>>>>> </html>
>>>>>>
>>>>>>
>>>>>> $ http PUT http://10.0.10.71:20888/proxy/
>>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23
>>>>>> d4f8b908/checkpoints
>>>>>>
>>>>>> HTTP/1.1 404 Not Found
>>>>>> Access-Control-Allow-Origin: *
>>>>>> Cache-Control: no-cache
>>>>>> Content-Length: 25
>>>>>> Content-Type: application/json; charset=UTF-8
>>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>>>> Pragma: no-cache
>>>>>> Pragma: no-cache
>>>>>>
>>>>>> {
>>>>>> "errors": [
>>>>>> "Not found."
>>>>>> ]
>>>>>> }
>>>>>>
>>>>>> ^ expected to get this from flink because there's no PUT /checkpoints.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>