You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Emily McMahon <em...@remitly.com> on 2017/09/19 08:23:02 UTC
Rest API cancel-with-savepoint: 404s when passing path as target-directory
I've tried every combination I can think of to pass an s3 path as the
target directory (url encode, include trailing slash, etc)
I can successfully pass a local path as the target directory (ie
/jobs/$jobID/cancel-with-savepoint/target-directory/tmp) so I don't think
there's a problem with the jobId or rest of the url. I also verified I can
create the savepoint on s3 from the command line so it's not a permission
issue.
Here's the same question on stack overflow
<https://stackoverflow.com/questions/45844298/send-flink-job-manager-a-fully-qualified-path-to-a-savepoint-directory-using-the>
(with
the exception that they are getting a 502 whereas I'm getting a 404)
using Flink 1.3.1
Anyone have a working example?
Thanks,
Emily
Re: Rest API cancel-with-savepoint: 404s when passing path as target-directory
Posted by Eron Wright <er...@gmail.com>.
It is not surprising to see fidelity issues with the YARN proxy. I suggest
opening a ticket on Flink side to update the cancel-with-savepoint API to
take the target directory as a query string parameter (of course, backwards
compatibility should be maintained).
On Wed, Sep 20, 2017 at 1:55 AM, Nico Kruber <ni...@data-artisans.com> wrote:
> Hi Emily,
> I'm not familiar with the details of the REST API either but if this is a
> problem with the proxy, maybe it is already interpreting the encoded URL
> and
> passes it on un-encoded - have you tried encoding the path again? That is,
> encoding the percent-signs:
>
> http://
> {ip}:20888/proxy/application_1504649135200_0001/jobs/
> 1a0fd176ec8aabb9b8464fa481f755f0/cancel-with-savepoint/target-directory/
> s3%253A%252F%252F%252Fremit-flink
>
>
> Nico
>
> On Wednesday, 20 September 2017 00:52:05 CEST Emily McMahon wrote:
> > Thanks Eron & Fabian.
> >
> > The issue was hitting a yarn proxy url vs the node itself. For example
> this
> > worked
> > http://
> > {ip}:37716/jobs/1a0fd176ec8aabb9b8464fa481f755
> f0/cancel-with-savepoint/targe
> > t-directory/s3%3A%2F%2F%2Fremit-flink
> >
> > But this did not
> > http://
> > {ip}:20888/proxy/application_1504649135200_0001/jobs/
> 1a0fd176ec8aabb9b8464fa
> > 481f755f0/cancel-with-savepoint/target-directory/s3%
> 3A%2F%2F%2Fremit-flink
> >
> > It's a bit confusing because the cancel api works with either and the
> proxy
> > url sometimes works as this was successful http://
> > {ip}:20888/proxy/application_1504649135200_0001/jobs/
> cca2dd609c716a7b0a19570
> > 0777e5b1f/cancel-with-savepoint/target-directory/tmp/
> >
> > Cheers,
> > Emily
> >
> > On Tue, Sep 19, 2017 at 2:37 PM, Eron Wright <er...@gmail.com>
> wrote:
> > > Good news, it can be done if you carefully encode the target directory
> > > with percent-encoding, as per:
> > > https://tools.ietf.org/html/rfc3986#section-2.1
> > >
> > > For example, given the directory `s3:///savepoint-bucket/my-
> awesome-job`,
> > > which encodes to `s3%3A%2F%2F%2Fsavepoint-bucket%2Fmy-awesome-job`, I
> was
> > > able to submit the following URL:
> > > http://localhost:8081/jobs/5c360ded6e4b7d8db103e71d68b7c8
> > > 3d/cancel-with-savepoint/target-directory/s3%3A%2F%2F%
> > > 2Fsavepoint-bucket%2Fmy-awesome-job
> > >
> > > And see the following in the log:
> > > 2017-09-19 14:27:45,939 INFO
> > > org.apache.flink.runtime.jobmanager.JobManager>
> > > - Trying to cancel job 5c360ded6e4b7d8db103e71d68b7c83d
> > >
> > > with savepoint to s3:///savepoint-bucket/my-awesome-job
> > >
> > > -Eron
> > >
> > > On Tue, Sep 19, 2017 at 1:54 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> > >> Hi Emily,
> > >>
> > >> thanks for reaching out.
> > >> I'm not familiar with the details of the Rest API but Ufuk (in CC)
> might
> > >> be able to help you.
> > >>
> > >> Best, Fabian
> > >>
> > >> 2017-09-19 10:23 GMT+02:00 Emily McMahon <em...@remitly.com>:
> > >>> I've tried every combination I can think of to pass an s3 path as the
> > >>> target directory (url encode, include trailing slash, etc)
> > >>>
> > >>> I can successfully pass a local path as the target directory (ie
> > >>> /jobs/$jobID/cancel-with-savepoint/target-directory/tmp) so I don't
> > >>> think there's a problem with the jobId or rest of the url. I also
> > >>> verified
> > >>> I can create the savepoint on s3 from the command line so it's not a
> > >>> permission issue.
> > >>>
> > >>> Here's the same question on stack overflow
> > >>> <https://stackoverflow.com/questions/45844298/send-flink-
> job-manager-a-f
> > >>> ully-qualified-path-to-a-savepoint-directory-using-the> (with the
> > >>> exception that they are getting a 502 whereas I'm getting a 404)
> > >>>
> > >>> using Flink 1.3.1
> > >>>
> > >>> Anyone have a working example?
> > >>>
> > >>> Thanks,
> > >>> Emily
>
>
Re: Rest API cancel-with-savepoint: 404s when passing path as target-directory
Posted by Nico Kruber <ni...@data-artisans.com>.
Hi Emily,
I'm not familiar with the details of the REST API either but if this is a
problem with the proxy, maybe it is already interpreting the encoded URL and
passes it on un-encoded - have you tried encoding the path again? That is,
encoding the percent-signs:
http://
{ip}:20888/proxy/application_1504649135200_0001/jobs/
1a0fd176ec8aabb9b8464fa481f755f0/cancel-with-savepoint/target-directory/
s3%253A%252F%252F%252Fremit-flink
Nico
On Wednesday, 20 September 2017 00:52:05 CEST Emily McMahon wrote:
> Thanks Eron & Fabian.
>
> The issue was hitting a yarn proxy url vs the node itself. For example this
> worked
> http://
> {ip}:37716/jobs/1a0fd176ec8aabb9b8464fa481f755f0/cancel-with-savepoint/targe
> t-directory/s3%3A%2F%2F%2Fremit-flink
>
> But this did not
> http://
> {ip}:20888/proxy/application_1504649135200_0001/jobs/1a0fd176ec8aabb9b8464fa
> 481f755f0/cancel-with-savepoint/target-directory/s3%3A%2F%2F%2Fremit-flink
>
> It's a bit confusing because the cancel api works with either and the proxy
> url sometimes works as this was successful http://
> {ip}:20888/proxy/application_1504649135200_0001/jobs/cca2dd609c716a7b0a19570
> 0777e5b1f/cancel-with-savepoint/target-directory/tmp/
>
> Cheers,
> Emily
>
> On Tue, Sep 19, 2017 at 2:37 PM, Eron Wright <er...@gmail.com> wrote:
> > Good news, it can be done if you carefully encode the target directory
> > with percent-encoding, as per:
> > https://tools.ietf.org/html/rfc3986#section-2.1
> >
> > For example, given the directory `s3:///savepoint-bucket/my-awesome-job`,
> > which encodes to `s3%3A%2F%2F%2Fsavepoint-bucket%2Fmy-awesome-job`, I was
> > able to submit the following URL:
> > http://localhost:8081/jobs/5c360ded6e4b7d8db103e71d68b7c8
> > 3d/cancel-with-savepoint/target-directory/s3%3A%2F%2F%
> > 2Fsavepoint-bucket%2Fmy-awesome-job
> >
> > And see the following in the log:
> > 2017-09-19 14:27:45,939 INFO
> > org.apache.flink.runtime.jobmanager.JobManager>
> > - Trying to cancel job 5c360ded6e4b7d8db103e71d68b7c83d
> >
> > with savepoint to s3:///savepoint-bucket/my-awesome-job
> >
> > -Eron
> >
> > On Tue, Sep 19, 2017 at 1:54 PM, Fabian Hueske <fh...@gmail.com> wrote:
> >> Hi Emily,
> >>
> >> thanks for reaching out.
> >> I'm not familiar with the details of the Rest API but Ufuk (in CC) might
> >> be able to help you.
> >>
> >> Best, Fabian
> >>
> >> 2017-09-19 10:23 GMT+02:00 Emily McMahon <em...@remitly.com>:
> >>> I've tried every combination I can think of to pass an s3 path as the
> >>> target directory (url encode, include trailing slash, etc)
> >>>
> >>> I can successfully pass a local path as the target directory (ie
> >>> /jobs/$jobID/cancel-with-savepoint/target-directory/tmp) so I don't
> >>> think there's a problem with the jobId or rest of the url. I also
> >>> verified
> >>> I can create the savepoint on s3 from the command line so it's not a
> >>> permission issue.
> >>>
> >>> Here's the same question on stack overflow
> >>> <https://stackoverflow.com/questions/45844298/send-flink-job-manager-a-f
> >>> ully-qualified-path-to-a-savepoint-directory-using-the> (with the
> >>> exception that they are getting a 502 whereas I'm getting a 404)
> >>>
> >>> using Flink 1.3.1
> >>>
> >>> Anyone have a working example?
> >>>
> >>> Thanks,
> >>> Emily
Re: Rest API cancel-with-savepoint: 404s when passing path as target-directory
Posted by Emily McMahon <em...@remitly.com>.
Thanks Eron & Fabian.
The issue was hitting a yarn proxy url vs the node itself. For example this
worked
http://
{ip}:37716/jobs/1a0fd176ec8aabb9b8464fa481f755f0/cancel-with-savepoint/target-directory/s3%3A%2F%2F%2Fremit-flink
But this did not
http://
{ip}:20888/proxy/application_1504649135200_0001/jobs/1a0fd176ec8aabb9b8464fa481f755f0/cancel-with-savepoint/target-directory/s3%3A%2F%2F%2Fremit-flink
It's a bit confusing because the cancel api works with either and the proxy
url sometimes works as this was successful http://
{ip}:20888/proxy/application_1504649135200_0001/jobs/cca2dd609c716a7b0a195700777e5b1f/cancel-with-savepoint/target-directory/tmp/
Cheers,
Emily
On Tue, Sep 19, 2017 at 2:37 PM, Eron Wright <er...@gmail.com> wrote:
> Good news, it can be done if you carefully encode the target directory
> with percent-encoding, as per:
> https://tools.ietf.org/html/rfc3986#section-2.1
>
> For example, given the directory `s3:///savepoint-bucket/my-awesome-job`,
> which encodes to `s3%3A%2F%2F%2Fsavepoint-bucket%2Fmy-awesome-job`, I was
> able to submit the following URL:
> http://localhost:8081/jobs/5c360ded6e4b7d8db103e71d68b7c8
> 3d/cancel-with-savepoint/target-directory/s3%3A%2F%2F%
> 2Fsavepoint-bucket%2Fmy-awesome-job
>
> And see the following in the log:
> 2017-09-19 14:27:45,939 INFO org.apache.flink.runtime.jobmanager.JobManager
> - Trying to cancel job 5c360ded6e4b7d8db103e71d68b7c83d
> with savepoint to s3:///savepoint-bucket/my-awesome-job
>
> -Eron
>
> On Tue, Sep 19, 2017 at 1:54 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Emily,
>>
>> thanks for reaching out.
>> I'm not familiar with the details of the Rest API but Ufuk (in CC) might
>> be able to help you.
>>
>> Best, Fabian
>>
>> 2017-09-19 10:23 GMT+02:00 Emily McMahon <em...@remitly.com>:
>>
>>> I've tried every combination I can think of to pass an s3 path as the
>>> target directory (url encode, include trailing slash, etc)
>>>
>>> I can successfully pass a local path as the target directory (ie
>>> /jobs/$jobID/cancel-with-savepoint/target-directory/tmp) so I don't
>>> think there's a problem with the jobId or rest of the url. I also verified
>>> I can create the savepoint on s3 from the command line so it's not a
>>> permission issue.
>>>
>>> Here's the same question on stack overflow
>>> <https://stackoverflow.com/questions/45844298/send-flink-job-manager-a-fully-qualified-path-to-a-savepoint-directory-using-the> (with
>>> the exception that they are getting a 502 whereas I'm getting a 404)
>>>
>>> using Flink 1.3.1
>>>
>>> Anyone have a working example?
>>>
>>> Thanks,
>>> Emily
>>>
>>
>>
>
Re: Rest API cancel-with-savepoint: 404s when passing path as target-directory
Posted by Eron Wright <er...@gmail.com>.
Good news, it can be done if you carefully encode the target directory with
percent-encoding, as per:
https://tools.ietf.org/html/rfc3986#section-2.1
For example, given the directory `s3:///savepoint-bucket/my-awesome-job`,
which encodes to `s3%3A%2F%2F%2Fsavepoint-bucket%2Fmy-awesome-job`, I was
able to submit the following URL:
http://localhost:8081/jobs/5c360ded6e4b7d8db103e71d68b7c83d/cancel-with-savepoint/target-directory/s3%3A%2F%2F%2Fsavepoint-bucket%2Fmy-awesome-job
And see the following in the log:
2017-09-19 14:27:45,939 INFO
org.apache.flink.runtime.jobmanager.JobManager - Trying to
cancel job 5c360ded6e4b7d8db103e71d68b7c83d with savepoint to
s3:///savepoint-bucket/my-awesome-job
-Eron
On Tue, Sep 19, 2017 at 1:54 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Emily,
>
> thanks for reaching out.
> I'm not familiar with the details of the Rest API but Ufuk (in CC) might
> be able to help you.
>
> Best, Fabian
>
> 2017-09-19 10:23 GMT+02:00 Emily McMahon <em...@remitly.com>:
>
>> I've tried every combination I can think of to pass an s3 path as the
>> target directory (url encode, include trailing slash, etc)
>>
>> I can successfully pass a local path as the target directory (ie
>> /jobs/$jobID/cancel-with-savepoint/target-directory/tmp) so I don't
>> think there's a problem with the jobId or rest of the url. I also verified
>> I can create the savepoint on s3 from the command line so it's not a
>> permission issue.
>>
>> Here's the same question on stack overflow
>> <https://stackoverflow.com/questions/45844298/send-flink-job-manager-a-fully-qualified-path-to-a-savepoint-directory-using-the> (with
>> the exception that they are getting a 502 whereas I'm getting a 404)
>>
>> using Flink 1.3.1
>>
>> Anyone have a working example?
>>
>> Thanks,
>> Emily
>>
>
>
Re: Rest API cancel-with-savepoint: 404s when passing path as target-directory
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Emily,
thanks for reaching out.
I'm not familiar with the details of the Rest API but Ufuk (in CC) might be
able to help you.
Best, Fabian
2017-09-19 10:23 GMT+02:00 Emily McMahon <em...@remitly.com>:
> I've tried every combination I can think of to pass an s3 path as the
> target directory (url encode, include trailing slash, etc)
>
> I can successfully pass a local path as the target directory (ie
> /jobs/$jobID/cancel-with-savepoint/target-directory/tmp) so I don't think
> there's a problem with the jobId or rest of the url. I also verified I can
> create the savepoint on s3 from the command line so it's not a permission
> issue.
>
> Here's the same question on stack overflow
> <https://stackoverflow.com/questions/45844298/send-flink-job-manager-a-fully-qualified-path-to-a-savepoint-directory-using-the> (with
> the exception that they are getting a 502 whereas I'm getting a 404)
>
> using Flink 1.3.1
>
> Anyone have a working example?
>
> Thanks,
> Emily
>