You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Aaron Langford <aa...@gmail.com> on 2020/03/20 04:09:15 UTC
Savepoint Location from Flink REST API
Hey Flink Community,
I'm combing through docs right now, and I don't see that a savepoint
location is returned or surfaced anywhere. When I do this in the CLI, I get
a nice message that tells me where in S3 it put my savepoint (unique
savepoint ID included). I'm looking for that same result to be available
via the REST API. Does this exist today?
Aaron
Re: Savepoint Location from Flink REST API
Posted by Ufuk Celebi <uc...@apache.org>.
Sorry for the copy & paste error in my earlier message. 🙄 I agree with
Robert.
On 2. Apr 2020, at 11:06, Robert Metzger <rm...@apache.org> wrote:
Good catch!. Yes, you can add this to FLINK-16696.
On Wed, Apr 1, 2020 at 10:59 PM Aaron Langford <aa...@gmail.com>
wrote:
> All, it looks like the actual return structure from the API is:
>
> 1. Success
>
>> {
>> "status": {
>> "id": "completed"
>> },
>> *"operation"*: {
>> "location": "string"
>> }
>> }
>
>
> 2. Failure
>
>> {
>> "status": {
>> "id": "completed"
>> },
>> *"operation"*: {
>> "failure-cause": {
>> "class": "string",
>> "stack-trace": "string",
>> "serialized-throwable": "string"
>> }
>> }
>> }
>
>
> The difference is that "operation" is the key in the response, not
> "savepoint". Should this go into the FLINK-16696 ticket or a separate one?
>
> Aaron
>
>
> On Fri, Mar 20, 2020 at 1:33 PM Ufuk Celebi <uc...@apache.org> wrote:
>
>> Hey Aaron,
>>
>> you can expect one of the two responses for COMPLETED savepoints [1, 2].
>>
>> 1. Success
>>
>> {
>> "status": {
>> "id": "completed"
>> },
>> "savepoint": {
>> "location": "string"
>> }
>> }
>>
>> 2. Failure
>>
>> {
>> "status": {
>> "id": "completed"
>> },
>> "savepoint": {
>> "failure-cause": {
>> "class": "string",
>> "stack-trace": "string",
>> "serialized-throwable": "string"
>> }
>> }
>> }
>>
>> – Ufuk
>>
>> [1]
>> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
>> [2]
>> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java
>>
>> On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <
>> aaron.langford31@gmail.com> wrote:
>> >
>> > Roman,
>> >
>> > Thanks for the info. That's super helpful. I'd be interested in picking
>> that ticket up.
>> >
>> > One additional question: the states that can return from this API are
>> only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures
>> represented for this endpoint?
>> >
>> > Aaron
>> >
>> > On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <
>> khachatryan.roman@gmail.com> wrote:
>> >>
>> >> Hey Aaron,
>> >>
>> >> You can use /jobs/:jobid/savepoints/:triggerid to get the location
>> when the checkpoint is completed.
>> >>
>> >> Please see
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>> >>
>> >> Meanwhile, I've created an issue to update the docs:
>> https://issues.apache.org/jira/browse/FLINK-16696
>> >>
>> >> Regards,
>> >> Roman
>> >>
>> >>
>> >> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <
>> aaron.langford31@gmail.com> wrote:
>> >>>
>> >>> Hey Flink Community,
>> >>>
>> >>> I'm combing through docs right now, and I don't see that a savepoint
>> location is returned or surfaced anywhere. When I do this in the CLI, I get
>> a nice message that tells me where in S3 it put my savepoint (unique
>> savepoint ID included). I'm looking for that same result to be available
>> via the REST API. Does this exist today?
>> >>>
>> >>> Aaron
>>
>
Re: Savepoint Location from Flink REST API
Posted by Robert Metzger <rm...@apache.org>.
Good catch!. Yes, you can add this to FLINK-16696.
On Wed, Apr 1, 2020 at 10:59 PM Aaron Langford <aa...@gmail.com>
wrote:
> All, it looks like the actual return structure from the API is:
>
> 1. Success
>
>> {
>> "status": {
>> "id": "completed"
>> },
>> *"operation"*: {
>> "location": "string"
>> }
>> }
>
>
> 2. Failure
>
>> {
>> "status": {
>> "id": "completed"
>> },
>> *"operation"*: {
>> "failure-cause": {
>> "class": "string",
>> "stack-trace": "string",
>> "serialized-throwable": "string"
>> }
>> }
>> }
>
>
> The difference is that "operation" is the key in the response, not
> "savepoint". Should this go into the FLINK-16696 ticket or a separate one?
>
> Aaron
>
>
> On Fri, Mar 20, 2020 at 1:33 PM Ufuk Celebi <uc...@apache.org> wrote:
>
>> Hey Aaron,
>>
>> you can expect one of the two responses for COMPLETED savepoints [1, 2].
>>
>> 1. Success
>>
>> {
>> "status": {
>> "id": "completed"
>> },
>> "savepoint": {
>> "location": "string"
>> }
>> }
>>
>> 2. Failure
>>
>> {
>> "status": {
>> "id": "completed"
>> },
>> "savepoint": {
>> "failure-cause": {
>> "class": "string",
>> "stack-trace": "string",
>> "serialized-throwable": "string"
>> }
>> }
>> }
>>
>> – Ufuk
>>
>> [1]
>> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
>> [2]
>> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java
>>
>> On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <
>> aaron.langford31@gmail.com> wrote:
>> >
>> > Roman,
>> >
>> > Thanks for the info. That's super helpful. I'd be interested in picking
>> that ticket up.
>> >
>> > One additional question: the states that can return from this API are
>> only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures
>> represented for this endpoint?
>> >
>> > Aaron
>> >
>> > On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <
>> khachatryan.roman@gmail.com> wrote:
>> >>
>> >> Hey Aaron,
>> >>
>> >> You can use /jobs/:jobid/savepoints/:triggerid to get the location
>> when the checkpoint is completed.
>> >>
>> >> Please see
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>> >>
>> >> Meanwhile, I've created an issue to update the docs:
>> https://issues.apache.org/jira/browse/FLINK-16696
>> >>
>> >> Regards,
>> >> Roman
>> >>
>> >>
>> >> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <
>> aaron.langford31@gmail.com> wrote:
>> >>>
>> >>> Hey Flink Community,
>> >>>
>> >>> I'm combing through docs right now, and I don't see that a savepoint
>> location is returned or surfaced anywhere. When I do this in the CLI, I get
>> a nice message that tells me where in S3 it put my savepoint (unique
>> savepoint ID included). I'm looking for that same result to be available
>> via the REST API. Does this exist today?
>> >>>
>> >>> Aaron
>>
>
Re: Savepoint Location from Flink REST API
Posted by Aaron Langford <aa...@gmail.com>.
All, it looks like the actual return structure from the API is:
1. Success
> {
> "status": {
> "id": "completed"
> },
> *"operation"*: {
> "location": "string"
> }
> }
2. Failure
> {
> "status": {
> "id": "completed"
> },
> *"operation"*: {
> "failure-cause": {
> "class": "string",
> "stack-trace": "string",
> "serialized-throwable": "string"
> }
> }
> }
The difference is that "operation" is the key in the response, not
"savepoint". Should this go into the FLINK-16696 ticket or a separate one?
Aaron
On Fri, Mar 20, 2020 at 1:33 PM Ufuk Celebi <uc...@apache.org> wrote:
> Hey Aaron,
>
> you can expect one of the two responses for COMPLETED savepoints [1, 2].
>
> 1. Success
>
> {
> "status": {
> "id": "completed"
> },
> "savepoint": {
> "location": "string"
> }
> }
>
> 2. Failure
>
> {
> "status": {
> "id": "completed"
> },
> "savepoint": {
> "failure-cause": {
> "class": "string",
> "stack-trace": "string",
> "serialized-throwable": "string"
> }
> }
> }
>
> – Ufuk
>
> [1]
> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
> [2]
> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java
>
> On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <aa...@gmail.com>
> wrote:
> >
> > Roman,
> >
> > Thanks for the info. That's super helpful. I'd be interested in picking
> that ticket up.
> >
> > One additional question: the states that can return from this API are
> only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures
> represented for this endpoint?
> >
> > Aaron
> >
> > On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <
> khachatryan.roman@gmail.com> wrote:
> >>
> >> Hey Aaron,
> >>
> >> You can use /jobs/:jobid/savepoints/:triggerid to get the location when
> the checkpoint is completed.
> >>
> >> Please see
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
> >>
> >> Meanwhile, I've created an issue to update the docs:
> https://issues.apache.org/jira/browse/FLINK-16696
> >>
> >> Regards,
> >> Roman
> >>
> >>
> >> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <
> aaron.langford31@gmail.com> wrote:
> >>>
> >>> Hey Flink Community,
> >>>
> >>> I'm combing through docs right now, and I don't see that a savepoint
> location is returned or surfaced anywhere. When I do this in the CLI, I get
> a nice message that tells me where in S3 it put my savepoint (unique
> savepoint ID included). I'm looking for that same result to be available
> via the REST API. Does this exist today?
> >>>
> >>> Aaron
>
Re: Savepoint Location from Flink REST API
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Aaron,
you can expect one of the two responses for COMPLETED savepoints [1, 2].
1. Success
{
"status": {
"id": "completed"
},
"savepoint": {
"location": "string"
}
}
2. Failure
{
"status": {
"id": "completed"
},
"savepoint": {
"failure-cause": {
"class": "string",
"stack-trace": "string",
"serialized-throwable": "string"
}
}
}
– Ufuk
[1]
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
[2]
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java
On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <aa...@gmail.com>
wrote:
>
> Roman,
>
> Thanks for the info. That's super helpful. I'd be interested in picking
that ticket up.
>
> One additional question: the states that can return from this API are
only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures
represented for this endpoint?
>
> Aaron
>
> On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <
khachatryan.roman@gmail.com> wrote:
>>
>> Hey Aaron,
>>
>> You can use /jobs/:jobid/savepoints/:triggerid to get the location when
the checkpoint is completed.
>>
>> Please see
https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>>
>> Meanwhile, I've created an issue to update the docs:
https://issues.apache.org/jira/browse/FLINK-16696
>>
>> Regards,
>> Roman
>>
>>
>> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <
aaron.langford31@gmail.com> wrote:
>>>
>>> Hey Flink Community,
>>>
>>> I'm combing through docs right now, and I don't see that a savepoint
location is returned or surfaced anywhere. When I do this in the CLI, I get
a nice message that tells me where in S3 it put my savepoint (unique
savepoint ID included). I'm looking for that same result to be available
via the REST API. Does this exist today?
>>>
>>> Aaron
Re: Savepoint Location from Flink REST API
Posted by Aaron Langford <aa...@gmail.com>.
Roman,
Thanks for the info. That's super helpful. I'd be interested in picking
that ticket up.
One additional question: the states that can return from this API are only
described as 'COMPLETED' or 'IN_PROGRESS'. How are failures represented for
this endpoint?
Aaron
On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <
khachatryan.roman@gmail.com> wrote:
> Hey Aaron,
>
> You can use /jobs/:jobid/savepoints/:triggerid to get the location when
> the checkpoint is completed.
>
> Please see
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>
> Meanwhile, I've created an issue to update the docs:
> https://issues.apache.org/jira/browse/FLINK-16696
>
> Regards,
> Roman
>
>
> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <aa...@gmail.com>
> wrote:
>
>> Hey Flink Community,
>>
>> I'm combing through docs right now, and I don't see that a savepoint
>> location is returned or surfaced anywhere. When I do this in the CLI, I get
>> a nice message that tells me where in S3 it put my savepoint (unique
>> savepoint ID included). I'm looking for that same result to be available
>> via the REST API. Does this exist today?
>>
>> Aaron
>>
>
Re: Savepoint Location from Flink REST API
Posted by Khachatryan Roman <kh...@gmail.com>.
Hey Aaron,
You can use /jobs/:jobid/savepoints/:triggerid to get the location when the
checkpoint is completed.
Please see
https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
Meanwhile, I've created an issue to update the docs:
https://issues.apache.org/jira/browse/FLINK-16696
Regards,
Roman
On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <aa...@gmail.com>
wrote:
> Hey Flink Community,
>
> I'm combing through docs right now, and I don't see that a savepoint
> location is returned or surfaced anywhere. When I do this in the CLI, I get
> a nice message that tells me where in S3 it put my savepoint (unique
> savepoint ID included). I'm looking for that same result to be available
> via the REST API. Does this exist today?
>
> Aaron
>