You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Siew Wai Yow <wa...@hotmail.com> on 2018/06/19 15:35:36 UTC
Questions regarding to Flink 1.5.0 REST API change
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
* The REST API to cancel a job was changed.
* The REST API to cancel a job with savepoint was changed.
I have few dump questions,
1. Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
2. For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
* http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
3. Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
* previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Siew Wai Yow <wa...@hotmail.com>.
Thank you @Chesnay Schepler<ma...@apache.org> and @Esteban Serrano<ma...@gmail.com>!
________________________________
From: Chesnay Schepler <ch...@apache.org>
Sent: Tuesday, June 19, 2018 11:55 PM
To: user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{
"target-directory" : {
"type" : "string"
},
"cancel-job" : {
"type" : "boolean"
}
}
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wa...@hotmail.com>> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
* The REST API to cancel a job was changed.
* The REST API to cancel a job with savepoint was changed.
I have few dump questions,
1. Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
2. For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
* http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
3. Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
* previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by sihua zhou <su...@163.com>.
I just create a JIRA for this: https://issues.apache.org/jira/browse/FLINK-9633
On 06/21/2018 15:10，Chesnay Schepler<ch...@apache.org> wrote：
That's quite weird that it tries to us the local file-system. Maybe it derives the FS from state.backend.fs.checckpointdir, but uses it for savepoints.dir?
What happens if you set state.backend.fs.checkpointdir also to HDFS?
On 21.06.2018 08:07, Siew Wai Yow wrote:
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.
Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
... 25 more
flink-conf.yaml:
--------------------------------
state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
-Yow
From: Chesnay Schepler <ch...@apache.org>
Sent: Wednesday, June 20, 2018 3:15 PM
To:user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...
On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,
Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler <ch...@apache.org>
Sent: Tuesday, June 19, 2018 11:55 PM
To: user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{
"target-directory" : {
"type" : "string"
},
"cancel-job" : {
"type" : "boolean"
}
}
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wa...@hotmail.com> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.
I have few dump questions,
Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Chesnay Schepler <ch...@apache.org>.
That's quite weird that it tries to us the local file-system. Maybe it
derives the FS from state.backend.fs.checckpointdir, but uses it for
savepoints.dir?
What happens if you set state.backend.fs.checkpointdir also to HDFS?
On 21.06.2018 08:07, Siew Wai Yow wrote:
>
> Thanks Chesnay, the application will take value from
> "state.savepoints.dir" as default if set target-directory to nul. But
> then it trying to create the directory in local machine, which
> caused the below error because it is a HDFS directory. The same URL
> works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or
> anything we need to extra configure? Thank you.
>
>
> Caused by: java.io.IOException: *Mkdirs failed to create*
> hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
> at
> org.apache.flink.core.fs.local.*LocalFileSystem*.create(LocalFileSystem.java:271)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
> at
> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
> ... 25 more
>
>
>
> flink-conf.yaml:
>
> --------------------------------
>
> state.backend: rocksdb
> state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
> state.backend.incremental: true
> state.backend.async: true
> state.checkpoints.num-retained: 5
> state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
>
>
> -Yow
>
>
>
> ------------------------------------------------------------------------
> *From:* Chesnay Schepler <ch...@apache.org>
> *Sent:* Wednesday, June 20, 2018 3:15 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
> I think you can set the target-directory to null. But I'm not sure why
> this particular request requires this, other request allow optional
> fields to simply be ommitted...
>
> On 20.06.2018 06:12, Siew Wai Yow wrote:
>>
>> Hi all,
>>
>>
>> Seems pass in target-directory is a must now for checkpoints REST
>> API, and the status will not response with save point directory
>> anymore. I can pass in but the information is redundant with the same
>> already defined in flink-config.yaml. May I know is there a way to
>> retrieve the save point directory from flink-config.yaml in flink
>> application? I am not able to get it from env.getConfig(). Thank you.
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Chesnay Schepler <ch...@apache.org> <ma...@apache.org>
>> *Sent:* Tuesday, June 19, 2018 11:55 PM
>> *To:* user@flink.apache.org <ma...@flink.apache.org>
>> *Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
>> 1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode"
>> query parameter
>>
>> 2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a
>> trigger id, used for 3).
>> {|
>> "target-directory" : { |
>> |"type" : "string" }, "cancel-job" : { "type" : "boolean" } } |
>> 3. GET to /jobs/:jobid/savepoints/:triggerid
>>
>> On 19.06.2018 17:40, Esteban Serrano wrote:
>>> For #1, you need to use a PATCH request to "*/jobs/:jobid*"
>>>
>>> On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wai_yow@hotmail.com
>>> <ma...@hotmail.com>> wrote:
>>>
>>> Hi,
>>>
>>>
>>> Regarding to Flink 1.5.0 REST API breaking change,
>>>
>>> * /The REST API to cancel a job was changed./
>>> * /The REST API to cancel a job with savepoint was changed./
>>>
>>> I have few dump questions,
>>>
>>>
>>> 1. Any replacement for cancellation ONLY without save-point?
>>> Only found "*/jobs/:jobid/savepoints*".
>>> 2. For "*/jobs/:jobid/savepoints*", how could I form the URL
>>> with cancellation and with directory afterward?
>>> 1. http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
>>>
>>> 3. Any cancellation progress monitoring in 1.5 like previous
>>> version 1.3/1.4?
>>> 1. previous version:
>>> /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
>>>
>>>
>>> Thank you.
>>>
>>> Regards,
>>> Yow
>>>
>>>
>>
>
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Siew Wai Yow <wa...@hotmail.com>.
Thanks @SihuaZhou, you are right that this is a bug. Just check the source code too. @Chesnay Schepler, <ma...@apache.org> Tested with both checkpoint and savepoint at same File system and it is working as expected.
Thanks guys!
-Yow
________________________________
From: Chesnay Schepler <ch...@apache.org>
Sent: Thursday, June 21, 2018 3:36 PM
To: user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
If you could open a JIRA this would be great.
On 21.06.2018 09:07, sihua zhou wrote:
Hi Yow,
I had a look at the related code, I think this seems like a bug. Flink use Checkpoint path's FileSystem to create the output stream for the Savepoint, but in your case the checkpoint & savepoint are not using the same file system. A workaround is to use the same file system for both checkpoint & savepoint.
Best, Sihua
On 06/21/2018 14:07，Siew Wai Yow<wa...@hotmail.com> wrote：
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.
Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
... 25 more
flink-conf.yaml:
--------------------------------
state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
-Yow
________________________________
From: Chesnay Schepler <ch...@apache.org>>
Sent: Wednesday, June 20, 2018 3:15 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...
On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,
Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
________________________________
From: Chesnay Schepler <ch...@apache.org>
Sent: Tuesday, June 19, 2018 11:55 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{
"target-directory" : {
"type" : "string"
},
"cancel-job" : {
"type" : "boolean"
}
}
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wa...@hotmail.com>> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
* The REST API to cancel a job was changed.
* The REST API to cancel a job with savepoint was changed.
I have few dump questions,
1. Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
2. For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
* http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
3. Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
* previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Chesnay Schepler <ch...@apache.org>.
If you could open a JIRA this would be great.
On 21.06.2018 09:07, sihua zhou wrote:
> Hi Yow,
>
> I had a look at the related code, I think this seems like a bug. Flink
> use Checkpoint path's FileSystem to create the output stream for the
> Savepoint, but in your case the checkpoint & savepoint are not using
> the same file system. A workaround is to use the same file system for
> both checkpoint & savepoint.
>
> Best, Sihua
>
>
>
> On 06/21/2018 14:07，Siew Wai Yow<wa...@hotmail.com>
> <ma...@hotmail.com> wrote：
>
> Thanks Chesnay, the application will take value from
> "state.savepoints.dir" as default if set target-directory to nul.
> But then it trying to create the directory in local machine, which
> caused the below error because it is a HDFS directory. The same
> URL works in previous Flink 1.3.2. Is something break in Flink
> 1.5.0? Or anything we need to extra configure? Thank you.
>
>
> Caused by: java.io.IOException: *Mkdirs failed to create*
> hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
> at
> org.apache.flink.core.fs.local.*LocalFileSystem*.create(LocalFileSystem.java:271)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
> at
> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
> ... 25 more
>
>
>
> flink-conf.yaml:
>
> --------------------------------
>
> state.backend: rocksdb
> state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
> state.backend.incremental: true
> state.backend.async: true
> state.checkpoints.num-retained: 5
> state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
>
>
> -Yow
>
>
>
> ------------------------------------------------------------------------
> *From:* Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>>
> *Sent:* Wednesday, June 20, 2018 3:15 PM
> *To:* user@flink.apache.org <ma...@flink.apache.org>
> *Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
> I think you can set the target-directory to null. But I'm not sure
> why this particular request requires this, other request allow
> optional fields to simply be ommitted...
>
> On 20.06.2018 06:12, Siew Wai Yow wrote:
>>
>> Hi all,
>>
>>
>> Seems pass in target-directory is a must now for checkpoints REST
>> API, and the status will not response with save point directory
>> anymore. I can pass in but the information is redundant with the
>> same already defined in flink-config.yaml. May I know is there a
>> way to retrieve the save point directory from flink-config.yaml
>> in flink application? I am not able to get it from
>> env.getConfig(). Thank you.
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Chesnay Schepler <ch...@apache.org>
>> <ma...@apache.org>
>> *Sent:* Tuesday, June 19, 2018 11:55 PM
>> *To:* user@flink.apache.org <ma...@flink.apache.org>
>> *Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
>> 1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the
>> "mode" query parameter
>>
>> 2. POST to /jobs/:jobid/savepoints, with a json payload. Returns
>> a trigger id, used for 3).
>> {|
>> "target-directory" : { |
>> |"type" : "string" }, "cancel-job" : { "type" : "boolean" } } |
>> 3. GET to /jobs/:jobid/savepoints/:triggerid
>>
>> On 19.06.2018 17:40, Esteban Serrano wrote:
>>> For #1, you need to use a PATCH request to "*/jobs/:jobid*"
>>>
>>> On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow
>>> <wai_yow@hotmail.com <ma...@hotmail.com>> wrote:
>>>
>>> Hi,
>>>
>>>
>>> Regarding to Flink 1.5.0 REST API breaking change,
>>>
>>> * /The REST API to cancel a job was changed./
>>> * /The REST API to cancel a job with savepoint was changed./
>>>
>>> I have few dump questions,
>>>
>>>
>>> 1. Any replacement for cancellation ONLY without
>>> save-point? Only found "*/jobs/:jobid/savepoints*".
>>> 2. For "*/jobs/:jobid/savepoints*", how could I form the
>>> URL with cancellation and with directory afterward?
>>> 1. http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
>>>
>>> 3. Any cancellation progress monitoring in 1.5 like
>>> previous version 1.3/1.4?
>>> 1. previous version:
>>> /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
>>>
>>>
>>>
>>> Thank you.
>>>
>>> Regards,
>>> Yow
>>>
>>>
>>
>
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by sihua zhou <su...@163.com>.
Hi Yow,
I had a look at the related code, I think this seems like a bug. Flink use Checkpoint path's FileSystem to create the output stream for the Savepoint, but in your case the checkpoint & savepoint are not using the same file system. A workaround is to use the same file system for both checkpoint & savepoint.
Best, Sihua
On 06/21/2018 14:07，Siew Wai Yow<wa...@hotmail.com> wrote：
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.
Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
... 25 more
flink-conf.yaml:
--------------------------------
state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
-Yow
From: Chesnay Schepler <ch...@apache.org>
Sent: Wednesday, June 20, 2018 3:15 PM
To:user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...
On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,
Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler <ch...@apache.org>
Sent: Tuesday, June 19, 2018 11:55 PM
To: user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{
"target-directory" : {
"type" : "string"
},
"cancel-job" : {
"type" : "boolean"
}
}
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wa...@hotmail.com> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.
I have few dump questions,
Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Siew Wai Yow <wa...@hotmail.com>.
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.
Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
... 25 more
flink-conf.yaml:
--------------------------------
state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
-Yow
________________________________
From: Chesnay Schepler <ch...@apache.org>
Sent: Wednesday, June 20, 2018 3:15 PM
To: user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...
On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,
Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
________________________________
From: Chesnay Schepler <ch...@apache.org>
Sent: Tuesday, June 19, 2018 11:55 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{
"target-directory" : {
"type" : "string"
},
"cancel-job" : {
"type" : "boolean"
}
}
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wa...@hotmail.com>> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
* The REST API to cancel a job was changed.
* The REST API to cancel a job with savepoint was changed.
I have few dump questions,
1. Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
2. For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
* http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
3. Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
* previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Chesnay Schepler <ch...@apache.org>.
I think you can set the target-directory to null. But I'm not sure why
this particular request requires this, other request allow optional
fields to simply be ommitted...
On 20.06.2018 06:12, Siew Wai Yow wrote:
>
> Hi all,
>
>
> Seems pass in target-directory is a must now for checkpoints REST API,
> and the status will not response with save point directory anymore. I
> can pass in but the information is redundant with the same already
> defined in flink-config.yaml. May I know is there a way to retrieve
> the save point directory from flink-config.yaml in flink application?
> I am not able to get it from env.getConfig(). Thank you.
>
>
>
> ------------------------------------------------------------------------
> *From:* Chesnay Schepler <ch...@apache.org>
> *Sent:* Tuesday, June 19, 2018 11:55 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
> 1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode"
> query parameter
>
> 2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a
> trigger id, used for 3).
> {|
> "target-directory" : { |
> |"type" : "string" }, "cancel-job" : { "type" : "boolean" } } |
> 3. GET to /jobs/:jobid/savepoints/:triggerid
>
> On 19.06.2018 17:40, Esteban Serrano wrote:
>> For #1, you need to use a PATCH request to "*/jobs/:jobid*"
>>
>> On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wai_yow@hotmail.com
>> <ma...@hotmail.com>> wrote:
>>
>> Hi,
>>
>>
>> Regarding to Flink 1.5.0 REST API breaking change,
>>
>> * /The REST API to cancel a job was changed./
>> * /The REST API to cancel a job with savepoint was changed./
>>
>> I have few dump questions,
>>
>>
>> 1. Any replacement for cancellation ONLY without save-point?
>> Only found "*/jobs/:jobid/savepoints*".
>> 2. For "*/jobs/:jobid/savepoints*", how could I form the URL
>> with cancellation and with directory afterward?
>> 1. http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
>>
>> 3. Any cancellation progress monitoring in 1.5 like previous
>> version 1.3/1.4?
>> 1. previous version:
>> /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
>>
>>
>> Thank you.
>>
>> Regards,
>> Yow
>>
>>
>
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Siew Wai Yow <wa...@hotmail.com>.
Hi all,
Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
________________________________
From: Chesnay Schepler <ch...@apache.org>
Sent: Tuesday, June 19, 2018 11:55 PM
To: user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{
"target-directory" : {
"type" : "string"
},
"cancel-job" : {
"type" : "boolean"
}
}
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wa...@hotmail.com>> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
* The REST API to cancel a job was changed.
* The REST API to cancel a job with savepoint was changed.
I have few dump questions,
1. Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
2. For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
* http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
3. Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
* previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Chesnay Schepler <ch...@apache.org>.
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode"
query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a
trigger id, used for 3).
{|
"target-directory" : { |
|"type" : "string" }, "cancel-job" : { "type" : "boolean" } } |
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
> For #1, you need to use a PATCH request to "*/jobs/:jobid*"
>
> On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wai_yow@hotmail.com
> <ma...@hotmail.com>> wrote:
>
> Hi,
>
>
> Regarding to Flink 1.5.0 REST API breaking change,
>
> * /The REST API to cancel a job was changed./
> * /The REST API to cancel a job with savepoint was changed./
>
> I have few dump questions,
>
>
> 1. Any replacement for cancellation ONLY without save-point? Only
> found "*/jobs/:jobid/savepoints*".
> 2. For "*/jobs/:jobid/savepoints*", how could I form the URL with
> cancellation and with directory afterward?
> 1. http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
> 3. Any cancellation progress monitoring in 1.5 like previous
> version 1.3/1.4?
> 1. previous version:
> /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
>
>
> Thank you.
>
> Regards,
> Yow
>
>
Re: Questions regarding to Flink 1.5.0 REST API change
Posted by Esteban Serrano <st...@gmail.com>.
For #1, you need to use a PATCH request to "*/jobs/:jobid*"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wa...@hotmail.com> wrote:
> Hi,
>
>
> Regarding to Flink 1.5.0 REST API breaking change,
>
> - *The REST API to cancel a job was changed.*
> - *The REST API to cancel a job with savepoint was changed.*
>
> I have few dump questions,
>
>
>
> 1. Any replacement for cancellation ONLY without save-point? Only
> found "*/jobs/:jobid/savepoints*".
> 2. For "*/jobs/:jobid/savepoints*", how could I form the URL with
> cancellation and with directory afterward?
> 1. http://192.168.56.151:8081
> /jobs/<jobid>/savepoints/cancel-job/true??
> 3. Any cancellation progress monitoring in 1.5 like previous version
> 1.3/1.4?
> 1. previous version:
> /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
>
>
> Thank you.
>
> Regards,
> Yow
>
>
>