You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by Eleanore Jin <el...@gmail.com> on 2020/03/05 00:39:43 UTC

How to test flink job recover from checkpoint

Hi,

I have a flink application and checkpoint is enabled, I am running locally
using miniCluster.

I just wonder if there is a way to simulate the failure, and verify that
flink job restarts from checkpoint?

Thanks a lot!
Eleanore

Re: How to test flink job recover from checkpoint

Posted by Zhu Zhu <re...@gmail.com>.
Hi Eleanore,

You can change your application tasks to throw exceptions in a certain
frequency.
Alternatively, if the application has external dependencies (e.g. source),
you can trigger failures manually by manipulating the status of the
external service (e.g. shutdown the source service, or break the network
connection between the Flink app and the source service).

Thanks,
Zhu Zhu

Eleanore Jin <el...@gmail.com> 于2020年3月5日周四 上午8:40写道：

> Hi,
>
> I have a flink application and checkpoint is enabled, I am running locally
> using miniCluster.
>
> I just wonder if there is a way to simulate the failure, and verify that
> flink job restarts from checkpoint?
>
> Thanks a lot!
> Eleanore
>

Re: How to test flink job recover from checkpoint

Posted by Eleanore Jin <el...@gmail.com>.
Hi Zhu Zhu and Abhinav,

I am able to verify the recovery from checkpoint based on your suggestions,
thanks a lot for the help!
Eleanore

On Wed, Mar 4, 2020 at 5:40 PM Bajaj, Abhinav <ab...@here.com>
wrote:

> I implemented a custom function that throws up a runtime exception.
>
>
>
> You can extend from simpler MapFunction or more complicated
> RichParallelSourceFunction depending on your use case.
>
> You can add logic to throw a runtime exception on a certain condition in
> the map or run method.               .
>
> You can use a count or timer to trigger the exception.
>
>
>
> Sharing a quick handwritten example.
>
>
>
> DataStream<String> stream = .....
>
> DataStream<String> mappedStream = stream.map(new MapFunction<String,
> String>>() {
>
>           @Override
>
>           public String map(String value) throws Exception {
>
>             if (SOME_CONDITION) {
>
>               throw new RuntimeException("Lets test checkpointing");
>
>             }
>
>                 return value;
>
>           }
>
> });
>
>
>
> ~ Abhinav Bajaj
>
>
>
>
>
> *From: *Eleanore Jin <el...@gmail.com>
> *Date: *Wednesday, March 4, 2020 at 4:40 PM
> *To: *user <us...@flink.apache.org>, user-zh <us...@flink.apache.org>
> *Subject: *How to test flink job recover from checkpoint
>
>
>
> Hi,
>
>
>
> I have a flink application and checkpoint is enabled, I am running locally
> using miniCluster.
>
>
>
> I just wonder if there is a way to simulate the failure, and verify that
> flink job restarts from checkpoint?
>
>
>
> Thanks a lot!
>
> Eleanore
>

Re: How to test flink job recover from checkpoint

Posted by Eleanore Jin <el...@gmail.com>.
Hi Zhu Zhu and Abhinav,

I am able to verify the recovery from checkpoint based on your suggestions,
thanks a lot for the help!
Eleanore

On Wed, Mar 4, 2020 at 5:40 PM Bajaj, Abhinav <ab...@here.com>
wrote:

> I implemented a custom function that throws up a runtime exception.
>
>
>
> You can extend from simpler MapFunction or more complicated
> RichParallelSourceFunction depending on your use case.
>
> You can add logic to throw a runtime exception on a certain condition in
> the map or run method.               .
>
> You can use a count or timer to trigger the exception.
>
>
>
> Sharing a quick handwritten example.
>
>
>
> DataStream<String> stream = .....
>
> DataStream<String> mappedStream = stream.map(new MapFunction<String,
> String>>() {
>
>           @Override
>
>           public String map(String value) throws Exception {
>
>             if (SOME_CONDITION) {
>
>               throw new RuntimeException("Lets test checkpointing");
>
>             }
>
>                 return value;
>
>           }
>
> });
>
>
>
> ~ Abhinav Bajaj
>
>
>
>
>
> *From: *Eleanore Jin <el...@gmail.com>
> *Date: *Wednesday, March 4, 2020 at 4:40 PM
> *To: *user <us...@flink.apache.org>, user-zh <us...@flink.apache.org>
> *Subject: *How to test flink job recover from checkpoint
>
>
>
> Hi,
>
>
>
> I have a flink application and checkpoint is enabled, I am running locally
> using miniCluster.
>
>
>
> I just wonder if there is a way to simulate the failure, and verify that
> flink job restarts from checkpoint?
>
>
>
> Thanks a lot!
>
> Eleanore
>

Re: How to test flink job recover from checkpoint

Posted by "Bajaj, Abhinav" <ab...@here.com>.
I implemented a custom function that throws up a runtime exception.

You can extend from simpler MapFunction or more complicated RichParallelSourceFunction depending on your use case.
You can add logic to throw a runtime exception on a certain condition in the map or run method.               .
You can use a count or timer to trigger the exception.

Sharing a quick handwritten example.

DataStream<String> stream = .....
DataStream<String> mappedStream = stream.map(new MapFunction<String, String>>() {
          @Override
          public String map(String value) throws Exception {
            if (SOME_CONDITION) {
              throw new RuntimeException("Lets test checkpointing");
            }
                return value;
          }
});

~ Abhinav Bajaj


From: Eleanore Jin <el...@gmail.com>
Date: Wednesday, March 4, 2020 at 4:40 PM
To: user <us...@flink.apache.org>, user-zh <us...@flink.apache.org>
Subject: How to test flink job recover from checkpoint

Hi,

I have a flink application and checkpoint is enabled, I am running locally using miniCluster.

I just wonder if there is a way to simulate the failure, and verify that flink job restarts from checkpoint?

Thanks a lot!
Eleanore

Re: How to test flink job recover from checkpoint

Posted by "Bajaj, Abhinav" <ab...@here.com>.
I implemented a custom function that throws up a runtime exception.

You can extend from simpler MapFunction or more complicated RichParallelSourceFunction depending on your use case.
You can add logic to throw a runtime exception on a certain condition in the map or run method.               .
You can use a count or timer to trigger the exception.

Sharing a quick handwritten example.

DataStream<String> stream = .....
DataStream<String> mappedStream = stream.map(new MapFunction<String, String>>() {
          @Override
          public String map(String value) throws Exception {
            if (SOME_CONDITION) {
              throw new RuntimeException("Lets test checkpointing");
            }
                return value;
          }
});

~ Abhinav Bajaj


From: Eleanore Jin <el...@gmail.com>
Date: Wednesday, March 4, 2020 at 4:40 PM
To: user <us...@flink.apache.org>, user-zh <us...@flink.apache.org>
Subject: How to test flink job recover from checkpoint

Hi,

I have a flink application and checkpoint is enabled, I am running locally using miniCluster.

I just wonder if there is a way to simulate the failure, and verify that flink job restarts from checkpoint?

Thanks a lot!
Eleanore

Re: How to test flink job recover from checkpoint

Posted by Zhu Zhu <re...@gmail.com>.
Hi Eleanore,

You can change your application tasks to throw exceptions in a certain
frequency.
Alternatively, if the application has external dependencies (e.g. source),
you can trigger failures manually by manipulating the status of the
external service (e.g. shutdown the source service, or break the network
connection between the Flink app and the source service).

Thanks,
Zhu Zhu

Eleanore Jin <el...@gmail.com> 于2020年3月5日周四 上午8:40写道：

> Hi,
>
> I have a flink application and checkpoint is enabled, I am running locally
> using miniCluster.
>
> I just wonder if there is a way to simulate the failure, and verify that
> flink job restarts from checkpoint?
>
> Thanks a lot!
> Eleanore
>