You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Blake Essing <es...@swbell.net> on 2017/04/21 16:48:36 UTC
Question on checkpointing
I have modified the WordCounter example program to use a NiFi DataStream as input. I would like to add checkpointing to the program to learn how it works. I have googled and added many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing should work. My goal is to save state so that if the program is stopped and then restarted the counts will continue from the last checkpoint. The counts variable is populated by
Re: Question on checkpointing
Posted by Aljoscha Krettek <al...@apache.org>.
Hi Blake,
I didn’t see any mails on the user mailing list regarding this. Is it still relevant.
From the looks of it you might be interested in the savepoints feature: https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/savepoints.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/savepoints.html>.
This uses the checkpointing mechanism to savepoint the state of a running job so that the job can be canceled and later restored.
Best,
Aljoscha
> On 21. Apr 2017, at 20:42, Blake Essing <es...@swbell.net> wrote:
>
> Sorry all for my earlier messages. I just realized that I'm on the wrong mailing list for what I'm asking. I will move my question to the user list.
>
> On Friday, April 21, 2017 11:54 AM, Blake Essing <es...@swbell.net> wrote:
>
>
> Sorry for the earlier incomplete email, E-mail editor issue.
> I have modified the WordCounter example program to use a NiFi DataStream as input. I would like to add checkpointing to the program to learn how it works. I have googled and added many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing should work. My goal is to save state so that if the program is stopped and then restarted the counts will continue from the last checkpoint.
> The counts variable is populated by
>
> DataStream<WordCounter> counts = text .flatMap(new Tokenizer()) .keyBy("word") .reduce(new ReduceFunction<WordCounter>() { public WordCounter reduce(WordCounter a, WordCounter b) throws Exception { return new WordCounter(a.getWord, a.getCount() + b.getCount()); } })
> text is just a DataStream<String> populated from NiFi where lines are broken up into individual strings.
> WordCounter is a simple pojo that stores the word and a count with get/set methods.
> Please let me know if you have any questions about the code.
>
> On Friday, April 21, 2017 11:48 AM, Blake Essing <es...@swbell.net> wrote:
>
>
> I have modified the WordCounter example program to use a NiFi DataStream as input. I would like to add checkpointing to the program to learn how it works. I have googled and added many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing should work. My goal is to save state so that if the program is stopped and then restarted the counts will continue from the last checkpoint. The counts variable is populated by
>
>
>
Re: Question on checkpointing
Posted by Blake Essing <es...@swbell.net>.
Sorry all for my earlier messages. I just realized that I'm on the wrong mailing list for what I'm asking. I will move my question to the user list.
On Friday, April 21, 2017 11:54 AM, Blake Essing <es...@swbell.net> wrote:
Sorry for the earlier incomplete email, E-mail editor issue.
I have modified the WordCounter example program to use a NiFi DataStream as input. I would like to add checkpointing to the program to learn how it works. I have googled and added many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing should work. My goal is to save state so that if the program is stopped and then restarted the counts will continue from the last checkpoint.
The counts variable is populated by
DataStream<WordCounter> counts = text .flatMap(new Tokenizer()) .keyBy("word") .reduce(new ReduceFunction<WordCounter>() { public WordCounter reduce(WordCounter a, WordCounter b) throws Exception { return new WordCounter(a.getWord, a.getCount() + b.getCount()); } })
text is just a DataStream<String> populated from NiFi where lines are broken up into individual strings.
WordCounter is a simple pojo that stores the word and a count with get/set methods.
Please let me know if you have any questions about the code.
On Friday, April 21, 2017 11:48 AM, Blake Essing <es...@swbell.net> wrote:
I have modified the WordCounter example program to use a NiFi DataStream as input. I would like to add checkpointing to the program to learn how it works. I have googled and added many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing should work. My goal is to save state so that if the program is stopped and then restarted the counts will continue from the last checkpoint. The counts variable is populated by
Re: Question on checkpointing
Posted by Blake Essing <es...@swbell.net>.
Sorry for the earlier incomplete email, E-mail editor issue.
I have modified the WordCounter example program to use a NiFi DataStream as input. I would like to add checkpointing to the program to learn how it works. I have googled and added many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing should work. My goal is to save state so that if the program is stopped and then restarted the counts will continue from the last checkpoint.
The counts variable is populated by
DataStream<WordCounter> counts = text .flatMap(new Tokenizer()) .keyBy("word") .reduce(new ReduceFunction<WordCounter>() { public WordCounter reduce(WordCounter a, WordCounter b) throws Exception { return new WordCounter(a.getWord, a.getCount() + b.getCount()); } })
text is just a DataStream<String> populated from NiFi where lines are broken up into individual strings.
WordCounter is a simple pojo that stores the word and a count with get/set methods.
Please let me know if you have any questions about the code.
On Friday, April 21, 2017 11:48 AM, Blake Essing <es...@swbell.net> wrote:
I have modified the WordCounter example program to use a NiFi DataStream as input. I would like to add checkpointing to the program to learn how it works. I have googled and added many of the checkpointing configuration values to the StreamExecutionEnvironment but I'm missing something critical to getting this to work or I'm misinterpreting how the checkpointing should work. My goal is to save state so that if the program is stopped and then restarted the counts will continue from the last checkpoint. The counts variable is populated by