Reg. the checkpointing mechanism

[Data Engineer <da...@gmail.com>]

I have a Flink application (v 1.4.2) where I have enabled checkpointing
with an interval of 200 ms.
This is what I have configured in my driver program:

            env.enableCheckpointing(checkpointInterval,
CheckpointingMode.EXACTLY_ONCE);
            CheckpointConfig config = env.getCheckpointConfig();

config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

In the JobManager logs, I see multiple entries saying "Checkpoint
triggered".
I would like to clarify whether the JobManager is triggering the checkpoint
every 200 ms? Or does the JobManager only initiate the checkpointing and
the TaskManager do it on its own?

Regards,
James

Re: Reg. the checkpointing mechanism

[Nico Kruber <ni...@data-artisans.com>]

Hi James,
The checkpoint coordinator at the JobManager is triggering the
checkpoints by inserting checkpoint barriers into the sources. These
will get to the TaskManagers via the same communication channels data is
flowing between them. Please refer to [1] for more details.


Nico


[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/internals/stream_checkpointing.html
On 06/04/18 13:47, Data Engineer wrote:
> I have a Flink application (v 1.4.2) where I have enabled checkpointing
> with an interval of 200 ms.
> This is what I have configured in my driver program:
> 
>             env.enableCheckpointing(checkpointInterval,
> CheckpointingMode.EXACTLY_ONCE);
>             CheckpointConfig config = env.getCheckpointConfig();
>            
> config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
> 
> In the JobManager logs, I see multiple entries saying "Checkpoint
> triggered".
> I would like to clarify whether the JobManager is triggering the
> checkpoint every 200 ms? Or does the JobManager only initiate the
> checkpointing and the TaskManager do it on its own?
> 
> Regards,
> James
> 
> 
> 

-- 
Nico Kruber | Software Engineer
data Artisans

Follow us @dataArtisans
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Stresemannstr. 121A,10963 Berlin, Germany
data Artisans, Inc. | 1161 Mission Street, San Francisco, CA-94103, USA
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen