Introduce Barriers in stream source

[Darshan Singh <da...@gmail.com>]

Hi,

I am implementing a source and I want to use checkpointing and would like
to restore the job from these external checkpoints. I used Kafka for my
tests and it worked fine.

However, I would like to know if I have my own source what do I need to do.
I am sure that I will need to implement CheckpointedFunctions
(initializeState and snapshotState) . Based on what I read and looked at
implemetation of Kafka source I do not need to do anything else apart from
this.
Maybe notifyCheckpointComplete also.

But I would like to confirm if I will need to implement something to create
Barrier in my source.

Thanks

Re: Introduce Barriers in stream source

[Fabian Hueske <fh...@gmail.com>]

Hi,

It is sufficient to implement the CheckpointedFunction interface.
Since SourceFunctions emit records in a separate thread, you need to ensure
that not record is emitted while the shapshotState method is called.
Flink provides a lock to synchronize data emission and state snapshotting.
See the JavaDocs of SourceFunction for details.

Best, Fabian

2018-08-13 16:42 GMT+02:00 Darshan Singh <da...@gmail.com>:

> Hi,
>
> I am implementing a source and I want to use checkpointing and would like
> to restore the job from these external checkpoints. I used Kafka for my
> tests and it worked fine.
>
> However, I would like to know if I have my own source what do I need to
> do. I am sure that I will need to implement CheckpointedFunctions
> (initializeState and snapshotState) . Based on what I read and looked at
> implemetation of Kafka source I do not need to do anything else apart from
> this.
> Maybe notifyCheckpointComplete also.
>
> But I would like to confirm if I will need to implement something to
> create Barrier in my source.
>
> Thanks
>