You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by HG <ha...@gmail.com> on 2022/01/23 18:03:17 UTC
Flink checkpoints
Hi,
Perhaps a dumb question.
How is checkpointing implemented when the final framework is Flink?
With Flink checkpoint barriers are inserted.
Can such an interval also be configured with HOP?
Regards Hans-Peter
Re: Flink checkpoints
Posted by Maximilian Michels <mx...@apache.org>.
I see! AFAIK the watermark strategy is not configurable via Hop at the
moment. You would have to use the default WatermarkStrategy of the
source you are using.
Cheers,
Max
On 27.01.22 17:13, HG wrote:
> Well I wanted to use HOP to avoid programming in Java or Scala😀.
> Since I am not a skilled Java coder.
>
>
> On Thu, Jan 27, 2022, 15:02 Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Oh yes! I now remember that Hop uses Beam to run on Flink. Anyhow, the
> basic checkpoint options are there. Very true, that we might have to
> add
> any new ones. Maybe a chance for Hans-Peter to contribute? :)
>
> Cheers,
> Max
>
> On 27.01.22 11:55, Matt Casters wrote:
> > You are correct. Hop uses the Flink runner of Apache Beam though
> and we
> > did upgrade to the Flink version 1.13 runner in our latest Hop
> version
> > so there's always the chance that new options became available in
> Flink
> > which are not yet reflected in the runner or in Hop.
> > There's really no good communications channel that I know of to
> signal
> > downstream projects of available changes so we just keep an eye
> out for
> > them.
> >
> > Cheers,
> > Matt
> >
> > Op do 27 jan. 2022 11:49 schreef Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>:
> >
> > It looks to me, Hans-Peter's question was not about Beam but
> Flink. The
> > provided link should answer his question regarding setting the
> > checkpoint interval.
> >
> > -Max
> >
> > On 27.01.22 11:21, Matt Casters wrote:
> > > It can happen that we lag a bit behind when Apache Beam adds
> > options for
> > > specific runners.
> > > So always let us know if there's an option that's not
> available.
> > > In fact, if the option is not available in Beam either we can
> > create a
> > > change against Apache Beam itself.
> > > Cheers,
> > > Matt
> > >
> > > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Hi Hans-Peter,
> > >
> > > Checkpoints are internal to Flink. Hop does not have any
> > business with
> > > it, other than configuring the checkpoint options.
> Those can
> > also be
> > > configured via the Flink config file.
> > >
> > > I see that Hop has checkpointing options listed here:
> > >
> >
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>
> >
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>
> > >
> >
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>>
> > >
> > > Cheers,
> > > -Max
> > >
> > > On 23.01.22 19:03, HG wrote:
> > > > Hi,
> > > >
> > > > Perhaps a dumb question.
> > > > How is checkpointing implemented when the final
> framework
> > is Flink?
> > > > With Flink checkpoint barriers are inserted.
> > > > Can such an interval also be configured with HOP?
> > > >
> > > > Regards Hans-Peter
> > >
> > >
> > >
> > >
> > >
> >
>
Re: Flink checkpoints
Posted by HG <ha...@gmail.com>.
Well I wanted to use HOP to avoid programming in Java or Scala😀.
Since I am not a skilled Java coder.
On Thu, Jan 27, 2022, 15:02 Maximilian Michels <mx...@apache.org> wrote:
> Oh yes! I now remember that Hop uses Beam to run on Flink. Anyhow, the
> basic checkpoint options are there. Very true, that we might have to add
> any new ones. Maybe a chance for Hans-Peter to contribute? :)
>
> Cheers,
> Max
>
> On 27.01.22 11:55, Matt Casters wrote:
> > You are correct. Hop uses the Flink runner of Apache Beam though and we
> > did upgrade to the Flink version 1.13 runner in our latest Hop version
> > so there's always the chance that new options became available in Flink
> > which are not yet reflected in the runner or in Hop.
> > There's really no good communications channel that I know of to signal
> > downstream projects of available changes so we just keep an eye out for
> > them.
> >
> > Cheers,
> > Matt
> >
> > Op do 27 jan. 2022 11:49 schreef Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>>:
> >
> > It looks to me, Hans-Peter's question was not about Beam but Flink.
> The
> > provided link should answer his question regarding setting the
> > checkpoint interval.
> >
> > -Max
> >
> > On 27.01.22 11:21, Matt Casters wrote:
> > > It can happen that we lag a bit behind when Apache Beam adds
> > options for
> > > specific runners.
> > > So always let us know if there's an option that's not available.
> > > In fact, if the option is not available in Beam either we can
> > create a
> > > change against Apache Beam itself.
> > > Cheers,
> > > Matt
> > >
> > > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Hi Hans-Peter,
> > >
> > > Checkpoints are internal to Flink. Hop does not have any
> > business with
> > > it, other than configuring the checkpoint options. Those can
> > also be
> > > configured via the Flink config file.
> > >
> > > I see that Hop has checkpointing options listed here:
> > >
> >
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> > <
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> >
> > >
> > <
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> <
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> >>
> > >
> > > Cheers,
> > > -Max
> > >
> > > On 23.01.22 19:03, HG wrote:
> > > > Hi,
> > > >
> > > > Perhaps a dumb question.
> > > > How is checkpointing implemented when the final framework
> > is Flink?
> > > > With Flink checkpoint barriers are inserted.
> > > > Can such an interval also be configured with HOP?
> > > >
> > > > Regards Hans-Peter
> > >
> > >
> > >
> > >
> > >
> >
>
Re: Flink checkpoints
Posted by Maximilian Michels <mx...@apache.org>.
Oh yes! I now remember that Hop uses Beam to run on Flink. Anyhow, the
basic checkpoint options are there. Very true, that we might have to add
any new ones. Maybe a chance for Hans-Peter to contribute? :)
Cheers,
Max
On 27.01.22 11:55, Matt Casters wrote:
> You are correct. Hop uses the Flink runner of Apache Beam though and we
> did upgrade to the Flink version 1.13 runner in our latest Hop version
> so there's always the chance that new options became available in Flink
> which are not yet reflected in the runner or in Hop.
> There's really no good communications channel that I know of to signal
> downstream projects of available changes so we just keep an eye out for
> them.
>
> Cheers,
> Matt
>
> Op do 27 jan. 2022 11:49 schreef Maximilian Michels <mxm@apache.org
> <ma...@apache.org>>:
>
> It looks to me, Hans-Peter's question was not about Beam but Flink. The
> provided link should answer his question regarding setting the
> checkpoint interval.
>
> -Max
>
> On 27.01.22 11:21, Matt Casters wrote:
> > It can happen that we lag a bit behind when Apache Beam adds
> options for
> > specific runners.
> > So always let us know if there's an option that's not available.
> > In fact, if the option is not available in Beam either we can
> create a
> > change against Apache Beam itself.
> > Cheers,
> > Matt
> >
> > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Hi Hans-Peter,
> >
> > Checkpoints are internal to Flink. Hop does not have any
> business with
> > it, other than configuring the checkpoint options. Those can
> also be
> > configured via the Flink config file.
> >
> > I see that Hop has checkpointing options listed here:
> >
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>
> >
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>
> >
> > Cheers,
> > -Max
> >
> > On 23.01.22 19:03, HG wrote:
> > > Hi,
> > >
> > > Perhaps a dumb question.
> > > How is checkpointing implemented when the final framework
> is Flink?
> > > With Flink checkpoint barriers are inserted.
> > > Can such an interval also be configured with HOP?
> > >
> > > Regards Hans-Peter
> >
> >
> >
> >
> >
>
Re: Flink checkpoints
Posted by Matt Casters <ma...@neo4j.com>.
You are correct. Hop uses the Flink runner of Apache Beam though and we
did upgrade to the Flink version 1.13 runner in our latest Hop version so
there's always the chance that new options became available in Flink which
are not yet reflected in the runner or in Hop.
There's really no good communications channel that I know of to signal
downstream projects of available changes so we just keep an eye out for
them.
Cheers,
Matt
Op do 27 jan. 2022 11:49 schreef Maximilian Michels <mx...@apache.org>:
> It looks to me, Hans-Peter's question was not about Beam but Flink. The
> provided link should answer his question regarding setting the
> checkpoint interval.
>
> -Max
>
> On 27.01.22 11:21, Matt Casters wrote:
> > It can happen that we lag a bit behind when Apache Beam adds options for
> > specific runners.
> > So always let us know if there's an option that's not available.
> > In fact, if the option is not available in Beam either we can create a
> > change against Apache Beam itself.
> > Cheers,
> > Matt
> >
> > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi Hans-Peter,
> >
> > Checkpoints are internal to Flink. Hop does not have any business
> with
> > it, other than configuring the checkpoint options. Those can also be
> > configured via the Flink config file.
> >
> > I see that Hop has checkpointing options listed here:
> >
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> > <
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> >
> >
> > Cheers,
> > -Max
> >
> > On 23.01.22 19:03, HG wrote:
> > > Hi,
> > >
> > > Perhaps a dumb question.
> > > How is checkpointing implemented when the final framework is
> Flink?
> > > With Flink checkpoint barriers are inserted.
> > > Can such an interval also be configured with HOP?
> > >
> > > Regards Hans-Peter
> >
> >
> >
> >
> >
>
Re: Flink checkpoints
Posted by Matt Casters <ma...@neo4j.com>.
You are correct. Hop uses the Flink runner of Apache Beam though and we
did upgrade to the Flink version 1.13 runner in our latest Hop version so
there's always the chance that new options became available in Flink which
are not yet reflected in the runner or in Hop.
There's really no good communications channel that I know of to signal
downstream projects of available changes so we just keep an eye out for
them.
Cheers,
Matt
Op do 27 jan. 2022 11:49 schreef Maximilian Michels <mx...@apache.org>:
> It looks to me, Hans-Peter's question was not about Beam but Flink. The
> provided link should answer his question regarding setting the
> checkpoint interval.
>
> -Max
>
> On 27.01.22 11:21, Matt Casters wrote:
> > It can happen that we lag a bit behind when Apache Beam adds options for
> > specific runners.
> > So always let us know if there's an option that's not available.
> > In fact, if the option is not available in Beam either we can create a
> > change against Apache Beam itself.
> > Cheers,
> > Matt
> >
> > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi Hans-Peter,
> >
> > Checkpoints are internal to Flink. Hop does not have any business
> with
> > it, other than configuring the checkpoint options. Those can also be
> > configured via the Flink config file.
> >
> > I see that Hop has checkpointing options listed here:
> >
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> > <
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> >
> >
> > Cheers,
> > -Max
> >
> > On 23.01.22 19:03, HG wrote:
> > > Hi,
> > >
> > > Perhaps a dumb question.
> > > How is checkpointing implemented when the final framework is
> Flink?
> > > With Flink checkpoint barriers are inserted.
> > > Can such an interval also be configured with HOP?
> > >
> > > Regards Hans-Peter
> >
> >
> >
> >
> >
>
Re: Flink checkpoints
Posted by Maximilian Michels <mx...@apache.org>.
It looks to me, Hans-Peter's question was not about Beam but Flink. The
provided link should answer his question regarding setting the
checkpoint interval.
-Max
On 27.01.22 11:21, Matt Casters wrote:
> It can happen that we lag a bit behind when Apache Beam adds options for
> specific runners.
> So always let us know if there's an option that's not available.
> In fact, if the option is not available in Beam either we can create a
> change against Apache Beam itself.
> Cheers,
> Matt
>
> On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Hi Hans-Peter,
>
> Checkpoints are internal to Flink. Hop does not have any business with
> it, other than configuring the checkpoint options. Those can also be
> configured via the Flink config file.
>
> I see that Hop has checkpointing options listed here:
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>
>
> Cheers,
> -Max
>
> On 23.01.22 19:03, HG wrote:
> > Hi,
> >
> > Perhaps a dumb question.
> > How is checkpointing implemented when the final framework is Flink?
> > With Flink checkpoint barriers are inserted.
> > Can such an interval also be configured with HOP?
> >
> > Regards Hans-Peter
>
>
>
>
>
Re: Flink checkpoints
Posted by Matt Casters <ma...@neo4j.com>.
It can happen that we lag a bit behind when Apache Beam adds options for
specific runners.
So always let us know if there's an option that's not available.
In fact, if the option is not available in Beam either we can create a
change against Apache Beam itself.
Cheers,
Matt
On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels <mx...@apache.org> wrote:
> Hi Hans-Peter,
>
> Checkpoints are internal to Flink. Hop does not have any business with
> it, other than configuring the checkpoint options. Those can also be
> configured via the Flink config file.
>
> I see that Hop has checkpointing options listed here:
>
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
>
> Cheers,
> -Max
>
> On 23.01.22 19:03, HG wrote:
> > Hi,
> >
> > Perhaps a dumb question.
> > How is checkpointing implemented when the final framework is Flink?
> > With Flink checkpoint barriers are inserted.
> > Can such an interval also be configured with HOP?
> >
> > Regards Hans-Peter
>
Re: Flink checkpoints
Posted by Matt Casters <ma...@neo4j.com>.
It can happen that we lag a bit behind when Apache Beam adds options for
specific runners.
So always let us know if there's an option that's not available.
In fact, if the option is not available in Beam either we can create a
change against Apache Beam itself.
Cheers,
Matt
On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels <mx...@apache.org> wrote:
> Hi Hans-Peter,
>
> Checkpoints are internal to Flink. Hop does not have any business with
> it, other than configuring the checkpoint options. Those can also be
> configured via the Flink config file.
>
> I see that Hop has checkpointing options listed here:
>
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
>
> Cheers,
> -Max
>
> On 23.01.22 19:03, HG wrote:
> > Hi,
> >
> > Perhaps a dumb question.
> > How is checkpointing implemented when the final framework is Flink?
> > With Flink checkpoint barriers are inserted.
> > Can such an interval also be configured with HOP?
> >
> > Regards Hans-Peter
>
Re: Flink checkpoints
Posted by Maximilian Michels <mx...@apache.org>.
Hi Hans-Peter,
Checkpoints are internal to Flink. Hop does not have any business with
it, other than configuring the checkpoint options. Those can also be
configured via the Flink config file.
I see that Hop has checkpointing options listed here:
https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
Cheers,
-Max
On 23.01.22 19:03, HG wrote:
> Hi,
>
> Perhaps a dumb question.
> How is checkpointing implemented when the final framework is Flink?
> With Flink checkpoint barriers are inserted.
> Can such an interval also be configured with HOP?
>
> Regards Hans-Peter