You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by Maximilian Michels <mx...@apache.org> on 2022/02/14 10:53:57 UTC
Re: Flink checkpoints
I see! AFAIK the watermark strategy is not configurable via Hop at the
moment. You would have to use the default WatermarkStrategy of the
source you are using.
Cheers,
Max
On 27.01.22 17:13, HG wrote:
> Well I wanted to use HOP to avoid programming in Java or Scala😀.
> Since I am not a skilled Java coder.
>
>
> On Thu, Jan 27, 2022, 15:02 Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Oh yes! I now remember that Hop uses Beam to run on Flink. Anyhow, the
> basic checkpoint options are there. Very true, that we might have to
> add
> any new ones. Maybe a chance for Hans-Peter to contribute? :)
>
> Cheers,
> Max
>
> On 27.01.22 11:55, Matt Casters wrote:
> > You are correct. Hop uses the Flink runner of Apache Beam though
> and we
> > did upgrade to the Flink version 1.13 runner in our latest Hop
> version
> > so there's always the chance that new options became available in
> Flink
> > which are not yet reflected in the runner or in Hop.
> > There's really no good communications channel that I know of to
> signal
> > downstream projects of available changes so we just keep an eye
> out for
> > them.
> >
> > Cheers,
> > Matt
> >
> > Op do 27 jan. 2022 11:49 schreef Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>:
> >
> > It looks to me, Hans-Peter's question was not about Beam but
> Flink. The
> > provided link should answer his question regarding setting the
> > checkpoint interval.
> >
> > -Max
> >
> > On 27.01.22 11:21, Matt Casters wrote:
> > > It can happen that we lag a bit behind when Apache Beam adds
> > options for
> > > specific runners.
> > > So always let us know if there's an option that's not
> available.
> > > In fact, if the option is not available in Beam either we can
> > create a
> > > change against Apache Beam itself.
> > > Cheers,
> > > Matt
> > >
> > > On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Hi Hans-Peter,
> > >
> > > Checkpoints are internal to Flink. Hop does not have any
> > business with
> > > it, other than configuring the checkpoint options.
> Those can
> > also be
> > > configured via the Flink config file.
> > >
> > > I see that Hop has checkpointing options listed here:
> > >
> >
> https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>
> >
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>
> > >
> >
> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options> <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>>
> > >
> > > Cheers,
> > > -Max
> > >
> > > On 23.01.22 19:03, HG wrote:
> > > > Hi,
> > > >
> > > > Perhaps a dumb question.
> > > > How is checkpointing implemented when the final
> framework
> > is Flink?
> > > > With Flink checkpoint barriers are inserted.
> > > > Can such an interval also be configured with HOP?
> > > >
> > > > Regards Hans-Peter
> > >
> > >
> > >
> > >
> > >
> >
>