Re: [DISCUSS] FLIP-182: Watermark alignment

[akshay padmanabhan <ak...@gmail.com>]

Thanks, Piotr and Arvid. Like Thomas even I'm interested in this feature 
and was wondering if I can also contribute in some means in this effort.

Thanks 
Akshay
On 2021/09/08 15:44:01, Thomas Weise <th...@apache.org> wrote: 
> Thank you Piotr and Arvid for the context.
> 
> I'm interested in helping with this feature. If I'm able to contribute
> before October, then I will reach out.
> 
> Thanks,
> Thomas
> 
> 
> On Tue, Sep 7, 2021 at 11:33 PM Arvid Heise <ar...@apache.org> wrote:
> 
> > Just to clarify: I specifically asked Piotr to not persue the FLIP if the
> > current state wouldn't make it in 1.14, such that someone else can take it
> > over and expand it towards per-split alignment. Having a minimalistic
> > version in 1.14 + amendment FLIP in 1.15 would have been fine but now I
> > rather want to have it completely done in one go.
> >
> > I expect to work on it in October, so feel free to go ahead if you can make
> > it sooner.
> >
> > Best,
> >
> > Arvid
> > On Tue, Sep 7, 2021 at 8:41 PM Piotr Nowojski <pn...@apache.org>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > Unfortunately me/Arvid didn't have enough time to finish this off for
> > > 1.14.0 as we were firefighting other efforts and we have re-focused on
> > > other more advanced FLIPs. We want to deliver it for 1.15 though. I'm not
> > > sure, but I remember Arvid saying something that he would like to
> > actually
> > > take a look at this in 1.15 cycle with per-split throttling. If not, at
> > the
> > > very least I would like to contribute the version without the per-split
> > > logic, as this is almost done.
> > >
> > > Piotrek
> > >
> > > wt., 7 wrz 2021 o 19:18 Thomas Weise <th...@apache.org> napisał(a):
> > >
> > > > Hi,
> > > >
> > > > I wanted to check if there is active development on FLIP-182 and what
> > the
> > > > target release for it might be? [1] still shows as under discussion.
> > > >
> > > > Regarding the per-subtask vs. per-split limitation: I think it will be
> > > > important that this eventually works per split, since in some cases it
> > > > won't be practical to limit a subtask to a single split (think
> > > KafkaSource
> > > > reading from many topics with diverse volumes).
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources
> > > >
> > > > On Wed, Jul 21, 2021 at 4:48 AM Piotr Nowojski <pn...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > >  I would not fully advertise this before we have the second part
> > > > > implemented as well.
> > > > >
> > > > > I'm not sure, maybe we could advertise with a big warning about this
> > > > > limitation. I mean it's not as if this change would break something.
> > At
> > > > > worst it just wouldn't fully solve the problem with multiple splits
> > per
> > > > > single operator, but only limit the scope of that problem. At the
> > same
> > > > time
> > > > > I don't have strong feelings about this. If the consensus would be to
> > > not
> > > > > advertise it, I'm also fine with it. Only in that case we should
> > > probably
> > > > > quickly follow up with the per split solution.
> > > > >
> > > > > Anyway, thanks for voicing your support and the discussions. I'm
> > going
> > > to
> > > > > start a voting thread for this feature soon.
> > > > >
> > > > > Best,
> > > > > Piotrek
> > > > >
> > > > > wt., 13 lip 2021 o 19:09 Stephan Ewen <se...@apache.org> napisał(a):
> > > > >
> > > > > > @Eron Wright <er...@gmail.com>  The per-split watermarks are
> > > the
> > > > > > default in the new source interface (FLIP-27) and come for free if
> > > you
> > > > > use
> > > > > > the SplitReader.
> > > > > >
> > > > > > Based on that, it is also possible to unsubscribe individual splits
> > > to
> > > > > > solve the alignment in the case where operators have multiple
> > splits
> > > > > > assigned.
> > > > > > Piotr and I already discussed that, but concluded that the
> > > > implementation
> > > > > > of that is largely orthogonal.
> > > > > >
> > > > > > I am a bit worried, though, that if we release and advertise the
> > > > > alignment
> > > > > > without handling this case, we create a surprise for quite a few
> > > users.
> > > > > > While this is admittedly valuable for some users, I think we need
> > to
> > > > > > position this accordingly. I would not fully advertise this before
> > we
> > > > > have
> > > > > > the second part implemented as well.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 12, 2021 at 7:18 PM Eron Wright <
> > ewright@streamnative.io
> > > > > > .invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > The notion of per-split watermarks seems quite interesting.  I
> > > think
> > > > > the
> > > > > > > idleness feature could benefit from a per-split approach too,
> > > because
> > > > > > > idleness is typically related to whether any splits are assigned
> > > to a
> > > > > > given
> > > > > > > operator instance.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 12, 2021 at 3:06 AM 刘建刚 <li...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > +1 for the source watermark alignment.
> > > > > > > > In the previous flink version, the source connectors are
> > > different
> > > > in
> > > > > > > > implementation and it is hard to make this feature. When the
> > > > consumed
> > > > > > > data
> > > > > > > > is not aligned or consuming history data, it is very easy to
> > > cause
> > > > > the
> > > > > > > > unalignment. Source alignment can resolve many unstable
> > problems.
> > > > > > > >
> > > > > > > > Seth Wiesman <sj...@gmail.com> 于2021年7月9日周五 下午11:25写道：
> > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > In my opinion, this limitation is perfectly fine for the MVP.
> > > > > > Watermark
> > > > > > > > > alignment is a long-standing issue and this already moves the
> > > > ball
> > > > > so
> > > > > > > far
> > > > > > > > > forward.
> > > > > > > > >
> > > > > > > > > I don't expect this will cause many issues in practice, as I
> > > > > > understand
> > > > > > > > it
> > > > > > > > > the FileSource always processes one split at a time, and in
> > my
> > > > > > > > experience,
> > > > > > > > > 90% of Kafka users have a small number of partitions scale
> > > their
> > > > > > > > pipelines
> > > > > > > > > to have one reader per partition. Obviously, there are
> > > > larger-scale
> > > > > > > Kafka
> > > > > > > > > topics and more sources that will be ported over in the
> > future
> > > > but
> > > > > I
> > > > > > > > think
> > > > > > > > > there is an implicit understanding that aligning sources adds
> > > > > latency
> > > > > > > to
> > > > > > > > > pipelines, and we can frame the follow-up "per-split"
> > alignment
> > > > as
> > > > > an
> > > > > > > > > optimization.
> > > > > > > > >
> > > > > > > > > On Fri, Jul 9, 2021 at 6:40 AM Piotr Nowojski <
> > > > > > > piotr.nowojski@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey!
> > > > > > > > > >
> > > > > > > > > > A couple of weeks ago me and Arvid Heise played around with
> > > an
> > > > > idea
> > > > > > > to
> > > > > > > > > > address a long standing issue of Flink: lack of
> > > watermark/event
> > > > > > time
> > > > > > > > > > alignment between different parallel instances of sources,
> > > that
> > > > > can
> > > > > > > > lead
> > > > > > > > > to
> > > > > > > > > > ever growing state size for downstream operators like
> > > > > > WindowOperator.
> > > > > > > > > >
> > > > > > > > > > We had an impression that this is relatively low hanging
> > > fruit
> > > > > that
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > > quite easily implemented - at least partially (the first
> > part
> > > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > the FLIP document). I have written down our proposal [1]
> > and
> > > > you
> > > > > > can
> > > > > > > > also
> > > > > > > > > > check out our PoC that we have implemented [2].
> > > > > > > > > >
> > > > > > > > > > We think that this is a quite easy proposal, that has been
> > in
> > > > > large
> > > > > > > > part
> > > > > > > > > > already implemented. There is one obvious limitation of our
> > > > PoC.
> > > > > > > Namely
> > > > > > > > > we
> > > > > > > > > > can only easily block individual SourceOperators. This
> > works
> > > > > > > perfectly
> > > > > > > > > fine
> > > > > > > > > > as long as there is at most one split per SourceOperator.
> > > > However
> > > > > > it
> > > > > > > > > > doesn't work with multiple splits. In that case, if a
> > single
> > > > > > > > > > `SourceOperator` is responsible for processing both the
> > least
> > > > and
> > > > > > the
> > > > > > > > > most
> > > > > > > > > > advanced splits, we won't be able to block this most
> > advanced
> > > > > split
> > > > > > > for
> > > > > > > > > > generating new records. I'm proposing to solve this problem
> > > in
> > > > > the
> > > > > > > > future
> > > > > > > > > > in another follow up FLIP, as a solution that works with a
> > > > single
> > > > > > > split
> > > > > > > > > per
> > > > > > > > > > operator is easier and already valuable for some of the
> > > users.
> > > > > > > > > >
> > > > > > > > > > What do you think about this proposal?
> > > > > > > > > > Best, Piotrek
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources
> > > > > > > > > > [2]
> > > https://github.com/pnowojski/flink/commits/aligned-sources
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] FLIP-182: Watermark alignment

[Piotr Nowojski <pn...@apache.org>]

Hi Thomas and Akshay,

Sorry for the late reply. Due to problems with other FLIPs, we had to scrap
our plans to publish FLIP-182 in Flink 1.14. However as far as I know,
Arvid is working on this right now, and we are tentatively aiming with this
feature for the 1.15.0 release. I hope Arvid will be able to report
something back in the next couple of weeks.

Best,
Piotrek

pon., 11 paź 2021 o 19:13 akshay padmanabhan <ak...@gmail.com>
napisał(a):

>
> Thanks, Piotr and Arvid. Like Thomas even I'm interested in this feature
> and was wondering if I can also contribute in some means in this effort.
>
> Thanks
> Akshay
> On 2021/09/08 15:44:01, Thomas Weise <th...@apache.org> wrote:
> > Thank you Piotr and Arvid for the context.
> >
> > I'm interested in helping with this feature. If I'm able to contribute
> > before October, then I will reach out.
> >
> > Thanks,
> > Thomas
> >
> >
> > On Tue, Sep 7, 2021 at 11:33 PM Arvid Heise <ar...@apache.org> wrote:
> >
> > > Just to clarify: I specifically asked Piotr to not persue the FLIP if
> the
> > > current state wouldn't make it in 1.14, such that someone else can
> take it
> > > over and expand it towards per-split alignment. Having a minimalistic
> > > version in 1.14 + amendment FLIP in 1.15 would have been fine but now I
> > > rather want to have it completely done in one go.
> > >
> > > I expect to work on it in October, so feel free to go ahead if you can
> make
> > > it sooner.
> > >
> > > Best,
> > >
> > > Arvid
> > > On Tue, Sep 7, 2021 at 8:41 PM Piotr Nowojski <pn...@apache.org>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > Unfortunately me/Arvid didn't have enough time to finish this off for
> > > > 1.14.0 as we were firefighting other efforts and we have re-focused
> on
> > > > other more advanced FLIPs. We want to deliver it for 1.15 though.
> I'm not
> > > > sure, but I remember Arvid saying something that he would like to
> > > actually
> > > > take a look at this in 1.15 cycle with per-split throttling. If not,
> at
> > > the
> > > > very least I would like to contribute the version without the
> per-split
> > > > logic, as this is almost done.
> > > >
> > > > Piotrek
> > > >
> > > > wt., 7 wrz 2021 o 19:18 Thomas Weise <th...@apache.org> napisał(a):
> > > >
> > > > > Hi,
> > > > >
> > > > > I wanted to check if there is active development on FLIP-182 and
> what
> > > the
> > > > > target release for it might be? [1] still shows as under
> discussion.
> > > > >
> > > > > Regarding the per-subtask vs. per-split limitation: I think it
> will be
> > > > > important that this eventually works per split, since in some
> cases it
> > > > > won't be practical to limit a subtask to a single split (think
> > > > KafkaSource
> > > > > reading from many topics with diverse volumes).
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources
> > > > >
> > > > > On Wed, Jul 21, 2021 at 4:48 AM Piotr Nowojski <
> pnowojski@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > >  I would not fully advertise this before we have the second
> part
> > > > > > implemented as well.
> > > > > >
> > > > > > I'm not sure, maybe we could advertise with a big warning about
> this
> > > > > > limitation. I mean it's not as if this change would break
> something.
> > > At
> > > > > > worst it just wouldn't fully solve the problem with multiple
> splits
> > > per
> > > > > > single operator, but only limit the scope of that problem. At the
> > > same
> > > > > time
> > > > > > I don't have strong feelings about this. If the consensus would
> be to
> > > > not
> > > > > > advertise it, I'm also fine with it. Only in that case we should
> > > > probably
> > > > > > quickly follow up with the per split solution.
> > > > > >
> > > > > > Anyway, thanks for voicing your support and the discussions. I'm
> > > going
> > > > to
> > > > > > start a voting thread for this feature soon.
> > > > > >
> > > > > > Best,
> > > > > > Piotrek
> > > > > >
> > > > > > wt., 13 lip 2021 o 19:09 Stephan Ewen <se...@apache.org>
> napisał(a):
> > > > > >
> > > > > > > @Eron Wright <er...@gmail.com>  The per-split watermarks
> are
> > > > the
> > > > > > > default in the new source interface (FLIP-27) and come for
> free if
> > > > you
> > > > > > use
> > > > > > > the SplitReader.
> > > > > > >
> > > > > > > Based on that, it is also possible to unsubscribe individual
> splits
> > > > to
> > > > > > > solve the alignment in the case where operators have multiple
> > > splits
> > > > > > > assigned.
> > > > > > > Piotr and I already discussed that, but concluded that the
> > > > > implementation
> > > > > > > of that is largely orthogonal.
> > > > > > >
> > > > > > > I am a bit worried, though, that if we release and advertise
> the
> > > > > > alignment
> > > > > > > without handling this case, we create a surprise for quite a
> few
> > > > users.
> > > > > > > While this is admittedly valuable for some users, I think we
> need
> > > to
> > > > > > > position this accordingly. I would not fully advertise this
> before
> > > we
> > > > > > have
> > > > > > > the second part implemented as well.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 12, 2021 at 7:18 PM Eron Wright <
> > > ewright@streamnative.io
> > > > > > > .invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > The notion of per-split watermarks seems quite interesting.
> I
> > > > think
> > > > > > the
> > > > > > > > idleness feature could benefit from a per-split approach too,
> > > > because
> > > > > > > > idleness is typically related to whether any splits are
> assigned
> > > > to a
> > > > > > > given
> > > > > > > > operator instance.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 12, 2021 at 3:06 AM 刘建刚 <
> liujiangangpeng@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 for the source watermark alignment.
> > > > > > > > > In the previous flink version, the source connectors are
> > > > different
> > > > > in
> > > > > > > > > implementation and it is hard to make this feature. When
> the
> > > > > consumed
> > > > > > > > data
> > > > > > > > > is not aligned or consuming history data, it is very easy
> to
> > > > cause
> > > > > > the
> > > > > > > > > unalignment. Source alignment can resolve many unstable
> > > problems.
> > > > > > > > >
> > > > > > > > > Seth Wiesman <sj...@gmail.com> 于2021年7月9日周五 下午11:25写道：
> > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > > In my opinion, this limitation is perfectly fine for the
> MVP.
> > > > > > > Watermark
> > > > > > > > > > alignment is a long-standing issue and this already
> moves the
> > > > > ball
> > > > > > so
> > > > > > > > far
> > > > > > > > > > forward.
> > > > > > > > > >
> > > > > > > > > > I don't expect this will cause many issues in practice,
> as I
> > > > > > > understand
> > > > > > > > > it
> > > > > > > > > > the FileSource always processes one split at a time, and
> in
> > > my
> > > > > > > > > experience,
> > > > > > > > > > 90% of Kafka users have a small number of partitions
> scale
> > > > their
> > > > > > > > > pipelines
> > > > > > > > > > to have one reader per partition. Obviously, there are
> > > > > larger-scale
> > > > > > > > Kafka
> > > > > > > > > > topics and more sources that will be ported over in the
> > > future
> > > > > but
> > > > > > I
> > > > > > > > > think
> > > > > > > > > > there is an implicit understanding that aligning sources
> adds
> > > > > > latency
> > > > > > > > to
> > > > > > > > > > pipelines, and we can frame the follow-up "per-split"
> > > alignment
> > > > > as
> > > > > > an
> > > > > > > > > > optimization.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 9, 2021 at 6:40 AM Piotr Nowojski <
> > > > > > > > piotr.nowojski@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey!
> > > > > > > > > > >
> > > > > > > > > > > A couple of weeks ago me and Arvid Heise played around
> with
> > > > an
> > > > > > idea
> > > > > > > > to
> > > > > > > > > > > address a long standing issue of Flink: lack of
> > > > watermark/event
> > > > > > > time
> > > > > > > > > > > alignment between different parallel instances of
> sources,
> > > > that
> > > > > > can
> > > > > > > > > lead
> > > > > > > > > > to
> > > > > > > > > > > ever growing state size for downstream operators like
> > > > > > > WindowOperator.
> > > > > > > > > > >
> > > > > > > > > > > We had an impression that this is relatively low
> hanging
> > > > fruit
> > > > > > that
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > quite easily implemented - at least partially (the
> first
> > > part
> > > > > > > > mentioned
> > > > > > > > > > in
> > > > > > > > > > > the FLIP document). I have written down our proposal
> [1]
> > > and
> > > > > you
> > > > > > > can
> > > > > > > > > also
> > > > > > > > > > > check out our PoC that we have implemented [2].
> > > > > > > > > > >
> > > > > > > > > > > We think that this is a quite easy proposal, that has
> been
> > > in
> > > > > > large
> > > > > > > > > part
> > > > > > > > > > > already implemented. There is one obvious limitation
> of our
> > > > > PoC.
> > > > > > > > Namely
> > > > > > > > > > we
> > > > > > > > > > > can only easily block individual SourceOperators. This
> > > works
> > > > > > > > perfectly
> > > > > > > > > > fine
> > > > > > > > > > > as long as there is at most one split per
> SourceOperator.
> > > > > However
> > > > > > > it
> > > > > > > > > > > doesn't work with multiple splits. In that case, if a
> > > single
> > > > > > > > > > > `SourceOperator` is responsible for processing both the
> > > least
> > > > > and
> > > > > > > the
> > > > > > > > > > most
> > > > > > > > > > > advanced splits, we won't be able to block this most
> > > advanced
> > > > > > split
> > > > > > > > for
> > > > > > > > > > > generating new records. I'm proposing to solve this
> problem
> > > > in
> > > > > > the
> > > > > > > > > future
> > > > > > > > > > > in another follow up FLIP, as a solution that works
> with a
> > > > > single
> > > > > > > > split
> > > > > > > > > > per
> > > > > > > > > > > operator is easier and already valuable for some of the
> > > > users.
> > > > > > > > > > >
> > > > > > > > > > > What do you think about this proposal?
> > > > > > > > > > > Best, Piotrek
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources
> > > > > > > > > > > [2]
> > > > https://github.com/pnowojski/flink/commits/aligned-sources
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>